Correct horse battery staple:
Exploring the usability of system-assigned passphrases
Richard Shay, Patrick Gage Kelley, Saranga Komanduri, Michelle L. Mazurek,
Blase Ur, Timothy Vidas, Lujo Bauer, Nicolas Christin, Lorrie Faith Cranor
Carnegie Mellon University

Pittsburgh, PA{rshay,pgage,sarangak,mmazurek,bur,tvidas,lbauer,nicolasc,lorrie}@cmu.edu
ABSTRACTin an effort to prevent users from choosing passwords that are too
easily guessed [10]. Unfortunately, strict password-composition
policies sometimes lead to user frustration without substantial security benefit [1, 23]. Also, even under a strict policy, users may
fulfill policy requirements in predictable ways [54], such as basing
their passwords on older passwords, names, or words [51, 60], or
reusing passwords across domains [17].
One approach to making passwords more secure is to remove
user choice and have the authentication system generate passwords
randomly. Such system-assigned passwords can be guaranteed tobe sufficiently difficult to guess, although they have been perceived
as difficult to remember and type [37].
A passphrase is a password composed of a sequence of words.
Passphrases are typically much longer than ordinary passwords,
and proponents argue that they are more secure and easier to remember. One NIST publication states that “any long password thatcan be remembered must necessarily be a ‘pass-phrase’ composedof dictionary words” [10]. The use of passphrases has recently
garnered appreciable attention [40, 49], and some institutions haveadopted passphrases as a password policy (e.g., [58]). Despite thisrecent interest in passphrases, however, there is little empirical evidence to support claims of superior usability over passwords.
This paper describes the results of a 1,476-participant study on
the usability of system-assigned passphrases and system-assigned
passwords. The passphrases we study are sequences of three or four
English words drawn at random from a set dictionary and separated
with spaces; our passwords are also system-assigned and are five toeight characters in length. We focus on system-assigned, rather
than user-selected, passphrases and passwords because this allows
us to control for guessability while evaluating usability.
We assigned each participant to one of 11 experimental conditions (three password conditions and eight passphrase conditions).
We measured how quickly and accurately participants could enter
their password or passphrase both shortly after assignment and several days later. We also asked our participants to complete two brief
surveys about their behavior and sentiment.
Our findings suggest that system-assigned passphrases are far
from a panacea for user authentication. Rather than committing
them to memory, users tend to write down or otherwise store both
passwords and passphrases when they are system assigned. When
compared to our password conditions, no passphrase condition significantly outperformed passwords in any of our usability metrics,
indicating that the system-assigned passphrase types we tested fail
to offer substantial usability benefits over system-assigned passwords of equivalent strength. We even find that system-assigned
passphrases might actually be less usable than system-assigned passwords. For instance, users were able to enter their passwords more
quickly and with fewer errors than passphrases of similar strength.

Users tend to create passwords that are easy to guess, while systemassigned passwords tend to be hard to remember. Passphrases,
space-delimited sets of natural language words, have been suggested as both secure and usable for decades. In a 1,476-participant
online study, we explored the usability of 3- and 4-word systemassigned passphrases in comparison to system-assigned passwords
composed of 5 to 6 random characters, and 8-character systemassigned pronounceable passwords. Contrary to expectations, system-assigned passphrases performed similarly to system-assigned
passwords of similar entropy across the usability metrics we examined. Passphrases and passwords were forgotten at similar rates,
led to similar levels of user difficulty and annoyance, and were both
written down by a majority of participants. However, passphrases
took significantly longer for participants to enter, and appear torequire error-correction to counteract entry mistakes. Passphrase
usability did not seem to increase when we shrunk the dictionary
from which words were chosen, reduced the number of words in a
passphrase, or allowed users to change the order of words.

Categories and Subject Descriptors
D.4.6 [Security and Protection]: Authentication; H.1.2 [User/
Machine Systems]: Human factors

Keywords
Passphrases, System-assigned passwords, Usability, Password composition policies

1.

INTRODUCTION

Passwords are the most common form of authentication, usedin both corporate and personal settings. Despite their importance,
however, the best approach to using passwords remains an open
question. Allowing users free rein to create their own passwords
often leads to weak, easily guessed passwords [5, 43, 60], resulting
in security breaches and loss of privacy for victims [12].
Many organizations attempt to address this problem using password-composition policies, which limit the password-creation space

Copyright is held by the author/owner. Permission to make digital or hardcopies of all or part of this work for personal or classroom use is granted
without fee.
Symposium on Usable Privacy and Security (SOUPS) 2012, July 11-13,
2012, Washington, DC, USA.

1

2.2

While our results in general do not strongly favor system-assigned
passwords over system-assigned passphrases or vice versa, we identify several areas for further investigation. For example, larger dictionary sizes do not appear to have a substantial impact on usability
for passphrases. This could be leveraged to make stronger passphrases without much usability cost. We also find that lowercase,pronounceable passwords are an unexpectedly promising strategy
for generating system-assigned passwords.
Researchers have proposed error-corrected passphrase systems [3,25,39]. Our results suggest that sophisticated error correction, suchas mapping the word a user enters to the closest word in the passphrase dictionary, is necessary to make passphrases comparably usable to passwords. Without error correction, many passphrase conditions perform significantly worse than our password conditions.
We next discuss the background and related work for our study
in Section 2, and turn to our methodology in Section 3. We present
results on usability, accuracy, and sentiment in Section 4, and describe our error analysis in Section 5. We consider ecological validity and summarize our findings in Section 6.

2.

System-assigned passwords and passphrases have also been studied, but to a lesser degree. Diceware uses manual die rolls to select words for a passphrase from a specialized 7,776-word dictionary [44]. In contrast, early work by Kurzban proposes a passphrase system using only a 100-word dictionary [34]. In our study,
we test and compare dictionaries from 181 to 1,024 words in size.
Attempts to create memorable system-assigned passwords include generating passwords that are “pronounceable” by speakers
of a natural language. Such passwords generally consist of a series of concatenated syllables from a natural language. Early work
by Gasser on generating pronounceable passwords [19] has been
adopted in modified form as a standard of the U.S. National Institute of Standards and Technology (NIST) [41]. However, thissystem uses the frequencies with which syllables appear in English as part of its generation process, which has been shown toincrease greatly the guessability of generated passwords [36]. Morerecent work has proposed other schemes for randomly generatingpronounceable text [13], though the usability of these schemes has
not been analyzed comprehensively. Jeyaraman and Topkara suggest randomly generating a lower-case password and then automatically creating a mnemonic for that random password in order tomake system-assigned passwords more memorable [26].
Other systems use partially system-assigned passphrases and passwords. For instance, Forget et al. insert randomness into userchosen passwords to increase strength [18]. Lee and Ewe use avariant of this technique to strenghthen user-generated passphrasses
by adding random semantic noise [35].

BACKGROUND AND RELATED WORK

Many years of research have shown that users have difficulty
picking strong passwords. Despite a litany of proposed “password
replacements” over the past decades, no system has proven superior to text passwords when evaluated according to a broad setof criteria [7, 20]. However, the fact remains that users tend tochoose predictable passwords [6,15] and reuse the same passwords
across multiple accounts [17]. Despite warnings to the contrary,
many users also write their passwords down [23, 45, 62]. Some
researchers argue this practice is not inherently bad [47].
While many user-selected passwords are easily guessed, systemassigned passwords can have much stronger security guarantees;
however, they may be difficult for users to remember without writing them down. The literature contains many proposals for making
system-assigned passwords easier for users to recall. In this paper,
we focus on one such technique, passphrases. We consider a passphrase to be a set, sometimes ordered, of natural-language words
separated by spaces. Passphrases thus tend to be longer than typicalpasswords. Hereafter, when we use the term password, we refer toa string of approximately 5–16 characters, usually without spaces,that may not have natural-language meaning. We use secret to referto the general class of password-like strings containing both passwords and passphrases.
In this section, we first provide background on user-selected passwords, focusing on techniques to make them stronger. We thenpresent related work on techniques for crafting usable, systemassigned secrets. We then discuss user-generated passphrases as
encountered “in the wild” and in literature, and, finally, prior studies comparing variations of passwords and passphrases.

2.1

System-Assigned Secrets

2.3

Passphrases

For three decades, academic literature has considered passphrases
as a potentially more memorable and secure alternative to shortpasswords [10, 27, 42], yet their usability vis-à-vis passwords has
not been well studied. Debate about passphrases was recently reignited by the online comic xkcd (Appendix A), which suggested
passphrases as an alternative to complex password policies [40].
This comic has been widely reprinted, including in advice to help
users create strong and memorable passwords [48].
Despite a lack of empirical evidence, passphrases are recommended by some system administrators. For more than five years,
Indiana University has required all new users to create a passphrase,
which they define as containing at least 15 characters split amongat least four words, delimited by a space or non-underscore symbol.1 In a recent blog post, the university pointed to the xkcd comic
as evidence in favor of this policy [58]. Creighton University [49]and UC Santa Cruz2 are among other universities that have citedthe xkcd comic while suggesting passphrases. Other organizations,such as Clemson University,3 have suggested that users create passphrases by envisioning sentence-like passphrases. In one of ourexperimental conditions, we test system-generated passwords constructed grammatically to resemble sentences.
Yan et al. studied mnemonic passwords, in which users selecta phrase but type in only the first character of each word [59].
Universities including Carnegie Mellon4 currently recommend thismnemonic approach on password advice pages. However, Kuo et
al. built a dictionary from Internet sources to crack user-generated

Passwords

Many techniques have been suggested to help users create better passwords. One technique requires that passwords comply witha password-composition policy, such as forcing the inclusion ofdigits in passwords. However, passwords that comply with suchpolicies often remain vulnerable because policy requirements are
sometimes fullfilled in predictable ways [51, 57]. Furthermore, inflexible policies can overly burden users, leading to frustration [23].
Another approach is to allow users to create their own passwords,
but proactively check their strength. Prior work has suggested rejecting unsuitable passwords using predefined lists of weak [5] or
popular passwords [46]. Other work has suggested informing usersof the strength of their proposed password [11, 52, 56].

1

http://kb.iu.edu/data/acpu.html (visited 5/2012)
http://its.ucsc.edu/security/training/password.
html (visited 5/2012)3
http://www.clemson.edu/ccit/about/policies/
strong_passwords.html (visited 5/2012)4
http://www.cs.cmu.edu/~help/security/choosing_passwords.html (visited 5/2012)22

mnemonic passwords, suggesting this technique does not prevent
users from choosing weak secrets [33]. Bonneau and Shutova examined 100,000 user-selected Amazon.com “payphrases,” or globally unique multi-word secrets. They find that lists of popular
movies and books, as well as digrams popular in natural language,
are effective in guessing these payphrases [8]. Both these works
find that user-chosen passphrases may be more easily guessed than
expected, which supports the investigation of system-assigned passphrases as an alternative.

2.3.1

survey, and were asked to recall that secret. Forty-eight hours later,
participants were invited to return, log in using this secret, and complete a second survey. In this section, we give an overview of ourstudy design, experimental conditions, and statistical analysis.

3.1

Comparative Evaluation

A handful of studies has comparatively evaluated various combinations of system-assigned and user-selected passwords, pronounceable passwords, and passphrases, usually with small sample sizes
and student participants. Leonhard and Venkatakrishnan compared
random passwords (six characters), pronounceable passwords oftheir own construction, and three-word passphrases drawn from
Diceware in a study of 29 student participants, finding that participants had difficulty remembering the system-assigned secrets theytested [37]. In two studies using about 50 undergraduate participants each, Keith et al. found that participants with user-selectedpassphrases experienced more login failures due to typographicalerrors, but fewer failures due to memory errors, than users withpasswords [27, 28]. Zviran and Haga studied 103 graduate students in a within-subjects design, where each participant used an 8character system-assigned password, an 8-character user-selectedpassword, and a user-selected passphrase. They found that systemassigned passwords were remembered best if they were pronouncable, but that participants preferred user-selected secrets [61]. Ina within-subjects study of 15 participants, Spector and Ginzberg
compared system-assigned passwords, user-selected passwords, anduser-selected “pass-sentences,” defined as entities with unique semantic meanings; they found that pass-sentences performed best [53].
In contrast to these studies, we use a much larger sample size andtest more conditions. We focus only on system-assigned passwords
and passphrases, allowing us to construct conditions with roughly
similar security guarantees.

2.3.2

Passphrase Entry and Error Correction

Some researchers have proposed using error correction to permit
small errors in passphrase entry, even though this may reduce security. For instance, Bard proposes using the Damerau-Levenshtein
string-edit-distance metric to tolerate up to two spelling errors per
word in a passphrase, as well as accepting the words of a passphrase
in any order, with minimal impact on security [3]. Mehler andSkiena propose a more general password-corrective hash in which
two strings differing by one edit would likely hash to the same key,successfully authenticating a user [39]. Jakobsson and Akavipat
propose several error-correction techniques specific to mobile devices: allowing words in any order, permitting substitution withsynonyms, and using auto-complete features for quicker entry [25].
Matsuura instead proposes a secure visual feedback mechanism tocue users when the password they have typed differs from their expectation [38]. In this paper, we evaluate participants’ login attempts based on the string-edit distance and other methods thatwould have allowed for authentication in the presence of typing
mistakes. We also test one condition in which users are explicitly
told that the words of the passphrase may be entered in any order.

3.

Study Overview

We recruited participants through Amazon’s Mechanical Turk
(MTurk) crowdsourcing service. We compensated them 55 centsfor completing the first part of the study and an additional 70 centsfor completing the second part. We required participants to be atleast 18 years old and not to have participated in a previous study on
passwords conducted by our research group. Since we tested passphrases generated with American English dictionaries, we allowed
only MTurk users who lived in the United States to participate.
In part one of the experiment, we told participants, “imagine thatyour main email service provider has been attacked . . . [and that]
because of the attack, your email service provider is also changing
its password rules. Instead of choosing your own password, onewill be assigned to you.” We informed them that they would use
their secret in a few days to log in to the second part of the study,
and that they should take whatever steps they would normally taketo remember and protect their email password. In prior work on
user-generated passwords, we observed that users created stronger
passwords when presented with this scenario than when they werecreating passwords simply for the purpose of a study [30, 32]. We
did not tell participants not to store their secrets, nor did we otherwise mention secret storage during the first part of the study.
We next assigned participants a secret in one of 11 conditions,described in Section 3.2. After being assigned the secret, participants were required to check a box on the screen to hide the secret
and then enter the secret twice, once as confirmation. They could
uncheck the box to see their assigned secret again, but could not
type while their secret was visible. After successfully entering andconfirming their secret, participants completed a five-minute survey. This survey asked participants about their experience learning
their new secret and about their actual email password. We thenasked participants to enter their system-assigned secret once again.
We refer to this as part one recall throughout the paper. After fiveunsuccessful attempts, participants were told their secret.
Forty-eight hours after completing the first part of the study, participants received an email through MTurk asking them to returnfor part two. To begin part two, participants were asked to log inusing their secret. Participants could click a “Forgot Password”
link to be emailed a link to retrieve their secret. Furthermore, after
five incorrect attempts, we showed participants their secret. Oncethey had logged in, participants completed a survey about how they
had remembered their secret, including whether they had written
it down on paper or stored it electronically. We analyzed data forpart two for only those participants who completed this part within
72 hours of being invited to return, 120 hours after completing partone. Participants who returned after more than 72 hours were still
paid, but their data were excluded. This ensures that all of the participants in our analysis had viewed their secret within five days ofcompleting part one of our study.
Mechanical Turk. As explained, we recruited participants using
MTurk. Although MTurk workers tend to be younger, more educated, and more technical than the general population, they represent a significantly more diverse population than is typically used inlab studies, which often rely on college-student participants [9,24].
Researchers have found that well-designed MTurk studies provide
high-quality user data [4, 16, 21, 31, 55]. Using MTurk allows us
to study a larger volume of participants in a controlled setting thanwould otherwise be possible. We have successfully used MTurk

METHODOLOGY

We conducted a two-part online study of system-assigned authentication secrets, both passwords and passphrases. In the firstpart of the study, participants were assigned a secret, completed a3

3.2.2to collect password data in several prior studies [30, 32, 51]. Adar
has criticized MTurk studies in general, although our use of crowdsourcing to understand human behavior fits his description of an
appropriate use [2].

3.2

We tested eight variations on passphrases. We generated dictionaries (fully specified in Appendix D) for all passphrase conditions using word-frequency data6 with part-of-speech (e.g., noun,verb) tags from the Corpus of Contemporary American English(COCA) [14]. This list ranks the most common words in the 425million-word COCA based on the number of times they appear andtheir diffusion throughout different sources. So that our dictionaries
would contain only well-known words, we chose the N most common words matching particular criteria for each dictionary. For
instance, a dictionary of 181 nouns would contain the 181 mostcommon nouns from COCA. We selected word lists of particularsizes so that different conditions would each have 30 bits of entropy. However, we later discovered that word lists not restricted toa particular part of speech contained duplicate words. For instance,“to” was present on the list as both an infinitive marker and as apreposition. Thus, the actual passphrase entropies in the next three
conditions, intended to be 30 bits, were as low as 29.3 bits.7

Conditions

We assigned participants round-robin to one of 11 experimental
conditions, which are summarized in Table 1. The conditions varied in the type of secret assigned to the participant. Participants
were unable to modify their assigned secret or to obtain a replacement. We focused on system-assigned secrets so that we could
precisely control their entropy (and their guessability), and focus
on their usability.
Three conditions were variants of passwords, and eight werevariants of passphrases, as defined in Section 2. Our password conditions did not use spaces. In the passphrase conditions we requiredthat participants enter words separated by spaces and in the sameorder they were assigned, unless otherwise specified. Secrets in all
11 conditions were case-sensitive.
We designed two of our three password conditions and six of
our eight passphrase conditions to have approximately 30 bits ofentropy so that we could compare system-assigned passwords toequally strong system-assigned passphrases. This entropy value
was chosen because guidelines frequently used in practice [10, 22]
recommend password policies that provide an estimated 30 bits ofentropy. While recent research has suggested that entropy may not
be the best indicator of resilience to attack [30, 57], when all elements from a set occur with equal probability (as is the case with
our system-assigned secrets), entropy maps directly to the probability that an attacker with knowledge of the password-generation
algorithm will succesfully guess a password.

3.2.1• pp-small: Participants were assigned four words, randomly
selected with replacement from a 181-word dictionary.
• pp-med-unorder: Participants were assigned four words,
randomly selected with replacement from a 401-word dictionary. Unlike all other conditions, participants could enter
the words in their passphrase in any order.
• pp-large-3word: Participants were assigned three words, randomly selected with replacement from a 1,024-word dictionary.
The next two conditions are similar to the pp-small condition, except they use larger dictionaries of the most common words in orderto test whether the size of the dictionary has a measurable impact
on usability:

Password conditions

Three of our conditions focused on passwords.

• pp-medium: Participants were assigned four words, randomly selected with replacement from the 401-word dictionary used in the pp-med-unorder condition. These passphrases
present 33.9 bits of entropy.

• pw-length5: Participants were assigned a five-character password, where each character is chosen randomly from a dictionary of 64 characters, including lowercase letters, uppercase letters, digits, and symbols. We removed characters that
could easily be confused with other characters, e.g., both theletter “O” and the digit “0.” A full list of characters in this
dictionary is shown in Appendix D.1. This password spacehas 30 bits of entropy.

• pp-large: Participants were assigned four words, randomly
selected with replacement from the 1,024-word dictionary
used in the pp-large-3word condition. These passphrases
present 39.2 bits of entropy.
We also tested whether passphrases that followed certain part-ofspeech patterns aid memorability. The next three conditions use
passphrases with 30 bits of entropy.

• pw-pronounce: Participants were assigned an eight-character
password likely to be pronounceable by an English speaker.
To generate these passwords, we used an implementation5of an algorithm originally proposed by Gasser [19] and later
adopted as a NIST standard [41]. Prior work has identified aflaw in this scheme: certain passwords are chosen with highprobability since the probability of a syllable occuring in a
password mirrors its relative frequency in English [36]. To
overcome this, we generated the full list of eight-character
pronounceable passwords without duplicates (≈ 1.2 billion)
and assigned each password on this list with equal probability, resulting in 30.2 bits of entropy.

• pp-sentence: Participants were assigned passphrases of theform “noun verb adjective noun,” where nouns, verbs, andadjectives are chosen from separate 181-word dictionaries.
So that it would make sense for the verb to be followed by anoun, the verb dictionary contained only verbs whose entryin The Free Dictionary8 listed at least one transitive definition. Since all nouns but one were singular, we manuallyconjugated all verbs to agree with a singular subject. Although these passphrases were unlikely to make semantic
sense due to the random selection of words, they might resemble English sentences.

• pw-length6: Participants were assigned a six-character password, where characters are chosen as in the pw-length5 condition. The extra character makes passwords in this condition
have 36 bits of entropy. This condition helps determine how
the length of randomly generated passwords affects usability.
5

Passphrase conditions

6
http://www.wordfrequency.info/top5000.asp
5/2012)(visited7
All entropies were calculated using Shannon’s formula on the frequency distribution of unique words [50].
8

http://www.adel.nursat.kz/apg/ (visited 5/2012)4

http://www.thefreedictionary.com/ (visited 5/2012)

Condition name

Entropy(bits)

LengthDictionary
size

Examples5 characters
8 characters
6 characters

64 characters
190 syllables
64 characters@J#8x
tufritvi
R6wy$_*2LxG
vadasabi
cW@.*H

4 words
4 words (any order OK)
3 words
4 nouns
4 nouns (w/ instructions)
4 words (N-V-Adj-N)
4 words
4 words

181 words
401 words
1,024 words
181 nouns
181 nouns
181 words each
401 words
1,024 words

one between high tellremember million state understandown decide somesense child reason paper
phone star record rightend determines red drug
also that research must
because strategy cover us

try there three comehelp any country our
feeling right reflect
death effect girl modelcase area interest situation
plan builds sure powerroom four face after
pull somebody white next

Password Conditions
pw-length5
pw-pronounce
pw-length6

30
30.2
36

Passphrase Conditions
pp-smallpp-med-unorderpp-large-3word
pp-nouns
pp-nouns-instr
pp-sentence
pp-medium
pp-large

29.4
29.3
29.4
30
30
30
33.9
39.2

Table 1: A summary of experimental conditions, with data about their characteristics and example secrets assigned to participants.
• pp-nouns: Participants were assigned four nouns, randomly
sampled with replacement from a dictionary containing the
181 most common nouns.

prise our 30-bit passphrase conditions that use 181-word dictionaries: pp-small, pp-sentence, pp-nouns, and pp-nouns-instr.

4.

• pp-nouns-instr: The condition is identical to pp-nouns, except that we gave the participant specific instructions for memorizing the passphrase. The instructions asked participants to“try to imagine a scene that includes all of the words in yourpassword phrase. This will help you to remember it moreeasily. Research has found that the more bizarre, unusual,and exaggerated you make your scene, the easier it will be toremember. So, take a moment to construct your scene, andthink about it whenever you need to enter your password.”
This specific instruction mimics the example from the xkcd
comic in Appendix A.

3.3

RESULTSIn this section, we present the results of our study. We begin by
discussing participant demographics in Section 4.1. We then lookat drop-out rates per condition in Section 4.2, as higher drop-out
rates may indicate participants are struggling more in those conditions. In Section 4.3, we define how we tracked whether participants stored their assigned secret. We examine how well participants were able to enter their secrets immediately upon assignment
in Section 4.4. Section 4.5 discusses part-one recall rates, and Section 4.6 part-two recall rates. We further investigate usability byexamining user sentiment in Section 4.8. In Section 5, we analyze
the errors participants made, and the degree to which automatederror correction would have helped.

Statistical Testing

4.1

Our statistical tests use a significance level of α = .05. For eachcomparison, we first ran an omnibus test across all conditions. We
used Kruskal-Wallis (indicated KW), an analogue of ANOVA that
does not assume normality, for omnibus tests on quantitative dataand χ2 on categorical. If the omnibus test showed significance, we
performed pairwise tests with Holm-Bonferroni correction (indicated HC). We used Mann-Whitney U (indicated MW) for pairwise
comparisons of quantitative data and Fisher’s Exact Test (indicatedFET) for pairwise categorical comparisons.9
In our pairwise tests, we compared a subset of all possible pairsof conditions. All eight of our 30-bit conditions are compared toeach other. We also compare pp-medium with pp-med-unorder,
because they both use a 401-word dictionary; and pp-large with pplarge-3word since both use 1,024-word dictionaries. We compare
pw-length5 with pw-length6 as the latter uses longer passwords,
but they are otherwise identical. Finally, we compare pp-medium
with pw-length6 to compare a password and a passphrase condition
with higher entropy.
In addition to looking at conditions independently, we sometimes
combine a subset of our password conditions and a subset of our
passphrase conditions to compare larger sample sizes of passwords
and equivalent-entropy passphrases. The combined passwords participants comprise our two 30-bit password conditions, pw-length5
and pw-pronounce. The combined passphrases participants com-

Demographics

2,689 participants began our study in February and early March
2012. 2,294 completed the first part of our study and 1,562 returned for the second part within three days of being sent an email
invitation two days after completing the first part. An additional
88 participants returned for the second part between three and 42days after completing the first part; we do not include them in our
analysis. Of the participants who returned within three days, 1,476
completed the second part of our study. With the exception of our
discussion of drop-out rates in Section 4.2, we focus on these 1,476
participants throughout our analysis. The number of participants ineach condition is shown in Table 2.
Of the 1,468 participants who reported gender, 51.9% reported
being female and 47.6% reported being male. The mean age was
31 years, while the median was 28. The standard deviation was
11.2, and our oldest participant reported being 74 years old.
Of the 1,464 participants who reported their highest academic
degree, 653 reported having at least a bachelor’s degree. Participants were asked whether they had degrees or jobs in “computerscience, computer engineering, information technology, or a related
field.” Of the 1,460 who answered, 263 answered in the affirmative.
We found no statistically significant differences between our conditions in reported gender, age, or background and education.
Because using a keyboard on a mobile phone could impact aparticipant’s ability to enter his or her secret, we examined participants’ user-agent strings. For example, if a user-agent string contains “iPhone,” that is strong evidence that the participant is taking

When using the χ2 test, some cell counts were less than 5, but weensured that Cochran’s conditions were satisfied: no cell had countzero, and more than 80% of cells had counts of at least 5.
95

Started

Finished
part one

Returned

Finished
part twopw-length5
pw-pronounce
combined password

342
342
684

90%
90%
90%

59%
58%
59%

56%
55%
55%

pp-smallpp-nouns
pp-nouns-instr
pp-sentence
combined passphrase

189
340
342
190
1061

84%
81%
87%
79%
83%

59%
59%
59%
54%
58%

59%
55%
56%
50%
55%

pp-med-unorderpp-large-3word
pp-mediumpw-length6pp-large

187
188
190
190
189

85%
88%
84%
83%
82%

58%
61%
57%
58%
54%

57%
55%
53%
55%
53%

total

2689

85%

58%

55%

success on first entry – omnibus χ210 =28.026, p=.002
pw-pronounce (90.4%) pp-nouns (73.4%)
HC FET, p=.001
pp-nouns-instr (71.2%) HC FET, p<.001
combined pw (84.9%) combined pp (74.5%) χ21 =14.233, p<.001
success on first entry (no-storage) – omnibus χ210 =27.021, p=.003
pw-pronounce (92.3%) pp-nouns-instr (67.1%) HC FET, p=.027
attempts needed – omnibus KW χ210 =28.573, p=.001
pw-pronounce (1.14) pp-nouns (1.45)
U=14526, p<.001
pp-nouns-instr (1.50) U=14367.5, p<.001
pp-sentence (1.33)
U=7473, p=.025
combined pw (1.23)
combined pp (1.41)
KW χ21 =14.979, p<.001
attempts needed (no-storage) – omnibus KW χ210 =28.888, p=.001
pw-length5 (1.24)
pw-length6 (2.55)
U=274.5, p=.037
pw-pronounce (1.08) pp-nouns-instr (1.53) U=1350, p=.025

Table 3: Statistically significant results for secret entry immediately upon assignment. Pairwise tests for attempts needed are
Holm-Bonferonni-corrected Mann-Whitney U.

Table 2: The number of participants who began the study ineach condition, and the percentage who continued through thesteps of the study. A participant counts as having returned forthe second part of the study if he or she returned within threedays of being invited. The analysis in this paper focuses on participants who completed the second part of the study.

we do not detect that he or she has pasted or autofilled the secret.
We label these participants, 410 of our 1,476 total (27.8%), as nostorage; other participants we call storage participants. The proportion of no-storage participants is low across conditions; it does
not vary significantly by condition (χ210 =17.351, p=.067), nor is it
significantly different between the combined password participants
and the combined passphrase participants (χ21 =2.444, p=.118).
The no-storage participants are most relevant when evaluating
the memorability of secrets. However, as users can and do store
secrets for their real accounts, the behavior of storage participantsalso provides useful insights. We do not know when storage participants stored their secrets (e.g., before entering them for the firsttime, or after completing part one). Hence, while we divide participants based on whether they eventually stored their secret, we cannot separately analyze them based on when the secret was stored.
In addition to analyzing data for all participants who did not dropout of the study, we conducted similar analyses separately for storage and no-storage participants. In some cases, differences between
conditions that were statistically significant when looking at all participants are no longer significant when looking only at no-storage
participants. However, this may be due in part to the small numberof no-storage participants. We revisit the results of these separate
analyses in the remainder of this section and in Section 5.

the study from an iPhone. Only 25 participants show evidence of
this,10 and there were no more than four per condition.

4.2

Study Dropouts

Of the 2,689 participants who started our study, 2,294 finishedthe first part; 1,562 participants returned within three days of receiving our email invitation to complete the second part of thestudy, and 1,476 of these completed the second part. These statistics, broken down by condition, are shown in Table 2.
The proportion of participants who completed the first part of thestudy varied by condition (χ210 =28.288, p=.002), with participantsin the two 30-bit password conditions, surprisingly, most likely tofinish. Completion rates for the first part ranged from 78.9% for
pp-sentence to 90.4% for pw-pronounce. Significantly more participants finished the first day in pw-pronounce than in pp-nouns(HC FET, p=.020) and pp-sentence (HC FET, p=.011). More also
completed the first day in pw-length5 than in pp-sentence (HC FET,
p=.035). Combined password participants were more likely to finish than combined passphrase participants (χ21 =14.768, p<.001).
The proportion of participants who returned for the second partof the study within five days after completing the first did not varysignificantly by condition (χ210 =6.759, p=.748), and neither did the
proportion of these who finished the second part (χ210 =15.956, p=.101). We also saw no significant difference between combined
password participants and combined passphrase participants for returning within five days (χ21 =3.423, p=.064) or finishing the secondday (χ21 =0.015, p=.901).

4.3

4.4

Assignment

After receiving instructions, our participants were assigned a secret (either a password or a passphrase) and immediately asked toenter it. They could toggle between being able to view the secretand being able to enter it. This was intended to ensure that participants observed and were able to type their secret. We measure
the number of attempts needed to enter the secret successfully, andthe fraction of participants who correctly entered their secret on thefirst try. Significant results of statistical tests are shown in Table 3.
Overall, participants needed an average of 1.3 attempts to enter
their secret and 78.7% entered it successfully on the first try. For
both metrics, there was a significant difference across conditions.
pw-pronounce secrets needed fewer attempts and were more likely
to be entered on the first try than pp-nouns-instr and pp-nouns, andalso needed fewer attempts than pp-sentence. In aggregate, combined password participants needed fewer attempts and were more
successful in entering their secret on the first attempt than combined passphrase participants.
Similar relationships hold for no-storage participants: pw-pronounce needed fewer attempts and was more successful on the first

Storage and No-Storage Participnts

During the second part of the study, we asked participants if theywrote down their secrets, either on paper or electronically, reassuring them that their compensation would not be affected by their
response. We consider a participant not to have stored his or her
secret if the participant affirms not writing down the secret, and10

We searched the user-agent strings for: Android, iPhone, iPod,
iPad, mobile, RIM Tablet, BlackBerry, Opera Mini, Windows
Phone, SymbianOS, Opera Mobi, nook, Windows CE, smartphone,
webOS, BREW.
6success on first entry – omnibus χ210 =34.936, p<.001
pw-length5 (94.8%)
pp-small (81.1%)
HC FET, p=.008
pp-med-unorder (80.2%) HC FET, p=.007
pw-pronounce (94.7%) pp-small (81.1%)
HC FET, p=.009
pp-med-unorder (80.2%) HC FET, p=.007
combined pw (94.7%) combined pp (87.2%)
χ21 =13.867, p<.001

attempt than pp-nouns-instr. No-storage participants also show difference based on password length, with pw-length5 requiring fewer
attempts than pw-length6.

4.5Part-One Recall

success on first entry (no-storage) – omnibus χ210 =26.558, p=.003
pw-pronounce (94.2%) pp-large-3word (64.3%) HC FET, p=.033

We asked participants to recall their secret after completing abrief survey. Participants who could not recall their secret after
five attempts were shown the secret on the screen. The vast majority of participants in each condition succeeded in entering theirsecret within five attempts, ranging from 92.5% in pp-med-unorderto 99.5% in pw-length5 and pw-pronounce. The significant resultsof our statistical tests are in Table 4.
Among those who entered the secret within five attempts, there
is a significant difference in the number of attempts needed; combined password participants took significantly fewer attempts than
combined passphrase participants, but pairwise tests do not reveal
any significant differences.
The proportion of participants who correctly entered their secret on the first try varied significantly across conditions. A larger
proportion in pw-length5 and pw-pronounce entered their password correctly on the first attempt than in pp-small or pp-medunorder. Combined password participants outperformed combined
passphrase participants.
Among no-storage participants, significantly more in pw-pronounce entered their secret correctly on the first try than in pplarge-3word. Looking at participants who entered their secret within
five attempts, we see omnibus significance between conditions, but
pairwise tests reveal no significant differences.
We measured the time between the first and last keystroke on
the first correct entry for participants whom we did not detect pasting or autofilling their secrets, and who entered the secret within
five attempts. Combined passphrase participants had a median timeof 7 seconds, significantly more than combined password participants, with a median of 3 seconds. pw-length5 and pw-pronounce
each performed significantly better than all of the 30-bit passphrase
conditions, and pw-length6 performed better than pp-medium. ppsmall and pp-large-3word performed significantly better than ppnouns, pp-nouns-instr, and pp-sentence; and pp-large-3word also
outperformed pp-large. Looking only at no-storage participants,
pw-pronounce performed better than any other 30-bit condition except pw-length5. pw-length5 and pp-small both outperformed ppnouns, pp-nouns-instr, and pp-sentence.

4.6

attempts needed – omnibus KW χ210 =19.122, p=.039
combined pw (1.08)
combined pp (1.16) KW χ21 =8.066, p=.005
attempts needed (no-storage) – omnibus KW χ210 =20.002, p=.029
secret entry time – omnibus KW χ210 =329.817, p<.001
pw-pronounce (3.1)
pp-small (5.3)
U=3296, p<.001
pp-nouns (7.4)
U=3187.5, p<.001
pp-nouns-instr (7.4)
U=2833, p<.001
pp-sentence (7.7)
U=1310.5, p<.001
pp-large-3word (4.7)
U=3626, p<.001
pp-med-unorder (6.1)
U=2548.5, p<.001
pw-length5 (3.4)
pp-small (5.3)
U=3962, p<.001
pp-nouns (7.4)
U=4126, p<.001
pp-nouns-instr (7.4)
U=3652.5, p<.001
pp-sentence (7.7)
U=1792, p<.001
pp-large-3word (4.7)
U=4223.5, p<.001
pp-med-unorder (6.1)
U=3141.5, p<.001
pw-length6 (4.2)
U=4857.5, p=.039
pw-length6 (4.2)
pp-medium (6.5)
U=1933, p<.001
pp-small (5.3)
pp-nouns (7.4)
U=4717.5, p=.009
pp-nouns-instr (7.4)
U=4328.5, p<.001
pp-sentence (7.7)
U=2212.5, p=.002
pp-large-3word (4.7) pp-nouns (7.4)
U=4106.5, p=.001
pp-nouns-instr (7.4)
U=3809, p<.001
pp-sentence (7.7)
U=1914, p<.001
pp-large (7.4)
U=2101.5, p=.001
pp-med-unorder (6.1) pp-nouns-instr (7.4)
U=4886, p=.015
combined pw (3.1)
combined pp (7.0) KW χ21 =249.884, p<.001
secret entry time (no-storage) – omnibus KW χ210 =130.86, p<0
pw-pronounce (2.6)
pp-small (3.9)
U=792, p=.017
pp-nouns (7.6)
U=200, p<.001
pp-nouns-instr (7.4)
U=226, p<.001
pp-sentence (6.9)
U=98, p<.001
pp-large-3word (4.2)
U=258, p=.008
pp-med-unorder (5.4)
U=144, p<.001
pw-length5 (3.4)
pp-nouns (7.6)
U=389, p<.001
pp-nouns-instr (7.4)
U=425, p<.001
pp-sentence (6.9)
U=189, p<.001
pp-small (3.9)
pp-nouns (7.6)
U=214, p=.002
pp-nouns-instr (7.4)
U=240, p<.001
pp-sentence (6.9)
U=93, p=.001
combined pw (2.8)
combined pp (6.9)
KW χ21 =89.758, p<.001

Part-Two Recall

Forty-eight hours after finishing the first part of our study, we
invited participants to return for the second part. Our analysis includes the participants who returned within 72 hours of being invited and completed both parts of the study. We find that a majority
in each condition wrote down their secrets, and nearly half that did
not store their secret clicked on the “Forgot Password” link.
Upon returning, a participant was asked to recall his or her secret.
Five incorrect entries resulted in the secret being shown on screen.
How participants fared in entering their secrets here is shown in
Table 5. Our statistical tests are shown in Table 6.
Returning participants could click a link to be emailed a link totheir secret. 48.8% of no-storage participants used this feature. The
proportion does not vary significantly by condition (χ210 =11.992, p=.286), nor between combined passphrase participants and combined password participants (χ21 =1.764, p=.184). 210 of our 1,476
participants did not use the email reminder or store their secrets.
Across all conditions, out of the 354 participants who used theemail reminder, 197 (55.6%) made no attempt to recall their secrets
and 87 (24.6%) made only one attempt before using the reminder.

Table 4: Statistically significant results for secret recall after
completing the survey in part one of the study. Times are shownas median seconds. Attempts needed are shown for participants
who succeeded in entering their secret within five attempts.
All pairwise tests for secret-entry time are Holm-Bonferonnicorrected Mann-Whitney U.

We consider a participant to have succeeded in recalling his or
her secret in the second part of our study if he or she entered thesecret within five attempts without needing to be reminded of it.
Overall, 74.7% of our participants were successful, including 48.5%
of no-storage participants and 84.7% of storage participants. There
were no significant differences between conditions, nor between
combined password and combined passphrase participants, in anyof these groups.
For all participants, and for just no-storage participants, we see
no significant difference in how many participants succeed on the7Login infive tries

Login onfirst try

Participants

Login infive tries

Login onfirst try

pw-length5
pw-pronounce
combined password
pp-smallpp-nouns
pp-nouns-instr
pp-sentence
combined passphrase
pp-med-unorderpp-large-3word
pp-mediumpw-length6pp-large
total

success – omnibus χ210 =15.584, p=.112
comb. pw (77.8%)
comb. pp (73.1%)

Storage

Participants

No-storage

46
52
98
26
57
70
27
180
25
28
34
20
25
410

65%
52%
58%
42%
47%
51%
41%
47%
36%
57%
35%
45%
44%
49%

57%
48%
52%
38%
42%
37%
37%
39%
36%
50%
32%
40%
40%
42%

145
135
280
85
131
121
67
404
81
75
67
84
75
1066

86%
83%
85%
86%
84%
83%
87%
85%
80%
85%
81%
92%
85%
85%

81%
77%
79%
76%
76%
78%
76%
77%
70%
75%
73%
82%
73%
77%

χ21 =2.413, p=.120
2
success (no-storage) – omnibus χ10 =11.786, p=.3
comb. pw (58.2%)
comb. pp (47.2%)
χ21 =2.618, p=.1062
success (storage) – omnibus χ10 =6.341, p=.786
comb. pw (84.6%)
comb. pp (84.7%)
χ21 =.011, p=.917
attempts for success. participants – omnibus KW χ210 =4.376, p=.929
comb. pw (1.112)
comb. pp (1.124)
KW χ21 =2.118, p=.146
2
successful first try – omnibus χ10 =4.774, p=.906
successful first try (no-storage) – omnibus χ210 =9.797, p=.458
edit distance – omnibus KW χ210 =12.579, p=.248
edit distance (no storage) – omnibus KW χ210 =10.407, p=.406
deletions – omnibus KW χ210 =36.614, p<.001
pp-nouns (3.02)
U=9660.5, p=.002
pp-nouns-ins (2.72)
U=9078.5, p=.001
pp-sentence (3.78)
U=4754, p=.017
pp-lg-3word (2.67)
U=5536, p=.010
pp-med-unorder (2.67) U=5372, p=.008
pw-pronounce (0.7) pp-nouns-ins (2.72)
U=9327.5, p=.025
pw-length5 (0.17)

Table 5: Successful logins in the second part of the study for nostorage and storage participants. Participants are considered
not to have been successful in five tries if they either entered
their secret unsuccessfully five times or requested to have theirsecret emailed to them.

deletions (no storage) – omnibus KW χ210 =18.324, p=.05
login time – omnibus KW χ210 =36.259, p<.001
pw-pronounce (25) pp-nouns (35)
U=13716.5, p=.018
login time (no-storage) – omnibus KW χ210 =15.79, p=.106
secret entry time – omnibus KW χ210 =204.592, p<.001
pw-length5 (4.0)
pw-length6 (5.5)
U=3931.5, p=.021
pp-small (5.3)
U=4063.5, p=.017
pp-nouns (8.4)
U=4256, p<.001
pp-nouns-ins (8.6)
U=4736, p<.001
pp-sentence (9.0)
U=2109.5, p<.001
pp-lg-3word (5.1)
U=4152.5, p=.016
pp-med-unorder (6.5) U=3193, p<.001
pw-pronounce (3.3) pp-small (5.3)
U=3765, p=.001
pp-nouns (8.4)
U=3894.5, p<.001
pp-nouns-ins (8.6)
U=4363.5, p<.001
pp-sentence (9.0)
U=1899.5, p<.001
pp-lg-3word (5.1)
U=3809, p<.001
pp-med-unorder (6.5) U=3002.5, p<.001
pp-small (5.3)
pp-nouns (8.4)
U=3675, p<.001
pp-nouns-ins (8.6)
U=4078, p=.002
pp-sentence (9.0)
U=1842, p=.003
pp-lg-3word (5.1)
pp-nouns (8.4)
U=4045, p=.002
pp-nouns-ins (8.6)
U=4438.5, p=.013
pp-sentence (9.0)
U=1982, p=.009
pp-large (7.8)
U=2274, p=.013
comb. pw (3.6)
comb. pp (7.9)
KW χ21 =148.919, p<.001

first attempt between conditions.
For each participant on his or her first attempt at secret entry, we
calculated the edit distance between what was entered and the assigned secret. We use the Damerau-Levenshtein edit-distance metric, which is the minimum number of insertions, deletions, substitutions, and adjacent transpositions required to transform one string
into another. The mean edit distance on the first attempt is shown,
per condition, in Table 7. The mean edit distance was less than
one for each of the password conditions, and for passphrases it
ranged between 1.12 for pp-large-3word and 2.96 for pp-nounsinstr. The median for each condition was zero, and the edit distance did not vary significantly between conditions, either for all orjust no-storage participants.
Looking at successful no-storage participants in the pp-med-unorder condition, six of nine entered the password in the same orderas it was assigned. Overall 68 out of 74 participants in pp-med-unorder entered the passphrase in the same order as it was assigned.
Another metric for usability we examined was the use of deletes
during secret entry. A delete may indicate a participant changing his or her mind about a secret while entering it. We counted
each instance of one or more characters being removed from thesecret-entry field as a single delete and recorded the number per
secret-entry attempt for each participant. Deletions per condition
are shown in Table 7. Looking only at participants who succeeded
in entering their secret on the first try in the second part of thestudy, we find the conditions did differ significantly in the number of deletions. The mean for each password condition was lessthan one, while for passphrase conditions it ranged from 1.76 forpp-medium to 3.78 for pp-sentence. pw-length5 had significantly
fewer deletions than any 30-bit passphrase condition except ppsmall, and pw-pronounce had significantly fewer deletions than ppnouns-instr. Looking only at no-storage participants, the omnibus
test shows significance, but pairwise comparisons do not.
Another usability metric is login time, the total time a participant took to enter his or her secret, measured from the participant’s
first arrival at the secret-entry screen until the end of the participant’s last visit to that screen. This includes anything between

secret entry time (no storage) – omnibus KW χ210 =91.124, p<.001
pw-length5 (3.7)
pp-nouns (7.6)
U=457, p<.001
pp-nouns-ins (7.2)
U=687, p<.001
pp-sentence (6.0)
U=230, p=.007
pw-pronounce (2.7) pp-nouns (7.6)
U=380, p<.001
pp-nouns-ins (7.2)
U=577, p<.001
pp-sentence (6.0)
U=186, p<.001
pp-med-unorder (5.1) U=270, p=.009
pp-small (4.2)
pp-nouns (7.6)
U=188, p<.001
pp-nouns-ins (7.2)
U=327, p=.005
pp-sentence (6.0)
U=109, p=.031
pp-lg-3word (4.1)
pp-nouns (7.6)
U=319, p=.008
comb. pw (3.1)
comb. pp (6.8)
KW χ21 =55.932, p<.001

Table 6: Selected results for secret recall after returning forthe second part of the study. Success indicates the percentageof participants who entered the secret in five attempts without
an email reminder. Times are shown as median seconds. Deletions are for participants who correctly entered their secret on
the first try. All pairwise tests are Holm-Bonferonni-corrected
Mann-Whitney U.

8Med. login time (s)Mean deletions

Mean edit dist.

27.5
25.0
26.0
26.0
35.0
34.0
34.0
33.0
36.0
27.0
35.0
24.0
34.5
31.0

Med. entry time (s)Med. login time (s)

4.0
3.3
3.6
5.3
8.4
8.6
9.0
7.9
6.5
5.1
6.6
5.5
7.8
6.0

Mean deletions

Med. entry time (s)pw-length5
5.0
pw-pronounce
8.0
combined password 6.5
pp-small18.3
pp-nouns
24.2
pp-nouns-instr
24.7
pp-sentence
25.5
combined passphrase 23.4
pp-med-u
21.3
pp-large-3word
18.4
pp-medium21.2
pw-length66.0pp-large
24.1
total
17.2

annoying – omnibus χ210 =30.116, p=.001
pw-length6 (61.5%)
pp-medium (33.7%)

No-storage

Mean edit dist.

Mean length (chars)

All participants

0.2
0.7
0.4
2.0
3.0
2.7
3.8
2.8
2.7
2.7
1.8
0.4
3.5
2.0

0.9
0.9
0.9
1.7
2.5
3.0
2.4
2.5
2.5
1.1
2.3
0.8
2.5
1.9

3.7
2.7
3.1
4.2
7.6
7.2
6.0
6.8
5.1
4.1
6.6
5.1
7.0
5.6

32.5
26.0
27.5
31.0
35.0
31.0
31.0
32.0
28.5
23.5
44.5
44.5
35.0
32.0

0.2
0.8
0.5
0.9
3.9
2.3
5.6
3.2
2.6
1.7
3.3
0.2
3.8
2.3

1.5
1.1
1.3
4.4
3.2
4.0
2.1
3.5
1.9
2.5
5.4
1.6
3.7
2.9

fun – omnibus χ210 =43.433, p<.001
pp-nouns-instr (25.7%)
pp-small (9.0%)
HC FET, p=.014
pp-large-3word (8.7%) HC FET, p=.012
pw-length5 (9.4%)
HC FET, p=.001
pp-medium (20.8%)
pw-length6 (4.8%)
HC FET, p=.018

Table 8: Statistically significant results for user sentiment.
of the above. Storage rate was not significantly different between
conditions (χ210 =17.351, p=.067). This result matches Zviran and
Haga’s [62]; they found, surprisingly, that “difficulty recalling apassword or writing it down is not related to password’s length.”
We asked participants “If you wrote down or stored your password for this study, how is it protected (choose all that apply)?” Of
our 1,066 storage participants, 21.9% did nothing to protect theirpasswords. 26.7% said they stored it on a computer or device used
only by themselves, the most popular response. 24.5% stored thepassword in a room or office used only by that participant.
We also asked our participants about their real email passwords.
308 indicated referring to a written-down or stored password whenlogging in with their real email password, and 1,168 did not. We
also asked if they had ever stored their real email password. 768
participants indicated never writing down their real email password, while 373 did so on paper and 430 electronically. This 52.6%
of participants who did not store their real passwords is a significantly larger proportion than the 32.8% who indicated not storing
their study secret (χ21 =114.287, p<.001).

Table 7: Length, entry time, login time, number of deletions
and edit distance for each condition. Entry time is the mediansecret-entry time, first to last keystroke, in the second part ofour study, for participants who did not paste or autofill their secrets, and who entered them within five attempts. Login time is
the median time between a participant first being shown thesecond-part recall screen and leaving that page for the final
time. Deletions are counted for those participants who succeeded at their first secret-entry attempt in the second part ofthe study. Edit distance is computed between the actual secret
and what was entered, for all participants in their first attemptin the second part of the study.

4.8

visits to that screen, like using the secret-reminder feature. Logintimes by condition are shown in Table 7. Login time varies significantly across conditions, with pw-pronounce performing significantly better than pp-nouns. Login time does not vary significantly
by condition for no-storage participants.
We next examined the time between the first and the last keystrokeon the initial correct entry for participants who neither pasted nor
autofilled their secrets in the second part of our study, and entered their secret within five attempts. pw-length5 and pw-pronounce each performed significantly better than most of the passphrase conditions, and pp-small and pp-large-3word outperformed
most other passphrase conditions; these and other comparisons areshown in Table 6, and login times are shown in Table 7. Similar relationships hold for no-storage participants: pw-length5, pwpronounce, and pp-small outperform pp-nouns, pp-nouns-instr, andpp-sentence; pw-pronounce also performed better than pp-med-unorder, and pp-large-3word better than pp-nouns.

4.7

HC FET, p=.003

difficult – omnibus χ210 =66.583, p<.001
pp-large (18.0%)
pp-large-3word (1.9%) HC FET, p=.002
all other 30-bit (14.4 - 25.0%)
HC FET, p<.023
pw-length6 (44.2%)
pp-medium (16.8%) HC FET, p=.001
pw-length5 (22.0%) HC FET, p=.003

User Sentiment

In the first part of the study, we asked participants to indicate
their agreement, from “strongly disagree” to “strongly agree,” withthe statements “learning my password was [fun/difficult/annoying].”
We classify participants as either agreeing (“agree” or “strongly
agree”) or not agreeing with each statement. An overview of results
is shown in Figure 1, with detailed statistical results in Table 8.
We see a significant difference in annoyance, fun, and difficulty
memorizing across conditions. Pairwise tests show that pw-length6
was substantially more annoying, difficult to learn, and less fun
than pp-medium, and was also more difficult than pw-length5. pplarge, and all 30-bit conditions, were more difficult to memorize
than pp-large-3word. Finally, pp-nouns-instr was more fun by awide margin than pp-small, pp-large-3word, or pw-length5.
Comparing our combined password and combined passphrase
participants, we see no significant difference in agreement that memorizing the secrets was annoying (χ21 =0.219, p=.639), difficult (χ21
=0.022, p=.882), or fun (χ21 =1.65, p=.199).
Looking only at no-storage participants, we see no significant
differences in annoyance (χ210 =10.952, p=.361), and omnibus difference in difficulty (χ210 =23.317, p=.01) and fun (χ210 =23.998, p=0.008) but no pairwise significance for either.

Storage Behavior

We examined how participants stored and protected the secrets
used in this study, as well as how they reported storing their realemail passwords. As indicated in Section 4.3, we assume a participant has stored his or her secret unless that participant explicitly
states he or she has not written it down or otherwise stored it, andhas not pasted or autofilled the secret. 72% of participants stored
their secret. Of these, 48.3% indicated writing their secret downon paper and 43.6% reported storing it electronically; 23% pasted
their secret. A single participant may have done more than one

5.

ERROR ANALYSIS RESULTS

Examining factors that lead to user error can help us understandwhy passphrases were less successful than we anticipated, and caninform research on improving their performance. In addition, passphrases (and to a lesser extent pronounceable passwords) offer sev9

Annoyance

Difficult

Fun

☆♦
☆

pw-length5
pw-pronounceablepp-smallpp-sentence☆
☆
☆

pp-medium-unordered☆

pp-nouns
pp-nouns-instruct

pp-large-3word

pp-large
0%25%Strongly agree

50%

75%

100%

Agree

★
☆
♦
♢

♦
♢

☆

pw-length6

☆

★

★

pp-medium

☆

0%

25%

50%

Neutral75%

Disagree

100%

0%

25%

50%

75%

100%

Strongly Disagree

Figure 1: Likert response data on annoyance, difficulty, and fun. Data significance is shown in symbol groups; a condition marked
with a solid symbol performs significantly better than conditions marked with the same symbol drawn as an outline.
eral opportunities for automatic error detection and correction, which
may be able to improve usability without loss of entropy.
The overall results of our error analysis are shown in Table 9.
This table displays the percentage of subjects who correctly entered
their secret, both with no correction and adjusted for the use ofdifferent error-correcting mechanisms, as discussed below.

5.1

small enough that they do not cause changes in the significancerelationships among conditions.
Looking only at no-storage participants, however, we do see another difference. As reported in Section 4.5, during part-one recall
significantly more participants in pw-pronounce entered their secretcorrectly on the first try than in pp-large-3word. This is still true
with the correction, and, in addition, pw-pronounce also performed
better than pp-small (HC FET, p=.018). Part-two recall continues
to have no significant difference (χ210 =7.252, p=0.701).

LengthWe hypothesized that longer secrets (in characters) lead to moretyping errors. Table 7 shows the mean length of secrets, per condition. For passphrase conditions, in which secrets are generally
longer than in our password conditions, we find that longer passphrases reduce the likelihood of authentication success at assignment, but not thereafter. For all passphrase participants, we usedlogistic regression on passphrase length with an outcome of firstattempt success at assignment, and found a significant relationship(p=.003). The shortest passphrase condition, pp-small, had a mean
length of 18.3 characters and a first-try success rate of 81% at assignment. By contrast, the longest condition, pp-sentence, had amean of 25.5 characters and a first-try success rate of only 76%.
The same analysis for first-try accuracy for recall in part one(excluding participants who pasted or auto-filled their passphrase)and part two (excluding storage participants) found that length was
not a significant factor in either case (p>.375). We also found no
relationship between length and overall rate of successful login forpart two (within five attempts, without using the reminder, p=.406).

5.2

5.3

Off-by-One Errors

It is possible to construct a passphrase dictionary in which noword is within one edit of another. With such a dictionary, users
who enter a word that is within one edit of the correct word intheir passphrase can be authenticated successfully, with no loss of
security. We did not attempt to create such a dictionary, but we didmeasure how many of our passphrase participants submitted entries
with each word within one edit of the correct entry, as shown in
Table 9. This correction is case-insensitive.
Applying this correction narrows the gap between passwords andpassphrases. For first-attempt success during day-one recall, we
still see an omnibus significant difference among conditions (χ210=18.463, p=.048), but the pairwise differences showing greater accuracy for passwords than passphrases (see Section 4.5) disappear.
Looking at only no-storage participants, however, pw-pronounce
remains more successful than pp-large-3word (HC FET, p=.033).
Recall attempts on the second day continue to show no significant variation among conditions, including when we examine only
no-storage participants.

Ignoring Spaces and Capitalization

We required participants to enter their secret exactly as we showed
it to them, including spaces and capitalization. In general, however,passphrase dictionaries can be designed to be case-insensitive andunambiguous, even when spaces are removed. In addition, our pwpronounce condition did not include uppercase. In such cases, removing spaces and ignoring case when checking input passwords
can potentially improve usability with no cost to security. We examine how our passphrase and pw-pronounce participants would
have performed had we ignored spaces and capitalization. We findthat while error correction provides a small benefit, it does notcause passphrase performance to improve relative to passwords.
As shown in Table 9, ignoring case and spaces improves firstattempt accuracy for every passphrase condition as well as pw-pronounce, but has minimal impact on overall success within five attempts, on either part one or part two. These improvements are

5.4

Closest Dictionary Word Correction

Our security analysis assumes the attacker knows the dictionaries used to generate passphrases, and would therefore never guessa non-dictionary word. As a result, if a passphrase participant enters a word not included in the dictionary for his or her condition,
we can replace the entered word with the closest (by edit distance)
dictionary word, with no loss of security. Ties are arbitrarily but
consistently broken using word order within the dictionary.
This correction can be applied only in our passphrase conditions
and is case-insensitive; we did not implement it for our participants,
but we examine how it would have affected their passphrase entries.
Results of our analysis are shown in Table 9. We find that thismechanism, like off-by-one correction, helps passphrase users some10

93 98
90 95
94 98
96
65 85
69
73
71 97
76
74
74
72 96
80
80
88
84 95
88 96
88 96
80 93
83
83
85 100
93
64 82
68 86
68 89
- 86
72 80
76
80 84

we categorize 29.4% (63) as completely wrong — these entries
had no apparent similarity to the participant’s assigned passphrase.
Most of these entries appear to be standard user-selected passwords,
likely associated with another account belonging to the participant.
We next classify the errors made by the 151 passphrase participants whose entries did relate to their assigned passphrases (related
errors). We believe that more than half of these errors could be
mitigated relatively simply, with a passphrase scheme designed totolerate common human errors.
In Section 5.2, we discuss ignoring spacing and capitalization
errors. Among related errors, 25.2% (38) involved mistakes onlyin spacing, and another 7.9% (12) only in capitalization. Another
15.2% (23) of related errors appear to be simple typos; 21 of these
could be corrected via the approach outlined in Section 5.3.
An additional 9.3% (14) of related errors involved the correct
words appearing in the wrong order. Tolerating order variation reduces entropy, but our pp-med-unorder condition results suggest
this entropy can be replaced by using a larger dictionary without
sacrificing usability, at least up to a point. Another approach is
to allow a user to succeed if he or she enters all but one of thewords in the passphrase (e.g., [25]); as with variation in order, thelost entropy could be made up by using a larger dictionary. This
technique, however, would have mitigated only 2.6% (4) of related
errors.
Other errors made by our participants would require more sophisticated detection techniques. We identified 10 errors containing synonym substitutions. These synonyms, however, were almost
all only loosely connected concepts — war to army, position toproximity, friend to family, political to president — suggesting thatbuilding a sufficiently synonym-tolerant dictionary without sacrificing security would be difficult More common than synonyms
were rough sound-alikes, typically words with the same first letterand at least one similar vowel sound. Examples include assort foraccording, over for officer, study for story, and meeting for morning. We considered 17.2% of related errors (26) to include at least
one error of this type. Although it is easy to understand how users
make these errors, it is difficult to imagine a dictionary that could
account for them without dramatically sacrificing security.
We hypothesized that participants would commonly confuse partsof speech — for example, typing walk instead of walked — orwould substitute synonyms. While we did see occurrences of theseerrors, they were very limited.
Separating storage and no-storage participants reveals interesting trends. Among no-storage participants, 75.0% (69) of day-two,
first-attempt recall errors were related; among storage participants,
only 67.2% (82). This suggests that many storage participants either did not attempt to reference their stored passphrase, or elsereferenced the wrong stored secret. Among related errors, storage
participants were more likely than non-storage participants to makespacing, capitalization, and typing errors, as might be expected
when copying the passphrase from storage to the entry screen. Whileno-storage participants also made many of these simple errors, they
were more likely than storage participants to make order, part ofspeech, and synonym errors; this is consistent with trying to recall
the passphrase from memory. The frequency of sound-alike errors
was similar for both storage and no-storage participants.

In five tries

On first try

83 99
82 99
83 99
84
79 95
85 96
82 96
83 96
77 100
80
80
81
75 99
81
78
79
79 97
82
84 98
84 98
74 98
81 99
77
77
78 98
83 99
82 99
82 99
83 99
89
89
89
78 98
81 99
83 99
85 99

In five tries

99
99
99
95
96
96
97
97
99
98
97
98
99
99
93
94
95
94
92
93

No-storage
PART
PART
O NE
T WO
On first try

On first try

95
90
95
96
81
83
85
88
85
88
87
87
81
87
86
91
89
90
91
91
88
89
91
91
89
93
90
91
84
87
85
80
81
83
85

In five tries

In five tries

Condition (correction)
pw-length5
pw-length6
pw-pronounce– Ignore space+case
pp-small– Ignore space+case– Closest dict. word
– Edit distance
pp-medium– Ignore space+case– Closest dict. word
– Edit distancepp-large– Ignore space+case– Closest dict. word
– Edit distance
pp-nouns– Ignore space+case– Closest dict. word
– Edit distance
pp-nouns-instr– Ignore space+case– Closest dict. word
– Edit distancepp-sentence– Ignore space+case– Closest dict. word
– Edit distancepp-large-3word– Ignore space+case– Closest dict. word
– Edit distance
pp-med-unorder– Ignore space+case– Closest dict. word
– Edit distanceOn first try

All participants
PART
PART
O NE
T WO

72 98
60 100
75 98
62 92
65
65
65
62 10068 100
70 98
75
75
75
61 97
73 99
66
67
70 93
78 96
74 96
74 96
79 96
82
82
82
80 96
88
88
-

Table 9: The percentage of participants in each condition who
successfully recalled their secret in one and in five attempts inthe first and second parts of the study. This includes participants who requested to have their secret emailed to them. The
first row for each condition shows uncorrected data. Subsequent rows show the impact of correction for cases where correction would have allowed more users to log in; in many cases,
correction did not help.

what but does not outperform uncorrected passwords. As with offby-one correction, the only change we see in statistical relationships among conditions is for recall on day one; there remains anomnibus difference among conditions (χ210 =23.808, p=.008), butthere are no longer any pairwise differences. For no-storage participants, likewise, we see omnibus significance in part one recall(χ210 =21.517, p=.018), but no pairwise significance.

5.5

Qualitative Error Analysis

6.

We examined passphrase participants’ recall errors in additional
depth by manually categorizing error types. In this section, we focus on participants’ first recall attempt during part two of the study.
Across all passphrase conditions, 214 participants (18.2% of those
who completed day two) made errors during this attempt. Of these,

DISCUSSION

We compared the usability of eight types of system-assigned
passphrases and three types of system-assigned passwords using
a number of metrics, including memorability, time to authenticate,
rate of user errors, tendency of users to store their secrets, and user
sentiment. In this section we discuss ecological validity, and sum11

marize our high-level results about passphrases and some surprising findings about pronounceable passwords.

6.1

and pp-nouns resulted in slightly longer passphrases than pp-small(which contained short words such as: the, be, and, a, to), but otherwise performed similarly. We had also predicted that pp-nounsinstr would perform better than pp-nouns because the instructions
would help people visualize and remember their passphrases. However, we found only small, statistically insignificant, differences between these two conditions. Different instructions, such as guiding
users to visualize their passphrase or construct a scene or story using words from their passphrase, could prove more effective.
Word order. Requiring users to enter the words in a passphrase
in a prescribed order increases the entropy of the passphrase. We
explored whether this entropy increase came at the expense of usability. The pp-med-unorder condition was the same as the ppmedium condition, except it did not impose order requirements.
Contrary to expectations, we did not find any significant differences between these conditions, nor between the pp-med-unorderand pp-small conditions, which used different dictionaries to maintain equivalent entropy. However, we found that participants did reorder their passphrases. 8.1% of participants in the pp-med-unordercondition took advantage of the ability to reorder their passphrases
when entering them in the second part of the study (33.3% if we
consider only no-storage participants). In passphrase conditions
that did not permit reordering, we found that 9.3% of passphrase
entry errors were due to entering words in the wrong order. Thus,
it appears that relaxing the order requirement may provide small
usability gains, but these gains were not significant in our study.
Error correction. Our analysis of the errors users made whenentering their passphrases suggests that usability could be improvedby selecting dictionaries that allow automatic correction of entry errors while maintaining a desired entropy. Even with the dictionaries we used, capitalization errors could be corrected without loss ofentropy, because no two words differed only in capitalization. For
the ordered passphrase conditions, missing spaces could also becorrected without loss of entropy. And, if every word in a dictionary had an edit distance of at least three from every other word in
the dictionary, then it would be possible to correct many common
typos as well as some errors where users misremember a word intheir passphrase as another word that sounds similar.
Pronounceable passwords. We designed our experiment to compare system-assigned passphrases and passwords. Based on thenegative sentiment and high storage rate associated with systemassigned passwords in a previous study [29], we were initially concerned that random-character system-assigned passwords might notprovide a fair comparison. We looked for algorithms to generate
relatively short, but high-entropy, system-assigned passwords that
had characteristics that might make them more memorable. We
found repeated mention in the literature of Gasser’s algorithm forgenerating pronounceable random passwords [19]. However, members of our research group found passwords produced by this algorithm neither easy to pronounce nor easier to remember.
We were hence surprised to discover pw-pronounce performed
very well — significantly better than some other conditions — in
accuracy and entry-speed during part-one recall. One advantage ofthe pw-pronounce condition seems to be that the passwords in thiscondition include character combinations that, even if marginallypronounceable, are all lowercase and relatively easy to type.

Ecological Validity

Several factors may affect the ecological validity and generalizability of our results. First, passphrases are unfamiliar to mostusers, whose behaviors and reactions might change given more experience. We assigned one, system-selected passphrase; we would
expect different behavior from users with self-selected passphrases,
and from those who keep track of multiple passphrases. Our memorability results are limited because so many participants stored theirsecrets. In addition, as mentioned in Section 3.1, our participants
are also younger and more educated than the population at large.
Ecological validity in many password studies is limited by thefact that participants are aware they are using passwords for a study,
rather than for accounts they value or expect to use long-term. We
attempt to mitigate this by comparing only conditions that shouldbe affected equally by this issue. In addition, we use a role-playing
scenario, which our prior work suggests can motivate users to taketheir passwords more seriously than a survey scenario [32].

6.2

Summary of Results

System-assigned secrets. The use of system-assigned secrets
eliminates the problem of users selecting low-entropy secrets, aswell as the problem of users selecting a secret that they use for another account. We found that, in general, our system-assigned passwords and passphrases were not well-liked by users, and that thevast majority of users opted to store them. These results are consistent with the results of a previous study in which we found that 60%
of participants stored their system-assigned 4-digit PINs [29]. In
contrast, a study with similar methodology found that user-selectedpasswords were stored between 17% and 50% of the time, depending on condition [32]. Despite their unpopularity with users,
system-assigned secrets may serve a role in situations where highentropy is a priority, and secure password storage poses minimal
security risk and user inconvenience.
Dictionary choice. We experimented with passphrases composed of words drawn from a variety of dictionaries. All of thedictionaries we used were generated from the most frequently used
words in COCA. We found that whether we used a dictionary of
the top 181 words, top 401 words, or top 1,024 words made little
difference for the metrics we studied. Using the top 181 nouns, ora sentence-like combination of the top 181 nouns, verbs, and adjectives also made little difference. This suggests that we may beable to create high-entropy passphrases while selecting dictionaries
that meet certain properties, for example, dictionaries of words thatare all at least three edits apart, which would allow the use of error
correction to improve usability without sacrificing security.
Passphrase length. We found few differences between 3-wordand 4-word passphrases. 3-word passphrases were shorter than 4word passphrases drawn from the same dictionary, and therefore
faster to type and resulted in fewer typing errors. But although 3word passphrases were perceived as significantly less difficult tolearn than several other conditions, the number of attempts needed
to authenticate did not vary significantly. In addition, 3-word and4-word passphrases with equivalent entropy are approximately thesame length and result in similar typing speeds and error rates. For
the conditions we studied, the number of characters in a passphrase
appears to affect usability more than does the number of words.
Memory aids. We hypothesized that passphrases would be easier to remember if they were sentence-like, and that passphrases
composed of nouns would be easier to visualize than passphrases
composed of random words. However, we found that pp-sentence

7.

ACKNOWLEDGMENTS

We thank Ashwini Rao for her feedback on study design. This
research was supported in part by NSF grants DGE-0903659, CNS1116776, and CCF-0424422; by CyLab at Carnegie Mellon under grants DAAD19-02-1-0389 and W911NF-09-1-0273 from the
Army Research Office; and by a gift from Microsoft Research.
12

8.

REFERENCES[22] InCommon Federation. Identity assurance profiles bronze
and silver v1.1, 2011.
[23] P. Inglesant and M. A. Sasse. The true cost of unusablepassword policies: password use in the wild. In Proc. ACM
CHI, 2010.
[24] P. G. Ipeirotis. Demographics of Mechanical Turk. Technical
Report CeDER-10-01, New York University, 2010.
[25] M. Jakobsson and R. Akavipat. Rethinking passwords toadapt to constrained keyboards. Proc. IEEE MoST, 2012.
[26] S. Jeyaraman and U. Topkara. Have the cake and eat it
too—Infusing usability into text-password based
authentication systems. In Proc. ACSAC, 2005.
[27] M. Keith, B. Shao, and P. Steinbart. A behavioral analysis ofpassphrase design and effectiveness. Journal of the
Association for Information Systems, 10(2):63–89, 2009.
[28] M. Keith, B. Shao, and P. J. Steinbart. The usability of
passphrases for authentication: An empirical field study. Int.
J. Human-Comp. Studies, 65(1):17–28, 2007.
[29] P. G. Kelley, S. Komanduri, M. L. Mazurek, R. Shay,
T. Vidas, L. Bauer, N. Christin, and L. F. Cranor. The impactof length and mathematical operators on the usability andsecurity of system-assigned one-time PINs, 2012. Under
review.
[30] P. G. Kelley, S. Komanduri, M. L. Mazurek, R. Shay,
T. Vidas, L. Bauer, N. Christin, L. F. Cranor, and J. Lopez.
Guess again (and again and again): Measuring password
strength by simulating password-cracking algorithms. In
Proc. IEEE Symp. Security & Privacy, 2012.
[31] A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies
with Mechanical Turk. In Proc. ACM CHI, 2008.
[32] S. Komanduri, R. Shay, P. G. Kelley, M. L. Mazurek,
L. Bauer, N. Christin, L. F. Cranor, and S. Egelman. Of
passwords and people: measuring the effect ofpassword-composition policies. In Proc. ACM CHI, 2011.
[33] C. Kuo, S. Romanosky, and L. F. Cranor. Human selection of
mnemonic phrase-based passwords. In Proc. SOUPS, 2006.
[34] S. A. Kurzban. Easily remembered passphrases: a better
approach. SIGSAC Rev., 3(2-4):10–21, Sept. 1985.
[35] K.-W. Lee and H.-T. Ewe. Passphrase with semantic noises
and a proof on its higher information rate. In Proc. CISW,
2007.
[36] M. Leonhard and V. Venkatakrishnan. A new attack on
random pronounceable password generators. In Proc. IEEE
EIT, 2007.
[37] M. D. Leonhard and V. N. Venkatakrishnan. A comparative
study of three random password generators. In Proc. IEEE
EIT, 2007.
[38] K. Matsuura. Echo back in implementation of passphrase
authentication. 2001.
[39] A. Mehler and S. Skiena. Improving usability throughpassword-corrective hashing. In Proc. SPIRE, 2006.
[40] R. Munroe. xkcd: Password strength.
https://www.xkcd.com/936/, 2012.
[41] NIST. Federal information processing standards publication
181: Automated password generator (APG). Technicalreport, 1993.
[42] S. N. Porter. A password extension for improved humanfactors. Computers and Security, 1(1), 1982.
[43] R. W. Proctor, M.-C. Lien, K.-P. L. Vu, E. E. Schultz, and
G. Salvendy. Improving computer security for authenticationof users: Influence of proactive password restrictions.

[1] A. Adams, M. A. Sasse, and P. Lunt. Making passwords
secure and usable. In Proc. HCI, 1997.
[2] E. Adar. Why I hate Mechanical Turk research (andworkshops). In Proc. CHI Workshop on Crowdsourcing andHuman Computation, 2011.
[3] G. V. Bard. Spelling-error tolerant, order-independentpass-phrases via the Damerau-Levenshtein string-edit
distance metric. In Proc. ACSW, pages 117–124, 2007.
[4] A. J. Berinsky, G. A. Huber, and G. S. Len. Using
Mechanical Turk as a subject recruitment tool forexperimental research. Political Analysis, 2011.
[5] M. Bishop and D. V. Klein. Improving system security via
proactive password checking. Computers & Security,14(3):233–249, 1995.
[6] J. Bonneau. The science of guessing: analyzing ananonymized corpus of 70 million passwords. In Proc. IEEE
Symposium on Security and Privacy, 2012.
[7] J. Bonneau, C. Herley, P. C. van Oorschot, and F. Stajano.
The quest to replace passwords: A framework forcomparative evaluation of Web authentication schemes. In
Proc. IEEE Symposium on Security and Privacy, 2012.
[8] J. Bonneau and E. Shutova. Linguistic properties ofmulti-word passphrases. In Proc. USEC, 2012.
[9] M. Buhrmester, T. Kwang, and S. D. Gosling. Amazon’s
Mechanical Turk: A new source of inexpensive, yethigh-quality, data? Persp. Psych. Sci., 6(1):3–5, 2011.
[10] W. E. Burr, D. F. Dodson, and W. T. Polk. Electronic
authentication guideline. Technical report, NIST, 2006.
[11] C. Castelluccia, M. Durmuth, and D. Perito. Adaptive
password-strength meters from Markov models. In Proc.
NDSS, 2012.
[12] D. Craddock. Hey! My friend’s account was hacked!
http://windowsteamblog.com/windows_live/b/
windowslive/archive/2011/07/14/
hey-my-friend-s-account-was-hacked.aspx, 2011.

[13] H. Crawford and J. Aycock. Kwyjibo: automatic domainname generation. Softw: Pract. Exper., 38(14):1561–1567,
2008.
[14] M. Davies. The corpus of contemporary American English:
425 million words, 1990–present. Available online at
http://corpus.byu.edu/coca/, 2008.
[15] M. Dell’Amico, P. Michiardi, and Y. Roudier. Password
strength: An empirical analysis. In Proc. INFOCOM, 2010.
[16] J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor.
Are your participants gaming the system? Screening
Mechanical Turk workers. In Proc. ACM CHI, 2010.
[17] D. Florêncio and C. Herley. A large-scale study of web
password habits. In Proc. WWW, 2007.
[18] A. Forget, S. Chiasson, P. C. van Oorschot, and R. Biddle.
Improving text passwords through persuasion. In Proc.
SOUPS, 2008.
[19] M. Gasser. A random word generator for pronouncablepasswords. Technical Report ESD-TR-75-97, The MITRE
Corporation, 1975.
[20] C. Herley and P. Van Oorschot. A research agenda
acknowledging the persistence of passwords. IEEE Security
and Privacy, 10(1):28–36, 2012.
[21] J. J. Horton, D. G. Rand, and R. J. Zeckhauser. The online
laboratory: Conducting experiments in a real labor market.
Experimental Economics, 2010.
13

Behavior Res. Methods, Instruments, & Computers,
34(2):163–169, 2002.
[44] A. G. Reinhold. Diceware.
http://world.std.com/~reinhold/diceware.html,
1995–2011.
[45] S. Riley. Password security: What users know and what they
actually do. Usability News, 8(1), Feb. 2006.
[46] S. Schechter, C. Herley, and M. Mitzenmacher. Popularity is
everything: a new approach to protecting passwords fromstatistical-guessing attacks. In Proc. HotSec, 2010.
[47] B. Schneier. Schneier on security blog.
http://www.schneier.com/blog/archives/2005/
06/write_down_your.html, 2005.

[48] S. Schoen, M. Hofmann, and R. Reynolds. Defending
privacy at the U.S. border: A guide for travelers carrying
digital devices. Electronic Frontier Foundation, 2011.
[49] A. Schumacher. Security @ CU—Making secure passwords,
2011.
[50] C. E. Shannon. A mathematical theory of communication.
Bell Syst. Tech. J., 27:379–423, 1949.
[51] R. Shay, S. Komanduri, P. G. Kelley, P. G. Leon, M. L.
Mazurek, L. Bauer, N. Christin, and L. F. Cranor.
Encountering stronger password requirements: user attitudes
and behaviors. In Proc. SOUPS, 2010.
[52] A. Sotirakopoulos, I. Muslukov, K. Beznosov, C. Herley, andS. Egelman. Motivating users to choose better passwords
through peer pressure. SOUPS Poster, 2011.
[53] Y. Spector and J. Ginzberg. Pass-sentence—a new approachto computer code. Comput. Secur., 13(2):145–160, Apr.
1994.
[54] J. M. Stanton, K. R. Stam, P. Mastrangelo, and J. Jolton.
Analysis of end user security behaviors. Comp. & Security,24(2):124–133, 2005.
[55] M. Toomim, T. Kriplean, C. Pörtner, and J. Landay. Utility ofhuman-computer interactions: toward a science of preference
measurement. In Proc. ACM CHI, 2011.
[56] B. Ur, P. G. Kelley, S. Komanduri, J. Lee, M. Maass,
M. Mazurek, T. Passaro, R. Shay, T. Vidas, L. Bauer,
N. Christin, and L. F. Cranor. How does your password
measure up? The effect of strength meters on password
creation. In Proc. USENIX Security, 2012. To appear.
[57] M. Weir, S. Aggarwal, M. Collins, and H. Stern. Testing
metrics for password creation policies by attacking large setsof revealed passwords. In Proc. CCS, 2010.
[58] S. Z. Wilson. The protect IU blog—xkcd agrees: Use apassphrase, 2011.
[59] J. Yan, A. Blackwell, R. Anderson, and A. Grant. Password
memorability and security: Empirical results. IEEE Security
and Privacy, 2(5), Sept. 2004.
[60] Y. Zhang, F. Monrose, and M. K. Reiter. The security ofmodern password expiration: An algorithmic framework andempirical analysis. In Proc. CCS, 2010.
[61] M. Zviran and W. J. Haga. A comparison of password
techniques for multilevel authentication mechanisms. The
Computer Journal, 36(3):227–237, 1993.
[62] M. Zviran and W. J. Haga. Password security: an empirical
study. J. Mgt. Info. Sys., 15(4), 1999.

14

APPENDIXA.

Do you have a password that you use for different accounts with aslight modification for each account?
( ) Yes( ) No( ) I prefer not to answer
Do you have an email password?
( ) Yes( ) NoXKCD COMIC

The questions on this page pertain to your real email password.
What is the domain for your primary email account (e.g. hotmail.com, gmail.com, cmu.edu)?
Thinking about the real password you use for your primary email
account, how many of the following does it contain? Write "0" ifthere are none.
Uppercase letters:
Lowercase letters:
Numbers:
Symbols:
Approximately how long ago did you last change your real email
password?
( ) Within the past month( ) Within the past six months( ) Within the past year( ) More than a year ago( ) More than 5 years ago( ) Never( ) I’m not sure( ) I prefer not to answer

Figure 2: This xkcd comic suggests that users can recall passphrases composed of random words better than lower-strength
passwords that meet complexity requirements [40].

B.

DAY ONE SURVEY

This appendix includes the survey questions shown to participants during the first part of the study. All questions were required.

Does your main email provider require you to change your password periodically?
( ) Yes( ) No( ) I’m not sure( ) I prefer not to answer

Learning my password was annoying.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree

If my main email account assigned me a password like the one I
used in this study, it would make my email account more secure.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree

Learning my password was difficult.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree
Learning my password was fun.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree

I would be annoyed if my main email account assigned me a password like the one I used in this study.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree

Describe anything you did to help yourself remember your password.

If my main email account assigned me a password like the one I
used in this study, it would be easier.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree

Do you have a password or set of passwords you reuse in different
places?
( ) Yes( ) No( ) I prefer not to answer

15

C.
Are you willing to return and try to recall your password again in a
few days?
( ) Yes( ) No( ) I prefer not to answer

DAY TWO SURVEY

This appendix includes the survey questions shown to participants during the second part of the study. All questions were required.
Thank you for participating in this Carnegie Mellon University
study. Please answer the following questions honestly. There areno right or wrong answers and everyone who finishes this task completely will receive his or her bonus payment.

If you have any additional feedback about passwords or this survey,
please enter your comments here.
What is your gender?
( ) Female( ) Male( ) I prefer not to answer

How did you just enter your password for this study (please be honest – you get paid regardless, and this will help our research)?
( ) I typed it in from memory( ) It was stored in my browser( ) I cut and pasted it from a text file( ) I looked it up in the place I had recorded it earlier and then I
typed it in( ) I use a password manager that filled it in for me( ) I prefer not to answer( ) Other:( ) It was automatically filled in( ) I forgot my password and followed the password reset link

How old are you?
Which of the following best describes your highest achieved education level?
( ) Some High School( ) High School Graduate( ) Some college, no degree( ) Associates degree( ) Bachelors degree( ) Graduate degree (Masters, Doctorate, etc.)( ) Other

Did you write down or store the password you created for this study(please be honest, you get paid regardless, this will help our research)?

Are you majoring in or do you have a degree or job in computer science, computer engineering, information technology, or a related
field?
( ) Yes( ) No( ) I prefer not to answer[ ] No[ ] Yes, on paper[ ] Yes, electronically (stored in computer, phone, etc.)[ ] Other[ ] I prefer not to answer
If you wrote down or stored your password for this study, how is it
protected (choose all that apply)?
[ ] I do not protect it[ ] I stored it in an encrypted file[ ] I hid it[ ] I stored it on a computer or device protected with another password[ ] I locked up the paper[ ] I always keep the password with me[ ] I wrote down a reminder instead of the actual password[ ] I keep the paper in an office or room that only I use[ ] I stored it on a computer or device that only I use[ ] Other[ ] I prefer not to answer[ ] I did not write down my password

Are you majoring in or do you have a degree or job in art, architecture, design, photography, or a related field?
( ) Yes( ) No( ) I prefer not to answer
Are you majoring in or do you have a degree or job in math, physics,
or engineering, or a related field?
( ) Yes( ) No( ) I prefer not to answer
What is your total household income?
( ) Less than $10,000( ) $10,000 to $19,999( ) $20,000 to $29,999( ) $30,000 to $39,999( ) $40,000 to $49,999( ) $50,000 to $59,999( ) $60,000 to $69,999( ) $70,000 to $79,999( ) $80,000 to $89,999( ) $90,000 to $99,999( ) $100,000 to $149,999( ) $150,000 or more( ) Prefer not to answer

Please describe how you store your password for this study, including what software you use or where you wrote it down.
What would you have done differently in protecting and remembering your password if this password were used for an account you
would use outside this study?
Did you imagine a scene related to the words or letters in your password to help you remember it?
( ) Yes( ) NoThank You!

If so, describe the scene that you imagined.

16

Did you think of a sentence or phrase based on the words or lettersin your password to help you remember it?
( ) Yes( ) No[ ] I would have written it down on paper[ ] I would not have written it down on paper[ ] I would have stored it electronically[ ] I would not have stored it electronically[ ] I would still write it on paper, but would secure the paper better[ ] I would have tried harder to remember it[ ] Other

If so, describe the sentence or phrase that you used.
Did you think of a story related to the words or letters in your password to help you remember it?
( ) Yes( ) No

How often do you type in your real email password (we are interested in when you type it in, not when your browser enters it
automatically)?
( ) Never( ) Several times per day( ) Once per day( ) Several times per week( ) Once per week( ) A few times per month( ) Once per month( ) Less than once a month( ) I prefer not to answer

If so, describe the story that you used.
What, if anything, about your new password makes it easy for you
to remember?
Do you have an email password?
( ) Yes( ) NoThe questions on this page pertain to your real email password.

Remembering the password I use for my real email account is difficult.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree
Remembering the password I used for this study was difficult.
( ) Strongly agree( ) Agree( ) Neutral( ) Disagree( ) Strongly disagree

When logging in with your real email password, do you refer to a
written down or stored password?
( ) Yes( ) NoPrior to this survey, have you ever written down or stored your realemail password?
[ ] No[ ] Yes, on paper[ ] Yes, electronically (stored in computer, phone, etc.)[ ] Other[ ] I prefer not to answer

If you have any additional feedback about passwords or this survey,
please enter your comments here.

If you ever wrote down or stored your real email password, how
was it protected (choose all that apply)?
[ ] I did not write down or store it[ ] I did not protect it[ ] I stored it in an encrypted file[ ] I hid it[ ] I stored it on a computer or device protected with another password[ ] I locked up the paper[ ] I always kept the password with me[ ] I wrote down a reminder instead of the actual password[ ] I kept the paper in an office or room that only I use[ ] I stored it on a computer or device that only I use[ ] Other[ ] I prefer not to answer

Thank you.

D.

CONDITIONSThis appendix includes details about the password and passphrase
construction for each condition introduced in Section 3.2 First we
detail the characters used for random password construction, thenwe enumerate the terms found in each dictionary for conditions thatused dictionaries. The ordering of the terms in each dictionary is
by frequency as calculated by the Corpus of Contemporary American English [14]. The pronounceable condition uses the algorithm
described in [19].

To how many people have you given your real email password?
()0()1( ) 2-5( ) 6-10( ) More than 10

D.1

Random character passwords

The random character conditions used passwords created fromthe following set of characters:
abcdefghjkmnpqrstuvwxyz
ABCDEFGHJKLMNPQRSTUVWXYZ
23456789

Consider the password you used for this study. If you were protecting/remembering a password for a real email account you would
use outside the study, what would you have done differently?
[ ] Nothing would have changed@!$*#.-&_

This set specifically omits characters that could be confused withother characters, such as mistaking the number zero for the letter17‘O’ or the number one for an uppercase ‘I’ or lowercase ’L’ character.

change morning walk reason low win research girl guy
early food before moment himself air teacher force offer
enough both education across although remember footD.2

181-word dictionary

second boy maybe toward able age off policy everything
love process music including consider appear actually buy

Our 181 word dictionary, used in the pp-small condition, consistsof the following words.

probably human wait serve

the be and of a in to have to it I that for you hewith on do say this they at but we his from that not byshe or as what go their can who get if would her all my

D.4

make about know will as up one time there year so think
when which them some me people take out into just see

Our 1024 word dictionary, used in pp-large and pp-large-3word
conditions, consists of the following words.

1024-word dictionary

him your come could now than like other how then its ourthe be and of a in to have to it I that for you he

two more these want way look first also new because daywith on do say this they at but we his from that not

more use no man find here thing give many well only thosetell one very her even back any good woman through us

by she or as what go their can who get if would her
all my make about know will as up one time there year

life child there work down may after should call world

so think when which them some me people take out intoover school still try in as last ask need too feel three

just see him your come could now than like other how

when state never become between high really something

then its our two more these want way look first also
new because day more use no man find here thing give

most another much family own out leave put old while
mean on keep student why let great same big group begin

many well only those tell one very her even back any

seem country help talk where turn problem every startgood woman through us life child there work down may

hand might American show part about against place overafter should call world over school still try in aslast ask need too feel three when state never become
between high really something most another much family

D.3

own out leave put old while mean on keep student why

401-word dictionary

let great same big group begin seem country help talkOur 401 word dictionary, used in pp-medium and pp-med-unorder conditions, consists of the following words.

where turn problem every start hand might American show
part about against place over such again few case mostthe be and of a in to have to it I that for you heweek company where system each right program hear sowith on do say this they at but we his from that not byquestion during work play government run small number off

she or as what go their can who get if would her all mymake about know will as up one time there year so thinkalways move like night live Mr point believe hold today

when which them some me people take out into just see

bring happen next without before large all million musthome under water room write mother area national money

him your come could now than like other how then its ourstory young fact month different lot right study booktwo more these want way look first also new because day
more use no man find here thing give many well only thoseeye job word though business issue side kind four headtell one very her even back any good woman through us

far black long both little house yes after since longprovide service around friend important father sit awaylife child there work down may after should call world

until power hour game often yet line political end among

over school still try in as last ask need too feel threeever stand bad lose however member pay law meet car citywhen state never become between high really something
most another much family own out leave put old while meanalmost include continue set later community much nameon keep student why let great same big group begin seem

five once white least president learn real change teamminute best several idea kid body information nothing ago

country help talk where turn problem every start handright lead social understand whether back watch together

might American show part about against place over suchagain few case most week company where system each rightfollow around parent only stop face anything create

program hear so question during work play government run

public already speak others read level allow add office
spend door health person art sure such war history party

small number off always move like night live Mr point

within grow result open change morning walk reason low

believe hold today bring happen next without before

win research girl guy early food before moment himself

large all million must home under water room write motherarea national money story young fact month different

air teacher force offer enough both education across

lot right study book eye job word though business issue

although remember foot second boy maybe toward able
age off policy everything love process music includingside kind four head far black long both little house yes

consider appear actually buy probably human wait serve

after since long provide service around friend importantfather sit away until power hour game often yet line

market die send expect home sense build stay fall ohpolitical end among ever stand bad lose however member

nation plan cut college interest death course someone
experience behind reach local kill six remain effect use

pay law meet car city almost include continue set lateryeah suggest class control raise care perhaps little latecommunity much name five once white least president

hard field else pass former sell major sometimes requirelearn real change team minute best several idea kid bodyinformation nothing ago right lead social understandalong development themselves report role better economic

whether back watch together follow around parent only

effort up decide rate strong possible heart drug show
leader light voice wife whole police mind finally pullstop face anything create public already speak others

return free military price report less according decision

read level allow add office spend door health person
art sure such war history party within grow result open

explain son hope even develop view relationship carry
town road drive arm true federal break better difference18thank receive value international building action fullbar beautiful property instead improve stuff detailmodel join season society because tax director early
position player agree especially record pick wear paper

method sign somebody magazine hotel soldier reflect heavy

special space ground form support event official whose

purpose exist pattern whom skin agent owner machinesexual cause bag heat fall marriage tough sing surface

matter everyone center couple site end project hit baseactivity star table need court produce eat Americanteach oil half situation easy cost industry figure facestreet image itself phone either data cover quite picture

D.5

clear practice piece land recent describe product doctor

Noun dictionary

wall patient worker news test movie certain north love

Our 181 noun dictionary, used in pp-nouns, pp-nouns-instr, andpp-sentence conditions, consists of the following words.

personal open support simply third technology catch
step baby computer type attention draw film Republicantime year people way day man thing woman life childworld school state family student group country problem

tree source red nearly organization choose cause hair

hand part place case week company system program question

look point century evidence window difficult listen

work government number night Mr point home water roomsoon culture billion chance brother energy period

mother area money story fact month lot right study bookcourse summer less realize hundred available plant
likely opportunity term short letter condition choice

eye job word business issue side kind head house service
friend father power hour game line end member law car

place single rule daughter administration south husband

city community name president team minute idea kid

Congress floor campaign material population well callbody information back parent face others level officeeconomy medical hospital church close thousand risk
current fire future wrong involve defense anyone increase

door health person art war history party result change
morning reason research girl guy food moment air teacher

security bank myself certainly west sport board seek perforce education foot boy age policy process music market

subject officer private rest behavior deal performancesense nation plan college interest death experiencefight throw top quickly past goal second bed order author

effect use class control care field development role

fill represent focus foreign drop plan blood upon agency
push nature color no recently store reduce sound note

effort rate heart drug show leader light voice wife
police mind price report decision son view relationshipfine before near movement page enter share than common

town road arm difference value building action model

poor other natural race concern series significant

season society tax director position player record paper

similar hot language each usually response dead riseanimal factor decade article shoot east save sevenspace ground form event official matter center couplesite project activity star table need court American

artist away scene stock career despite central eightoil situation cost industry figure street image phonethus treatment beyond happy exactly protect approach
lie size dog fund serious occur media ready sign thought
list individual simple quality pressure accept answer
hard resource identify left meeting determine prepare

D.6

disease whatever success argue cup particularly amount

Sentence-like dictionaries

The sentence like condition is constructed from a noun followed
by a transitive verb, then an adjective and finally another noun. Dueto this construction a dictionary is required for each part of speech
type. For nouns, the same 181 words pp-nouns dictionary is used.
For verbs, the following 181 words are used.

ability staff recognize indicate character growth loss
degree wonder attack herself region television box TV
training pretty trade deal election everybody physicallay general feeling standard bill message fail outside
arrive analysis benefit name sex forward lawyer present
section environmental glass answer skill sister PM

has does says goes cans gets makes knows wills thinks
takes sees wants looks uses finds gives tells works

professor operation financial crime stage ok compare
authority miss design sort one act ten knowledge guncalls tries asks needs feels becomes leaves puts means
keeps lets begins helps talks turns starts shows hears

station blue state strategy little clearly discuss indeed
force truth song example democratic check environment

plays runs moves likes lives believes holds brings writes
provides sits stands loses pays meets includes continues

leg dark public various rather laugh guess executive
set study prove hang entire rock design enough forget

sets learns changes leads understands watches follows
stops creates speaks reads allows adds spends grows

since claim note remove manager help close sound enjoy

opens walks wins offers remembers loves considers buys

network legal religious cold form final main sciencegreen memory card above seat cell establish nice trial

waits serves dies sends expects builds stays falls cuts
reaches kills suggests raises passes sells requires

expert that spring firm Democrat radio visit management
care avoid imagine tonight huge ball no close finish

reports decides pulls returns explains hopes develops
carries drives breaks thanks receives joins agrees picks

yourself talk theory impact respond statement maintaincharge popular traditional onto reveal direction weapon

wears supports ends hits bases produces eats teaches
faces covers describes catches draws chooses causes

employee cultural contain peace head control base pain
apply play measure wide shake fly interview manage chair

points realizes places closes involves increases seeks
deals fights throws fills represents focuses drops plans

fish particular camera structure politics perform bit

pushes reduces notes enters shares rises shoots saveweight suddenly discover candidate top production treat
trip evening affect inside conference unit best styleprotects lies accepts identifies determines preparesargues recognizes indicates wonders lays fails names

adult worry range mention rather far deep past edge
individual specific writer trouble necessary throughout

presents answers compares misses acts states discusses
forces checks laughs guesses studies proves hangs designs

challenge fear shoulder institution middle sea dreamforgets claims removes sounds enjoys forms establishes

19For adjectives, the following 181 words are used.
other new good high old great big American small large
national young different black long little important
political bad white real best right social only public
sure low early able human local late hard major better
economic strong possible whole free military true
federal international full special easy clear recent
certain personal open red difficult available likely
short single medical current wrong private past foreign
fine common poor natural significant similar hot deadcentral happy serious ready simple left physical generalenvironmental financial blue democratic dark various
entire close legal religious cold final main green nicehuge popular traditional cultural wide particular top
far deep individual specific necessary middle beautiful
heavy sexual tough commercial total modern positive civil
safe interesting rich western senior key professional
successful southern fresh global critical concerned
effective original basic powerful perfect involved
nuclear British African very sorry normal Chinese front
supposed Soviet future potential European independent
Christian willing previous interested wild average quick
light bright tiny additional present warm annual French
responsible regular soft female afraid native broad
wonderful growing Indian quiet aware complete active
chief20