An Empirical Study of Textual
Key-Fingerprint Representations
Sergej Dechand, University of Bonn; Dominik Schürmann, Technische Universität
Braunschweig; Karoline Busse, University of Bonn; Yasemin Acar and Sascha Fahl,
Saarland University; Matthew Smith, University of Bonnhttps://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/dechandThis paper is included in the Proceedings of the25th USENIX Security Symposium
August 10–12, 2016 • Austin, TX
ISBN 978-1-931971-32-4

Open access to the Proceedings of the25th USENIX Security Symposiumis sponsored by USENIXAn Empirical Study of Textual Key-Fingerprint Representations
Sergej Dechand
USECAP, University of Bonn

Dominik Schürmann
IBR, TU Braunschweig

Karoline BusseUSECAP, University of Bonn

Yasemin Acar
CISPA, Saarland University

Sascha Fahl
CISPA, Saarland University

Matthew SmithUSECAP, University of Bonn

Abstract
Many security protocols still rely on manual ﬁngerprintcomparisons for authentication. The most well-known
and widely used key-ﬁngerprint representation are hexadecimal strings as used in various security tools. With
the introduction of end-to-end security in WhatsApp andother messengers, the discussion on how to best represent key-ﬁngerprints for users is receiving a lot of interest.
We conduct a 1047 participant study evaluating six
different textual key-ﬁngerprint representations with regards to their performance and usability. We focus on
textual ﬁngerprints as the most robust and deployable
representation.
Our ﬁndings show that the currently used hexadecimal representation is more prone to partial preimage attacks in comparison to others. Based on our ﬁndings,
we make the recommendation that two alternative representations should be adopted. The highest attack detection rate and best usability perception is achieved with asentence-based encoding. If language-based representations are not acceptable, a simple numeric approach still
outperforms the hexadecimal representation.

1

Introduction

Public key cryptography is a common method for authentication in secure end-to-end communication andhas been a part of the Internet throughout the last two
decades [7, 11]. While security breaches have shownthat systems based on centralized trusted third parties
such as Certiﬁcate Authorities and Identity Based Private
Key Generators are prone to targeted attacks [42], decentralized approaches such as Web of Trust and Namecoinstruggle with beeing adopted in practice due to usability
issues [7, 13, 30]. Certiﬁcate transparency systems, suchas CONIKS and others [24, 39, 27], aim to solve a subsetof these issues by providing an auditable directory of all

USENIX Association

user keys. Still, manual key veriﬁcation, i. e., the link between public keys and the entities, such as hostnames or
people, remains a challenging subject, especially in decentralized systems without pre-deﬁned authorities, such
as SSH, OpenPGP, and secure messaging [12, 41].
Many traditional authentication systems still rely on
manual key-ﬁngerprint comparisons [17]. Here, keyﬁngerprints are generated by encoding the (hashed) public key material into a human readable format, usually
encoded in hexadecimal representation. A variety of alternatives such as QR Codes, visual ﬁngerprints, Near
Field Communication (NFC), and Short Authentication
Strings (SAS) have been proposed. Most of these systems offer speciﬁc beneﬁts, e. g., QR codes and NFC do
not require users to compare strings, but they also comewith speciﬁc disadvantages, e. g., they require hardware
and software support on all devices. While advances arebeing made in these areas, the text-based representationis still the dominant form in most applications.
However, due to the recent boom of secure messaging tools, the debate of how to best represent and evaluate textual ﬁngerprints has opened up again and there
are many very active discussions among security experts [28, 33]. In April 2016, WhatsApp serving over
one billion users enabled end-to-end encryption as default by implementing the Signal protocol. Key veriﬁcation is optional and can be done by using QR codes
or comparing numeric representations, in their case 60digit numbers [43]. However, it is not clear whether theirsolution is more usable than traditional representations.
In this paper, we present an evaluation of different textual key-ﬁngerprint representation schemes to aid in the
secure messenger discussion. The requirements posed tothe developers are as follows:• The ﬁngerprint representation scheme should provide ofﬂine support and work asynchronously. One
reason for this is that ﬁngerprints are often printed
on business cards or exchanged by third parties.

25th USENIX Security Symposium 193

• The ﬁngerprint should be transferable via audio
channels, e. g., it should be possible to compare ﬁngerprint over the phone.

a l i c e @ l o c a l h o s t :~ $ ssh alice@example . com
The authenticity of host ’ example . com (93.184.216.34) ’
can ’ t be established .
RSA key fingerprint is
6 f :85:66: da : e3 :7 a :02: c6 :5 e :62:3 f :36: b7 : d9 : b4 :2 c .
Are you sure you want to continue connecting ( yes / no )?

• The representation scheme should be as technically
inclusive as possible. No special hardware or software should be required to verify the ﬁngerprints:
both require a concerted and coordinated effort between many actors to get enough coverage for acomparison mechanism to be worthwhile for users
to adopt.

(a) OpenSSH: Lowercase Hexadecimal with Colons
alice@localhost :~ $ gpg -- fingerprint Bob
pub
2048 R /00012282 2015 -01 -01 [ expires : 2020 -01 -01]
Key fingerprint =73 EE 2314 F65F A92E C239 0 D3A 718 C 0701 0001 2282
uid
Bob < bob@example . com >(b) GnuPG: Uppercase Hexadecimal with Spaces• The representation should be as inclusive as possible, i. e., excluding as few people with sensory impairments (visual, color, audio, etc.) as possible.

Figure 1: Alphanumeric Fingerprints Used in Practiceevaluate the self-reported user perception to get feedback about which systems are preferred by end users.
Finally, we present our results, discuss their implications
and takeaways, and conclude our work.

The above requirements exclude many proposed representation schemes and offer an explanation why they
have not seen any adoption outside of academia. For thisreason, we focus exclusively on textual ﬁngerprint representations in our study. Textual key-ﬁngerprints do not
require hardware support and work in synchronous andasynchronous scenarios, i. e., they can be compared via
voice or printed on business cards. Depending on thescheme, they even could be recalled from memory andexchanged over a voice channel.
This paper presents our study testing the usability ofvarious textual key-ﬁngerprint representation schemes.
Our study consists of two parts: (1) an experiment where
we measured how fast and accurate participants performfor different schemes, and (2) a survey about their perception and sentiment. These also contained a directcomparison between the representations.
Our ﬁndings suggest that the most adopted alphanumeric approaches such as the Hexadecimal and Base32
scheme perform worse than other alternatives: under arealistic threat model, more than 10% of the users failed
to detect attacks targeting Hexadecimal representations,whereas our best system had failure rates of less than 3%.
While the best system for accuracy is not the fastest, itis the system which received the highest usability rating
and is preferred by users.
In the following sections, we discuss related work followed by an analysis of current implementations deploying in-persona key-ﬁngerprint representation techniquesand discuss our evaluated representation schemes. Then,
we describe our experiment evaluating text-based keyﬁngerprint veriﬁcation techniques with regards to their
attack-detection accuracy and speed. Our experiment
was conducted as an online study with 1047 participants
recruited via the Amazon Mechanical Turk (MTurk) platform. We consider the scenario outlined above, wherea user compares two key-ﬁngerprint strings encoded by
the different representation schemes. In addition to theimplicit measurements of accuracy and speed, we also

2

Related Work

Various key-ﬁngerprint representations have been proposed in academia and industry. Various cryptographic
protocol implementations still rely on manual ﬁngerprintcomparisons, while the hexadecimal representation isused in most of them. However, previous work suggeststhat ﬁngerprint veriﬁcations are seldom done in practice [17, 37].

2.1

Key-Fingerprint Representations

Previous work has shown that users struggle with comparing long and seemingly “meaningless” ﬁngerprints
and it is suspected that they even might perform poorly
in this task [19]. While most previous work has focusedon the family of visual ﬁngerprints [35, 32, 19, 10], toour knowledge, none of those focused on the differences
between various different textual ﬁngerprint representations.
Hsiao et al. have conducted a study with some textual and visual representation methods for hash veriﬁcation [19]. They compared Base32 and simple word
list representations with various algorithms for visualﬁngerprints and hash representation with Asian character sets (a subset of Chinese, Japanese Hiragana, andKorean Hangul, respectively). A within-subjects online
study with 436 participants revealed that visual ﬁngerprints score very well in both accuracy and speed, together with the Base32 text representation. Hsiao et al.
conclude that depending on the available computation
power and display size, either Base32 or one of the visual ﬁngerprinting schemes should be used. They explicitly did not evaluate hexadecimal representation or digits2

194 25th USENIX Security Symposium

USENIX Association

“because that scheme is similar to Base32 and known tobe error-prone” [19]. However, our work shows that numeric representations actually perform signiﬁcantly better than Base32 and is less error prone. In addition, ourresults suggest that language-based schemes, e. g., generated sentences achieve excellent results comparable tovisual schemes. At the same time, textual approaches are
more ﬂexible (can be read out loud) and do not exclude
people with sensory impairments.
Another study by Olembo et al. also focused mainlyon the topic of visual ﬁngerprints [32]. They developed a new family of visual ﬁngerprints and comparedthem against a Base32 representation. The Base32
strings were twelve characters long and displayed without chunking. The participants performed better withthe visual ﬁngerprints than with Base32, regarding bothaccuracy and speed. Olembo et al. conclude that theBase32 representation is far away from optimal whenit comes to manual key-ﬁngerprint veriﬁcation. We
test this claim by comparing Base32 representation withother textual key-ﬁngerprint representation and eventually prove it wrong.
Regarding chunking, Miller et al. have published The
magical number seven and succeeding work that showsthat most people can recall 7 ± 2 items from their memory span [29]. It has been shown that although there
are slight differences between numbers, letters and words(numbers perform slightly better than letters, and lettersslightly better than words), they perform similar in studies. More recent studies have shown that human working
memory easily remembers up to 6 digits, 5.6 letters and5.2 words [1, 6, 8]. Adjusting chunk sizes to these numbers can help users when comparing hashes.
While all of the above studies offer interesting insights
into different (mainly visual) ﬁngerprint representations,to the best of our knowledge there is not work focusing
on which textual representation performs the best. However, this knowledge would be extremely important tohelp in the current debate in the secure messaging community. The representations currently being put forward
and implemented are far from optimal and the results ofour study can help improve the accuracy and usability ofﬁngerprint representations. Unlike the above studies we
conduct our study with a more realistic attacker strenth,
as presented in subsection 4.1).

2.2

SchemeExampleHexadecimal

18e2 55fd b51b c808
601b ee5c 2d69

Base32

ddrf l7nv dpea
qya3 5zoc 22i

Numeric

2016 507 6420 1070 394
1136 2973 991 70

PGP

locale voyager wafﬂe disable
Belfast performance slingshot Ohio
spearhead coherence hamlet liberty
reform hamburger

Peeriobates talking duke rummy slurps
iced farce pound day

SentencesYour line works for this kind power cruelly.
That lazy snow agrees upon our tall offer.

Table 1: Examples for different textual key-ﬁngerprint representations for the same hash value

based systems, key-ﬁngerprints cannot be chosen by
the end-user and thus are more related to the systemassigned passphrases ﬁeld: Bonneau et al. have shownthat users are able to memorize 56-bit passwords [4].
miniLock1 and its commercial successor Peerio2 use
system-assigned passphrases to generate cryptographic
key pairs easing key backup and synchronization among
multiple devices.
Contrary to widespread expectations, Shay et al. were
not able to ﬁnd any signiﬁcant recall differences between
system-assigned passphrases and system-assigned passwords [40]. However, they reported reduced usability
due to longer submission times due to typing.
Similar to passphrases, the usage of language-based
key-ﬁngerprint representations is claimed to provide better memorability than just an arbitrary series of character strings despite the lack of empirical evidence. In ourstudy, we measure the performance of the different approaches and also collect perception and feedback fromend users.

3

BackgroundIn the past years, various textual key-ﬁngerprint representations have been proposed. In this section, we analyze currently practised in-persona key veriﬁcation techniques in well-known applications. For comparison, Table 1 lists the approaches we used in our evaluation generated from the same hash value.
Only applications requiring manual key-ﬁngerprint

Passwords and Passphrases

A passphrase is basically a password consisting of a series of words rather than characters. In academic literature, passphrases are often considered as a potentially more memorable and more secure alternative topasswords and are often recommended by system administrators [23, 40]. In contrast to most passphrase-

1 https://minilock.io
2 https://peerio.com

3
USENIX Association

25th USENIX Security Symposium 195

veriﬁcation are considered. In mechanisms like S/MIME
or X.509, ﬁngerprints play only a secondary role because
certiﬁcates are veriﬁed via certiﬁcate chains.
In the following, SHA-1(x)16 deﬁnes the execution of
16 rounds of nested SHA-1 on x, a truncation to the leftmost 16 bits is deﬁned by x[0, . . . , 16], and pk is used as
an abbreviation for the values of a public key (differs for
RSA, DSA, or ECC).

3.1

Note that the case of the letters do not make any difference. Regarding chunking, both spaces (cf. Figure 1b)
and colons (cf. Figure 1a) are commonly used as separation characters.
Key ﬁngerprints in OpenPGP version 4 are deﬁned in
RFC 4880 [7] by
Hex(SHA-1(0x99  len  4  creation time  algo  pk))
where len is the length of the packet, creation time is thetime the key has been created and algo is unique identiﬁer for the public-key algorithm. While the inclusionof creation time makes sure that even two keys with thesame key material have different ﬁngerprints, it allows an
attacker to iterate through possible past times to generate
similar ﬁngerprints skipping the key generation step [5].
The actual representation of OpenPGP ﬁngerprints is not
deﬁned in RFC 4880, but most implementations chose to
encode them in hexadecimal form, e. g., GnuPG displaysthem uppercase in 16 bit blocks separated by whitespaces
with an additional whitespace after 5 blocks (cf. Figure 1b), e. g.,

Numeric

Numeric representation describes the notation of data using only numeric digits (0-9). The primary advantage ofa such system is that Arabic numerals are universally understood, and in addition, numeric key-ﬁngerprints show
a similarity to phone numbers. The encoding is achievedby splitting a binary hash into chunks of equal lengthand expressing each chunk as a decimal number, e. g., bysimply switching the representation base from 2 to 10.
The messaging and data exchange application SafeSlinger3 implements this as a fallback scheme for unsupported languages [14]. A 24 bit SAS in SafeSlinger (cf.
Figure 2a) can be expressed by three decimal encoded
8-bit numbers.
In the messaging platform WhatsApp, a ﬁngerprintis calculated by SHA-256(pk)5200 [0, . . . , 240]. This ﬁngerprint is split up into six chunks, where each chunkis represented by a ﬁve digits long number modulo
100,000 [43]. Concatenating this ﬁngerprint with the ﬁngerprint of the communication partner results in the displayed representation, e. g.,

73EE 2314 F65F A92E C239

Other implementations, such as OpenKeychain, deviate
only slightly, for example by displaying them lowercase
or with colored letters to ease comparison but still provide compatibility with GnuPG.
SSH ﬁngerprint strings, as deﬁned in RFC 4716 andRFC 4253 [15, 44], are calculated byHex(MD5(Base64(algo  pk)))

77658 87428 72099 51303
34908 23247 95615 27317
09725 59699 62543 54320

3.2

where algo is a string indicating the algorithm, for example “ssh-rsa”. Fingerprints are displayed as “hexadecimal with lowercase letters and separated by colons” [15](cf. Figure 1a), e. g.,

Alphanumeric

Alphanumeric approaches use numbers and letters to
represent data. Depending on the representation typeand its parameters, the letters can be presented either
in lower-case or in upper-case. The string can be chunked into groups of characters, which are usually of equal
length. Chunking does not alter the information contained, while changing lower-case letters to upper-caseletters (and vice versa) may does, depending on the coding scheme. Commonly used representations are Hexadecimal, Base32, and Base64.
3.2.1

0D3A 718C 0701 0001 2282

6f:85:66:da:e3:7a:02:c6:5e:62:3f:36:b7:d9:b4:2c

3.2.2

Base32

Base32 uses the Latin alphabet (A-Z) without the letters O and I (due to the confusion with numbers 1 and0). There is no difference between lower-case letters andupper-case letters. In addition, a special padding character “=” is used, since the conversion algorithm processes blocks of 40 bit (5 Byte) in size. The source stringis padded with zeroes to achieve a compatible lengthand sections containing only zeroes are represented by“=” [20, 21].
The ZRTP key exchange scheme for real-time applications is based on a Difﬁe-Hellman key exchange
extended by a preceding hash commitment that allows
for very short ﬁngerprints, called Short Authentication

Hexadecimal

Hexadecimal digits use the letters A-F in addition to numerical digits and are a common representation for keyﬁngerprints and primarily used in SSH and OpenPGP.
3 https://www.cylab.cmu.edu/safeslinger

4
196 25th USENIX Security Symposium

USENIX Association

Strings (SAS) without compromising security [45]. The
Base32 encoding used in ZRTP uses a special alphabet
to produce strings that are easier to read out loud. VoIP
applications such as CSipSimple4 use this Base32 option, usually named “B32” inside the protocol. Here, theleftmost 20 bits of the 32 bit SAS value are encoded as
Base32. , e. g.,
5 e m g

3.2.3

Base64

There exist a number of speciﬁcations for encoding data
into the Base64 format, which uses the Latin alphabet inboth lower-case and upper-case (a-z, A-Z) as well as thedigits 0-9 and the characters “+”, “/”, and “=” to represent text data. Again, the character “=” is used to encode
padded input [20]. Starting with OpenSSH 6.8 a newﬁngerprint format has been introduced that uses SHA256 instead of MD5 and Base64 instead of hexadecimal
representation. In addition the utilized hash algorithm isprepended, e. g.,(a) SafeSlinger: List of words

Figure 2: Language-based ﬁngerprint representations

quota holiness

The messaging application SafeSlinger is based on
a Group Difﬁe-Hellman protocol [14] implementing a
key veriﬁcation with SASs for up to 10 participants. In
SafeSlinger the leftmost 24 bits of a SHA-1 hash is used
to select 3 words from the PGP Word List, e. g.,

SHA256:mVPwvezndPv/ARoIadVY98vAC0g+P/5633yTC4d/wXE

3.3(b) OpenKeychain: Sentences

Unrelated Words

Instead of (alpha)numeric representation, ﬁngerprints
can be mapped to lists of words. Here, the binary representation is split into chunks, where each possible
value of a chunk is assigned to a word in a dictionary.
To increase readability, such a dictionary usually contains no pronouns, articles, prepositions and such. Wordlists, such as the PGP Word List [22] and the Basic English word list compiled by K.C. Ogden [31], are primarily used for veriﬁcation mechanisms based on SAS.
Key-Fingerprints represented by words have been implemented for VoIP applications based on the ZRTP key
exchange and other real-time communication protocols.
Examples are Signal5 , and the messaging and contact
sharing application SafeSlinger [14] (cf. Figure 2). Besides their use in SAS based mechanisms, miniLock andPeerio utilize unrelated words for passphrase generation.
An example for a modern VoIP implementation thatutilizes ZRTP for key exchange over Secure Real-TimeTransport Protocol (SRTP) is Signal’s private calling feature, previously distributed as Redphone. The developers chose to implement only a speciﬁc subset of the
ZRTP speciﬁcation [45], namely Difﬁe-Hellmann key
exchange via P-256 elliptic curves using “B256” SASs,i. e., Base256 encoding that maps to the leftmost 16 bitsof the 32 bit SAS values to the previously introduced
PGP Word List [22], e. g.,suspense unify talon.

Besides this, two other 3 word triples are selected to force
users to make a selection before proceeding (cf. Figure 2a).
In contrast to Signal and SafeSlinger, Peerio (basedon miniLock) does not use any SAS based veriﬁcation
mechanism. It uses pictures for veriﬁcation and wordlists for code generation. The word list is generated frommost occurring words in movie subtitles. Besides key
veriﬁcation, these are also used to generate so called
passphrases, which are used to derive their ECC private
keys.

3.4

Generated Sentences

The words from the previous dictionaries can also be
used to generate syntactically correct sentences as proposed by previous research: Goodrich et al. proposed
to use a “syntactically-correct English-like sentence”
representation for exchanging hash-derived ﬁngerprints
over audio by using text-to-speech (TTS) [16]. Michael
Rogers et al. implemented a simple deterministic sentence generator [16, 38]6 Though the sentences fromboth approaches rarely make sense in a semantic fashion, they are syntactically correct and are claimed to pro-

4 https://github.com/r3gis3r/CSipSimple

5 https://github.com/WhisperSystems/Signal-Android

6 https://github.com/akwizgran/basic-english

5
USENIX Association

25th USENIX Security Symposium 197

vide good memorability. In our study, we used Michael
Roger’s approach for our sentence generator.
We implemented this method for PGP ﬁngerprints
in OpenKeychain 3.67 (cf. Figure 2b). To the best of
the authors’ knowledge, to this date, it is the ﬁrst integration of key veriﬁcation via sentences although otherprojects are considering to change their ﬁngerprint encoding scheme [38, 36].

4

It is important to note that collision resistance is not required for our scenarios. It is required for infrastructurebased trust models such as X.509, where certiﬁcates are
veriﬁed by machines and trust is established by authority.
In these schemes, a signature generated by a trusted authority can be requested for a certiﬁcate by proving thecontrol over a domain, but then reused maliciously fora different certiﬁcate/domain. This is already possible
with a collision attack, without targeting a full preimage.
In contrast, the direct human-based trust schemes considered in this study only need to be protected against
preimage attacks, because no inherently trusted authority is involved here.
While machine comparison needs to be done fast, e. g.,on key import, manual ﬁngerprint veriﬁcation by humans
is done asynchronously in person or via voice. Thus, we
can use a key derivation function to provide a proof-ofwork, effectively trading calculation time for a shorter
ﬁngerprint length. Secure messaging applications suchas Signal or OpenPGP-based ones could pre-calculate
the ﬁngerprints after import and cache these before displaying them for veriﬁcation later.
Thus, modern memory-hard key derivation functions
such as scrypt [34] or Argon2 [3] can be utilized toshorten the ﬁngerprint length. These key derivation functions are parametrized to allow for different work factors.
Suitable parameters need to be chosen by implementations based on their targeted devices and protocol.
As discussed in Section 3.2.1, while the generation of
new ﬁngerprints consists of the creation of a new key pairand the key derivation step, an attacker can potentiallyskip the key creation. Thus, in the following we only
consider the key derivation performance as the limiting
factor for brute force attacks.
When utilizing a properly parametrized key derivation
function for bit stretching, the security of a 112 bit longﬁngerprint can be increased to require a brute force attack comparable to a classical 2128 brute force attacker.
Consequently, a ﬁngerprint length of 112 bit is assumed
throughout our study.

Methodology

In order to evaluate the effect and perception of the different textual key-ﬁngerprint representations, we conducted an online study on Amazon’s Mechanical Turk
(MTurk) crowdsourcing service. Our Universities do nothave an IRB, but the study conformed to the strict dataprotection law of Germany and informed consent was
gathered from all participants. Our online study is divided into two parts: The experiment for performance
evaluation followed by a survey extracting self-reported
data from users. The survey ended with demographic
questions.

4.1

Security Assumptions

In this section, we deﬁne the underlying security assumptions of our study, such as ﬁngerprint method, length, andstrength against an adversary. The ﬁngerprint method
and parameters are utilized consistently for all experiments in our study to offer comparability between all
possible ﬁngerprint representations. This attack model isimportant for the usability since an unrealistically strong
or weak attacker could skew the results. Obviously, if the
ﬁngerprint strength is not kept equal between the systems
this would also skew the results.
4.1.1

Fingerprint Method

To decide upon a ﬁngerprint method for humanly veriﬁable ﬁngerprints in our study, we ﬁrst have to differentiate between human and machine veriﬁcation to illustrate their differences. While a full ﬁngerprint comparison can be implemented for machine veriﬁcation, humans can fall for ﬁngerprints that match only partially.
Additionally, machine comparison can work with longvalues, whereas for human veriﬁcation the length must
be kept short enough to ﬁt on business cards and to keep
the time needed for comparison low.
For machine comparison, full SHA-256 hashes should
be calculated binding a unique ID to the public key material. The probability of ﬁnding a preimage or collision attack is obviously negligible, but the ﬁngerprints can still
be computed fast in an ad-hoc manner when needed.

4.1.2

Attacker Strength for Partial Preimages

In our user study, we assume an average attacker trying to impersonate an existing ID using our ﬁngerprint
method. Thus, an attacker would need to ﬁnd a 112 bit
preimage for this existing ﬁngerprint using a brute force
search executing the deployed key derivation function ineach step. Due to the work factor, we consider this to be
infeasible and instead concentrate on partial preimages.
For comparability and to narrow the scope of our study,
an attacker is assumed that can control up to 80 bits ofthe full 112 bit ﬁngerprint.
Attackers might aim to ﬁnd partial preimages where

7 https://www.openkeychain.org

6
198 25th USENIX Security Symposium

USENIX Association

the uncontrolled bits occur at positions that are more easily missed by inattentive users. First, the bits at the beginning and the end should be ﬁxed as users often begin
their comparison with these bits. Thus, we assume that,for any representation method, the ﬁrst 24 and last 24
bits are controlled by the attacker and thus the same as in
the existing ﬁngerprint. Based on the feedback from our
pre-study participants and reports from related work, thiscan be considered best-practice [17, 37]. Second, of theremaining 64 bits in the middle of our 112 bit ﬁngerprint,
we assume that 32 bits are controlled by the attacker in
addition to the ﬁrst 24 and last 24 bits. In total, we assume that 80 bits are controlled by the attacker, i. e., arethe same as in the existing ﬁngerprint, and 32 bit are uncontrolled.
The probability of ﬁnding such a partial preimage fora ﬁngerprint when executing 249 brute force steps is calculated approximately by1−

Figure 3: A screenshot of the actual task a user had to perform in the experiment. A user rates whether the security codes
match, in this case with the Peerio word list approach, by clicking on the corresponding buttons shown on the phone.

64 249
2112 − ∑32
k=1 k≈ 0.66.
2112

4.2

Pre-Study

To get additional feedback from participants and evaluate our study design for ﬂaws and misunderstandings,
we conducted two small pre-studies: A lab study with 15
participants and an MTurk experiment with 200 participants, all required to perform 10 comparisons for each
representation scheme (totally 60 comparisons in a randomized order). In our lab-study, we mainly focusedon qualitative feedback, whereas the main goal of the
MTurk pre-study was to ﬁnd ﬂaws in the presentationand task descriptions, as well as to check whether our
proposed methodology is received as expected.
The biggest problem we found regarding the study design was that participants were uncertain if they should
check for spelling mistakes in the words and sentencebased representation or if the all attacks would change
entire words. To clarify this, a speech bubble was included in the task description that the participants do nothave to look for spelling mistakes for language-based approaches.
We tested different rates of attack during the pre-study.
The results showed that participants who were exposed
to frequently occurring attacks were more aware and had
a much higher attack detection rate. For our main study,
we reduced the number of attacks to 40 comparisons with
4 attacks to have a good balance between true positives
and false negatives. We received feedback that attacks
on anchor parts of the strings, i. e., in the beginning,
end, and at line breaks could be easily detected. Many
users had problems with distinguishing the hexadecimalfrom the Base32 representation as well as distinguishing different word list approaches (Peerio vs. OpenPGP
word list). Thus, we opted for a mixed factorial study

The inner parentheses of this equation deﬁne the probability that no partial preimage exists
  for one speciﬁc
bit permutation. Instead of using 64
32 , a sum over 32
variations has been inserted to include permutations withmore than the uncontrolled 32 bit that are also valid partial preimages. Finally, the probability to ﬁnd a partial preimage is deﬁned by the inverse of the exponentiation. Assuming the scrypt key derivation function
parametrized with (N, r, p) = (220 , 8, 1), Percival calculates the computational costs of a brute force attack
against 238 (≈ 268 ) hashed passwords with $610k and
253 (≈ 958 ) with $16B [34]. These costs can be considered a lower and upper bound for our attacker, which we
assume to have average capabilities and resources. While
238 has a probability of ﬁnding a partial preimage of only
0.05%, with 242 the probability reaches nearly 1%, andwith 249 , as in our example, a partial preimage is foundwith over 50%.
In our study, we simulate attacks by inverting the bitsfrom the existing ﬁngerprint which are uncontrolled by
the attacker, while the controlled bits are unchanged.
For our theoretical approximation, we assume that theﬁrst 24 and last 24 bits should be controlled as well as
32 bits from the middle. In our study, we simulate aneven more careful selection of appropriate ﬁngerprints
from the ones that an attacker would brute force. A general criteria here is to minimize the inﬂuence of uncontrolled bits on the entire ﬁngerprint: For numeric and alphanumeric representations all bits affecting a characteror digit are inverted together. For unrelated words, allbits affecting a word are changed. Sentences are neverchanged in a way that would alter the sentence structure.
7
USENIX Association

25th USENIX Security Symposium 199

design where users test only one scheme of each type.
We grouped the hexadecimal and Base32 scheme for thealphanumeric type and the PGP and Peerio for the wordlist type together. These two groups were tested betweensubjects in a split-plot design, i. e., the participants test
either hexadecimal or Base32 for the alphanumeric type.
See Table 2 for a graphical representation of our condition assignment design.

4.3

Type (Within-Group)

Scheme (Between-Group)

Alphanumeric
Numeric
Unrelated Words
Generated Sentences

Hexadecimal XOR Base32
Numeric
PGP XOR Peerio
Generated Sentences

Table 2: To avoid confusion between too similar approaches(cf. Section 4.2), in our condition assignment, scheme types(left column) can consist of multiple representation schemes(right column). Each participant tests only one randomly assigned scheme of each type in a randomized order. .

Experiment Design

The main part of our online study is the experimentpart where users perform actual ﬁngerprint comparisons.
Here, we conducted two separate experiments with a distinct set of participants: (1) our main experiment testing different textual high-level representation schemes
against each other and (2) a secondary experiment testing
different chunk sizes for the hexadecimal representation.
We opted for two distinct experiments due to the exponential growth of experiment conditions, as described in
Section 4.3.1.
Before letting the participants start our experiment, we
explained the scenario:

started with 4 training tasks (each method once) not considered in the evaluation. The user’s only task is torate whether the shown ﬁngerprints match by clicking on
Match or Doesn’t Match on the phone. Based on the condition assignment, participants see different approaches
in a randomized order. We measure whether their answer was correct and their speed, i. e., the amount of timespent on the comparison. The experiment is concluded
with a survey collecting feedback on the used approaches
and the tasks and demographic information discussed in
the “Results” section.

“With this HIT, we are conducting an academic usability study followed by a short survey about different types of security codesused in the IT world. Security codes are often used in encrypted communications to identify the participants in a communication. If the
security codes match, you are communicating
securely. If they don’t match, an eavesdropper
may be intercepting your communication”.

4.3.1

Variables and Conditions

In the main experiment, the used representation scheme
is our controlled independent variable whereas its values deﬁne our experiment conditions. In our additional
chunking experiment, the chunking size is our controlled
independent variable instead of the representation algorithm. During all tasks, we measure how fast participants
perform with their given conditions and whether they areable to detect attacks by rating “incorrect” (speed andaccuracy as our measured dependent variables).
In both experiments, each user had to perform 46 comparisons in total. To detect users clicking randomly, 2obviously distinct comparisons were added to test a participant’s attention. Training comparisons and attention
tests are not included in the evaluation. Based on the
feedback in our pre-study, we added tooltips during thetraining comparisons giving hints for language-based approaches telling the user that spelling attacks would notoccur. We set the number of attacks to six: two obvious
attacks where all bits are altered serving as control questions and 4 actual attacks with partial 80-bit preimages(one for each representation scheme). Participants failingat the control attacks are not considered in the evaluation
but still received a payment if ﬁnishing all tasks. The
major challenge in the study design is a high attack detection rate in general: most users perform comparisonscorrectly for the given attacker strength.
To avoid side effects, we chose ﬁxed font size, color

On MTurk, the term Human Intelligence Task, or HIT
stands for a self-contained task that a worker can workon, submit answers, and get a reward for completing.
Since our participants might not be familiar with the keyﬁngerprint representation term, we replaced it with security codes for the sake of the study.
We opted not to obfuscate the goal of the study since
our research aims at ﬁnding the best possible representation for the comparison of key-ﬁngerprints in a security
context. This is closest to how users interact with ﬁngerprints in the real world — their secure messaging applications also ask them to compare the strings for security
purposes. The question how to motivate users to compare
ﬁngerprints is an entirely different research question. So
in our case, we believe it was not necessary or desirable
to use deception and since deception should be used as
sparingly as possible we opted for the “honest” approach.
After agreeing the terms, participants are shown a ﬁctitious business card next to a mobile phone, both displaying a security code (as shown in Figure 3). To
become more familiar with the task, the experiment is
8
200 25th USENIX Security Symposium

USENIX Association

Figure 4: A screenshot showing a statement rating in the post-experiment survey. Since the participants might
not distinguish the different types, we have provided an example from their previous task.

and style, i. e., the same typeface for all ﬁngerprint representations. In addition, we set ﬁxed line breaks for sentences and word lists. In the main experiment, the same
chunking style was used for all representations: For (alpha)numeric approaches a chunk consists of four characters separated by spaces. For word lists, we opted for aline break every four words. In the generated sentences
representation, one sentence per line is displayed. We areaware that all these design decisions can have an effecton the comparison of the representations. However, ourpre-study results show a signiﬁcantly lower effect size.
More importantly, we are mainly interested in comparing the concepts, therefore we did not vary any of thevisual attributes like font size or style. In particular, differences resulting from the font’s typeface have not been
evaluated. Lund showed in his meta-analysis that there
are no signiﬁcant legibility differences between serif andsans serif typefaces [25].

cusing on self-reported user perception and opinions
about the different approaches. This is the main reason
we opted to compare as much as possible in a withingroups fashion and only selected a small number of conditions in total. Since users might not notice the difference between the various dictionary or alphabet approaches, we designed a mixed factorial design where
the users would only get one of the alphabets/dictionaries (between-subjects) but they would test all different
high-level systems (within-group) as depicted in Table 2.
The between-group conditions have been assigned randomly with a uniform distribution. Since participantsfrom our pre-study had difﬁculties to distinguish the different chunking approaches, we skipped the survey partin the chunk-size experiment.
4.3.2

Online Survey

The experiment was followed by an online survey gathering self-reported data and demographics from participants. To measure perception, we asked the participants whether they agreed with statements discussed insubsection 5.2 on a 5 point Likert scale: from strongly
disagree to neural strongly agree as shown in Figure 4.
Participants had to rate each representation type for allstatements. Since users might not distinguish the different representation schemes, we provide an example fromtheir previously ﬁnished task.

Chunk-Size Testing A question was raised whether
the chunking of a hexadecimal string plays a greater rolein comparison to the different approaches. Thus, in addition to the main experiment testing different representation types, we conducted a second experiment with newparticipants testing different chunk sizes for the hexadecimal representation. Here, we used chunk-sizes ranging
from 2 to 8 in addition to “zero-chunk size” (8 cases).
The zero-chunk size means that no spaces have been included. To make the results more comparable, we opted
for a similar design as done in the major experiment, i. e.,
we required the same amount of comparisons, used thesame font settings, and had the same amount of attacks.
For each participant, we assigned 4 out of 8 different
chunk-sized randomly. Same as in the major experiment,
all participants had to compare 46 ﬁngerprints whereas
the ﬁrst 4 are considered as training comparisons, 4 attacks (one for each chunk size), and 2 control attacks
with obviously distinct ﬁngerprints.
The major experiment is followed by a survey fo-

4.3.3

Statistical Testing

We opted for the common signiﬁcance level of α = 0.05.
To counteract the multiple comparisons problem, we use
the Holm-Bonferronicorrection for our statistical significance tests [18]. Consequently, all our p-values are reported in the corrected version.
We test the comparison duration with the MannWhitney-Wilcoxon (MWW) test (two-tailed). We opt forthis signiﬁcance test due to a few outliers, consequently a
9

USENIX Association

25th USENIX Security Symposium 201

SchemeHexadecimal

Hexadecimal – Base32
Hexadecimal – Numeric
Hexadecimal – PGP
Hexadecimal – Peerio
Hexadecimal – Sentences

Base32

Base32 – Hexadecimal
Base32 – Numeric
Base32 – PGP
Base32 – Peerio
Base32 – Sentences

Numeric

Numeric – Hexadecimal
Numeric – Base32
Numeric – PGP
Numeric – Peerio
Numeric – Sentences

PGP

PGP – Hexadecimal
PGP – Base32
PGP – Numeric
PGP – Peerio
PGP – Sentences

PeerioPeerio – Hexadecimal
Peerio – Base32
Peerio – Numeric
Peerio – PGP
Peerio – Sentences

Sentences

Sentences – Hexadecimal
Sentences – Base32
Sentences – Numeric
Sentences – PGP
Sentences – Peerio

Speedmean [s]med [s]

stdev

11.2

10.0

6.4

1.0
0.6
−1.8
2.5
−1.1

p-val

0.0
0.6
−1.0
0.8
−0.6

<0.001<0.001<0.001<0.001<0.001

−1.0
−0.4
−2.8
1.5
−2.1

−1.1
−0.6
−2.3
1.6
−1.8

−0.0
0.6
−1.0
0.8
−0.6

<0.001<0.001<0.001<0.001<0.001

−0.6
0.4
−2.4
1.9
−1.7

−0.5
0.6
−1.7
2.2
−1.2

−0.6
−0.6
−1.6
0.2
−1.2

<0.001<0.001<0.001<0.001<0.001

1.2
2.3
1.7
3.9
0.5

1.0
1.0
1.6
1.8
0.4

<0.001<0.001<0.001<0.001<0.001

10.6

13.0

1.8
2.8
2.4
4.3
0.7

8.9

9.5

11.2

6.4

5.8

7.3

5.6

−2.7
−1.6
−2.2
−3.9
−3.4

−0.8
−0.8
−0.2
−1.8
−1.4

<0.001<0.001<0.001<0.001<0.001

0.7
1.8
1.2
−0.5
3.4

0.6
0.6
1.2
−0.4
1.4

<0.001<0.001<0.001<0.001<0.001

10.7

fails

attacks

tests

0.690
0.048
0.690
0.048
<0.001

0.49

50

4765

−2.09
0.21
−0.01
0.08
−0.99

12
−9
11
22
22

479

−1.94
−4.10
−1.65
−4.69
−7.45

1.94
−2.16
0.28
−2.75
−5.51

0.690
0.404
0.714
0.404
<0.001

2.09
2.30
2.08
2.17
1.10

−12
−21
−1
10
10

4.10
2.16
2.45
−0.59
−3.35

0.048
0.404
0.404
0.714
0.004

0.28

59

1.65
−0.28
−2.45
−3.03
−5.79

0.690
0.714
0.404
0.337
<0.001

0.01
−2.08
0.22
0.09
−0.98

39

444

4425

−11
1
−20
11
11

0.048
0.404
0.714
0.337
0.075

28

−340
−71
−4867
−431
−4858

4.69
2.75
0.59
3.03
−2.76

0.41

−35
−3
−487
−43
−492

−0.08
−2.17
0.13
−0.09
−1.07

−22
−10
−31
−11
0

8
40
−444
43
−449

91
360
−4436
431
−4427

7.45
5.51
3.35
5.79
2.76

<0.001<0.001
0.004<0.001
0.075

0.99
−1.10
1.20
0.98
1.07

−22
−10
−31
−11
0

457
489
5
492
449

4518
4787
−9
4858
4427

8.78

8.7

1.1
2.1
1.7
−0.7
3.6

f-pos

6.34

7.4

5.75

7.0

Total

p-val

8.50

−2.5
−1.5
−1.9
−4.3
−3.6

12.3

Accuracy

10.44

1.1
0.5
−1.2
2.7
−0.7

10.2

fail-rate

2.99

2.58

−0.21
−2.30
−0.22
−0.13
−1.20

0.50

1.48

38

9
21
20
31
31

28

32
−452
35
−8
−457

269
−4527
340
−91
−4518

−32
−484
3
−40
−489

−269
−4796
71
−360
−4787

452
484
487
444
−5

4527
4796
4867
4436
9

447

931

487

936

4496

9292

4856

9283

Table 3: Our experiment results showing the differences between the representation schemes. The top rows of each row groupseparated by a rule, show the raw performance of a baseline scheme, followed by italic rows showing a direct comparison delta.
Greyed-out values are not backed by statistical signiﬁcance. The columns fail-rate (undetected attacks) and false-pos (same string
rated as an attack) display percentage values.

5slightly skewed normal distribution, and a large amountof collected data. The common language effect size isshown by mean and median comparisons [26].

Results

In this section, we present our results: our online study
with 1047 participants has been conducted in August and
September 2015. The study for testing the chunk size has
been conducted in February 2016 with 400 participants.
Starting with our online experiment evaluation showing
the raw performance of users, we then present user perception results from the follow-up survey. Finally, we
discuss the demographics of our participants.

The attack detection rate is tested with a pairwise
Holm-Bonferroni-corrected Barnard’s exact test (Exakt
package in R) achieving one of highest statistical power
for 2x2 contingency tables [2].
Survey ratings are, again, tested by using the MWW
signiﬁcance test (two-tailed test). As has been shown inprevious research [9], it is most suitable for 5-point Likert scales, especially if not multimodal distributed as inour survey results. In case two ﬁngerprint representation
schemes are statistically tested against each other, only
participants encountering both schemes were considered.

5.1

Online ExperimentParticipants who have not ﬁnished all comparisons or
failed the attention tests were excluded from our eval10

202 25th USENIX Security Symposium

USENIX Association

SchemeHexadecimal (4)hex (4) – hex (0)hex (4) – hex (2)hex (4) – hex (3)hex (4) – hex (5)hex (4) – hex (6)hex (4) – hex (7)hex (4) – hex (8)mean [s]
12.3

Speed
med [s]
10.4

p-val

−2.4
−0.3
−0.3
−1.4
−1.9
−1.7
−2.8

−2.6
−0.9
0.1
−1.2
−1.8
−1.8
−3.2

<0.001<0.001
0.362<0.001<0.001<0.001<0.001

fail-rate
6.78

0.33
1.37
−0.64
1.01
2.43
3.35
1.35

Accuracy
p-val false-pos
0.38
1.000
1.000
1.000
1.000
1.000
1.000
1.000

−0.28
0.00
0.09
−0.40
0.09
0.19
−0.12

fails
16

Total
attacks
236

tests
2360

−2
−3
2
−2
−5
−8
−4

−17
3
8
5
8
−1
−10

−170
30
80
50
80
−10
−100

Table 4: Comparison of the chunking experiment results showing the differences between the representation schemes. The top row
shows the raw performance of the hexadecimal scheme with a four-character chunking, followed by italic rows showing a directcomparison delta. Greyed-out values are not backed by statistical signiﬁcance. The columns fail-rate (undetected attacks) andfalse-pos (same string rated as an attack) display percentage values.

97.97% with a similar average speed as the hexadecimal
scheme. On the downside, this scheme has produced aslightly higher false-positive rate. We found that the false
positives occurred mostly with longer sentences where
there has been a line break on the phone mock-up due
to portrait orientation. This is a realistic problem of thissystem if used with portrait orientation and not a problem
with our mock-up in itself. Improvements on making thesentences shorter could mitigate this situation.

uation: all participant compared 46 security codes in arandomized order, whereas 40 (10 of each scheme) wereconsidered in the evaluation. The four training samples
and the control questions are excluded. Few comparisonsdone in less than 2 seconds and more than one minute
have been excluded. The reason for such can either bemultiple clicks during the page load, or external interruptions of the participants. None of the attack could be
successfully detected in under 4 seconds.
Our experiment results, summarized in Table 3, show
the raw performance of all schemes regarding their
speed, accuracy and false-positive rate. The top rows ofeach row group, separated by a rule, show the raw performance of a representation scheme as baseline (negativevalues indicate lower values than the baseline). The following rows show a direct comparison delta between between two schemes. The speed column group consists ofthe mean and median (in seconds), the standard deviationand the according p-values for a direct comparison. The
fail-rate column shows the rate of the undetected attacks
with the according p-values for a direct comparison. The
total column group simply shows the total numbers oftests, attacks and undetected attacks.
The results show that the average time spent on comparisons plays only a minor role among the schemes:
4.3 s difference between the best and the worst scheme.
Note that the Peerio word-list scheme performed bestwith 8.7 s mean whereas the PGP word list performed
worst with 13 s mean (p < 0.001).
However, there is a clear effect regarding the attack detection rate (see Table 3). All alternative key-ﬁngerprint
representations performed better than the state-of-theart hexadecimal representation, where 10.1% of attacks
have not been detected by the users. Previous workshows similar numbers for Base32 [19]. To our surprise,
the numeric approach performs better in both categories:
it features an attack detection rate of 93.57% (p < 0.01)
and an average speed of 10.6s (p < 0.001). Generated
sentences achieved the highest attack detection rate of

Chunk-Size Experiment
Table 4 summarizes the results of our secondary chunksize experiment. As can be seen, no statistically significant results have been achieved for the attack detection
fail-rate (undetected attacks by end users). However, weobserved that the chunk sizes with 3 and 4 characters performed best in speed, even though the effect sizes wereminor: only 3.3 seconds difference with similar standard
deviations between the best and worst chunk size setting.
Firstly, we notice that despite the same attack strength
as in our major experiment, participants were able to detect more attacks. We suspect that the higher attack detection rate is based on (1) a higher learning effect due
to the same scheme for all comparisons and (2) in contrast to our major study, participants had a slightly higher
drop-out rate and thus only more motivated participants
were considered. This is supported by the numbers in the
total tests column of Table 4: here, we can see that forthe zero-chunking and chunking with 8 characters lesstests have been performed. This is based on the fact that
although the chunk sizes have been assigned almost uniformly, participants assigned with harder chunk settings
often dropped out before even ﬁnishing their entire task.
More importantly, our results also support the claim
from our pre-study: The chunking parameter in hexadecimal strings plays only a minor role in the attack detection fail-rate.
11

USENIX Association

25th USENIX Security Symposium 203

Trustworthiness
Alphanumeric
Numeric
Unrelated Words
Generated Sentences

5%
8%
8%
12%

82%
76%
75%
67%

Usability
Alphanumeric
Numeric
Unrelated Words
Generated Sentences

17%
9%
6%
5%

63%
76%
80%
85%

100

Response

50

0

50

Percentage

Strongly Disagree

Disagree

Neutral

Agree

100

Strongly Agree

Figure 5: Aggregated survey results for statement rating regarding the usability and trustworthiness.

5.2

Online Survey

tests are Holm-Bonferroni corrected. In case of a direct
statistical test between two schemes, only users encountering both schemes have been considered. All in all, theusability perception of the participants is almost consistent with the performance results from the experiment.

To measure the usability and trustworthiness of all representation schemes, we asked our participants whether
they agreed with the following statements:
S1 The comparisons were easy for me with this method

To measure the perception of the task difﬁculty, we
asked the participants whether they agreed with the statements S1 , S2 and S3 respectively. As illustrated in Figure 6 in the Appendix A, the effect size between the different approaches is low. However, the participants weremore likely to agree that language-based representation
schemes are easier to use. For instance, we see that incomparison to the alphanumeric schemes (average rating of 3.4), word list (average rating of 3.9, p < 0.001)
and generated sentence schemes (average rating of 4.2,
p < 0.001 ) are rated to be easier by our participants (S1 ,S5 ). While the experiment results of the sentence generators clearly outperformed all other approaches, they also
were rated better by the participants. Same applies forthe low-performing hexadecimal and Base32 schemes
which clearly received lower ratings. Consistently withthe surprising performance results in the experiment, thenumeric scheme is also considered to be easier by many
participants: average rating of 3.9 and p < 0.001.

S2 I am conﬁdent that I can make comparisons using thismethod without making mistakes
S3 I think making comparisons using this method would
help me keep my communications secure
S4 I was able to do the comparisons very quickly withthis method
S5 I found this method difﬁcult to use
S6 Overall, I liked this method
We mixed positive and negative statements, e. g., S1
and S5 , to create a more robust measure. S6 is used to
calculate the overall ranking of the different representation schemes.
Figure 5 shows the aggregated results where the usability statements are grouped to one usability featureand the trustworthiness derived from the rating on thestatement S3 . Negative statement ratings have been inverted for a better comparison. Figure 6 shows the rating
results for each speciﬁc statement in the survey. The order of the tested schemes has been chosen randomly, but
was kept consistent across all statements. Same as inour online experiment evaluation, the pairwise statistical

The sentence generator scheme achieved the highest
user conﬁdence rating “making comparisons without any
mistakes” (S2 , p < 0.001 for all pairwise comparisons).
The participants’ perception is consistent with the experiment results where the word-list-based and sentence
generator schemes lead to higher attack detection rates.
12

204 25th USENIX Security Symposium

USENIX Association

The ratings for S4 illustrate that more complex representation schemes from the user’s point of view, such as
hexadecimal and Base32, are considered to be more secure by participants, even though all approaches provide
the same level of security.

5.3

performed. For instance, the numeric representation was
more suitable than hexadecimal and Base32. The raw
performance results suggest a similar speed for the numeric representation with a higher attack detection rate,and it also has received better usability ratings from endusers.
Our chunking experiment has shown that chunk-sizes
play only a minor role in improving attack detection rates(we could not ﬁnd statistically signiﬁcant differences).
However, if a hexadecimal representation is used chunksof 3 and 4 characters perform best.
As shown by the word list representations, the comparison speed can be increased by larger dictionaries leaving room for improvement in this area. Even though all
representation schemes provide the same level of security, exotic looking solutions are considered to be moresecure by end users.

Demographics

A total of 1047 users participated in the online study
while only 1001 have been considered in the evaluation
due to our two control questions. Out of the evaluatedparticipants, 534 participants were male, 453 were female, 4 chose other while the rest opted to not give any
information. No signiﬁcant difference between genders
could be found, with a subtle trend of a higher accuracy
for women and higher speed among men. The medianage was 34 (34.4 average) years, while 34 participantschose not to answer (no statistically signiﬁcant differences between ages).
A total of 39 people reported to have “medical conditions that complicated the security code comparisons(e. g., reading disorders, ADHD, visual impairments,etc.)” with a slightly higher undetected attack rate (statistically insigniﬁcant due to small sample size and thus
low statistical power).
The majority of the participants stated to have a Bachelor’s degree (399 of 1047) as their highest education
whereas 34% chose not to answer. 931 participants havestarted our HIT but stopped early during the experiment(mostly after the ﬁrst few comparisons). 160 users reported the general task to be annoying.

6

6.2

Most importantly, our study design does not test whether
end users are actually willing to compare any ﬁngerprints in practice. We only aim to study how easy different representations are to compare from the users’ pointof view.
As with any user study conducted with MTurk, there
is concern about the external validity of the results: users
in the real world might show different behaviour. Thisis mainly because of two reasons: (1) in practice ﬁngerprint comparisons will seldom occur in a such frequency,and (2) when performed in practice play a more important role than just participating in an anonymous online
study. Additionally, MTurkers have been shown to be
more tech-savvy and are better in solving textual and visual tasks in comparison to the average population. Thus,
they could have performed better in most of the comparison conditions than the average population. It is also
known that some MTurkers just “click through” studies
to get the fee and thus distort study results. Our counterbalanced study design with included control questions
and statistical signiﬁcance tests mitigate this effect. For
instance, we excluded 46 out of 1047 participants fromour main study part based on these questions being answered incorrectly.
Due to the within-group part of our factorial design,
many parameter choices such as different fonts, fontsizes, attack rates, etc. could not be considered. These
are, however, interesting avenues for future work. As
shown in our additional chunking experiment, anotherchallenge in testing different parameters is the high attack detection rate, where subtle changes would requirea high amount of users to produce statistically signiﬁcantresults.
Due to the anonymous nature of online studies, it isDiscussion

The results of our study show that while there are subtle
speed variations among all approaches, the attack detection rate and user perception for the current state-of-theart hexadecimal key-ﬁngerprint representation is significantly lower than those of most alternative representation schemes. Language-based representations (with the
exception of the PGP word list) show improved user behaviour leading to a higher detection rate of attacks. To
improve the usability of key-ﬁngerprints, we propose thefollowing takeaways based on our study results.

6.1

Limitations

Takeaways

Our results show that all representation schemes achievea high accuracy (high attack detection rate) and can be
performed quickly by users. As expected, languagebased ﬁngerprint representations are more resilient
against attacks (higher attack detection rate) and achieve
better usability scores. Among all conditions, alphanumeric approaches performed worse and have been out13
USENIX Association

25th USENIX Security Symposium 205

Acknowledgments

also impossible to reliably tell which languages a participant is ﬂuent in. We speciﬁed that we only wanted participants from English-speaking countries, however we
had no way of checking compliance except by relying
on self-reported data. Language-based representation approaches might induce additional barriers for non-native
speakers, e. g., due to unknown or unfamiliar words.

7

The authors would like to thank the anonymous reviewers for their insightful comments, and Trevor Perrin, Jake
McGinty, Tom Ritter and Skylar Nagao for their discussion and excellent feedback.

References[1] BADDELEY, A. Working memory. Science 255, 5044 (1992),
556–559.

Conclusion and Future Work

[2] BARNARD , G. Signiﬁcance tests for 2×2 tables. Biometrika(1947), 123–138.

We evaluated six different key-ﬁngerprint representation
types with regards to their comparison speed, attack detection accuracy and usability, which encompasses attack detection but also resilience against human errors
in short-term memory. An online study with 1047 participants was conducted to compare numeric, alphanumeric(Hexadecimal and Base32), word lists (PGP and Peerio),
as well as generated sentences representation schemes
for key-ﬁngerprint veriﬁcation. All ﬁngerprint representations were conﬁgured to offer the same level of security
with the same attacker strength.
Our results show that usage of the large word lists (asused in Peerio) lead to the fastest comparison performance, while generated sentences achieved highest attack detection rates. In addition, we found that additionalparameters such as chunking of characters plays only aminor role in the overall performance. The widely-used
hexadecimal representation scheme performed worst in
all tested categories which indicates that it should be replaced by more usable schemes. Unlike proposals whichcall for radically new ﬁngerprint representations, westudied only textual ﬁngerprint representations, whichmeans that the results of our work can be directly applied
to various encryption applications with minimal changes
needed. Speciﬁcally, no new hardware or complex software is required: applications merely need to replace thestrings they output to achieve a signiﬁcant improvement
in both attack-detection accuracy and usability.
There are various interesting areas of future work.
Firstly, we chose to study only a selected sample from thedesign space of ﬁngerprint representations in a withinsubjects design, so we could facilitate a direct comparison between the different classes of ﬁngerprints. Further work exploring line breaks, font settings, dictionaries, different attacker strengths, etc. will likely lead tofurther improvement possibilities.
While this work shows that there are better ways torepresent key-ﬁngerprints than currently being used, it
does not explore what can be done to motivate more users
to actually compare the ﬁngerprints in the ﬁrst place.
Follow-up studies to research this important question are
naturally an interesting and vital area of research.

[3] B IRYUKOV, A., D INU , D., AND K HOVRATOVICH , D. Argon2:
the memory-hard function for password hashing and other applications. Tech. rep., Password Hashing Competition (PHC), December 2015.
[4] B ONNEAU , J., AND S CHECHTER , S. Towards reliable storageof 56-bit secrets in human memory. In Proceedings of the 23rd
USENIX Security Symposium (August 2014).
[5] B REITMOSER , V. pgp-vanity-keygen. https://github.com/
Valodim/pgp-vanity-keygen, 2014.
[6] B UCKNER , R. L., P ETERSEN , S. E., O JEMANN , J. G., M IEZIN ,F. M., S QUIRE , L., AND R AICHLE , M. Functional anatomical
studies of explicit and implicit memory retrieval tasks. The Journal of Neuroscience 15, 1 (1995), 12–29.
[7] C ALLAS , J., D ONNERHACKE , L., F INNEY, H., S HAW, D., AND
T HAYER , R. OpenPGP Message Format. RFC 4880 (ProposedStandard), Nov. 2007. Updated by RFC 5581.
[8] C RANNELL , C., AND PARRISH , J. A comparison of immediate memory span for digits, letters, and words. The Journal of
Psychology 44, 2 (1957), 319–327.
[9] DE W INTER , J. C., AND D ODOU , D. Five-point Likert items: t
test versus Mann-Whitney-Wilcoxon. Practical Assessment, Research & Evaluation 15, 11 (2010), 1–12.
[10] D HAMIJA , R. Hash visualization in user authentication. In CHI’00 Extended Abstracts on Human Factors in Computing Systems(New York, NY, USA, 2000), CHI EA ’00, ACM, pp. 279–280.
[11] D IERKS , T., AND R ESCORLA , E. The Transport Layer Security
Protocol Version 1.2. RFC 5246, Aug. 2008. Updated by RFCs
5746, 5878, 6176.
F RONTIERF OUNDATION.
Secure[12] E LECTRONIC
Messaging
Scorecard.
https://www.eff.org/
secure-messaging-scorecard, 2014.
[13] E LLISON , C., ET AL . Establishing identity without certiﬁcation
authorities. In USENIX Security Symposium (1996), pp. 67–76.
[14] FARB , M., L IN , Y.-H., K IM , T. H.-J., M C C UNE , J., AND
P ERRIG , A. Safeslinger: easy-to-use and secure public-key exchange. In Proceedings of the 19th annual international conference on Mobile computing & networking (2013), ACM, pp. 417–
428.
[15] G ALBRAITH , J., AND T HAYER , R. The Secure Shell (SSH) Public Key File Format. RFC 4716 (Informational), Nov. 2006.
[16] G OODRICH , M. T., S IRIVIANOS , M., S OLIS , J., T SUDIK , G.,
AND U ZUN , E. Loud and clear: Human-veriﬁable authentication based on audio. In Distributed Computing Systems, 2006.
ICDCS 2006. 26th IEEE International Conference on (2006),
IEEE, pp. 10–10.
[17] G UTMANN , P. Do users verify SSH keys? USENIX;login: 36, 4(2011).

14
206 25th USENIX Security Symposium

USENIX Association

[38] ROGERS , M., AND P ERRIN , T.
Key-Fingerprint Poems.
https://moderncrypto.org/mail-archive/messaging/
2014/000125.html, 2014.

[18] H OLM , S. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65–70.
[19] H SIAO , H.-C., L IN , Y.-H., S TUDER , A., S TUDER , C., WANG ,K.-H., K IKUCHI , H., P ERRIG , A., S UN , H.-M., AND YANG ,B.-Y. A study of user-friendly hash comparison schemes. In
Computer Security Applications Conference, 2009. ACSAC’09.
Annual (2009), IEEE, pp. 105–114.

[39] RYAN , M. D. Enhanced certiﬁcate transparency and end-to-end
encrypted mail. In NDSS (2014), NDSS.
[40] S HAY, R., K ELLEY, P. G., KOMANDURI , S., M AZUREK , M. L.,
U R , B., V IDAS , T., BAUER , L., C HRISTIN , N., AND C RANOR ,L. F. Correct horse battery staple: Exploring the usability ofsystem-assigned passphrases. In Proceedings of the Eighth Symposium on Usable Privacy and Security (2012), ACM, p. 7.

[20] J OSEFSSON , S. The Base16, Base32, and Base64 Data Encodings. RFC 3548 (Informational), July 2003.
[21] J OSEFSSON , S. The Base16, Base32, and Base64 Data Encodings. RFC 4648, Oct. 2006.

[41] U NGER , N., D ECHAND , S., B ONNEAU , J., FAHL , S., P ERL ,H., G OLDBERG , I., AND S MITH , M. Sok: Secure messaging.
In Security and Privacy (SP), 2015 IEEE Symposium on (2015),
IEEE, pp. 232–249.

[22] J UOLA , P. Whole-word phonetic distances and the PGPFonealphabet. In Spoken Language, 1996. ICSLP 96. Proceedings.,
Fourth International Conference on (1996), vol. 1, IEEE, pp. 98–
101.

http://www.vasco.com/company/[42] VASCO . COM.
about_vasco/press_room/news_archive/2011/news_
diginotar_reports_security_incident.aspx,
Sept.
2011.

[23] K EITH , M., S HAO , B., AND S TEINBART, P. A behavioral analysis of passphrase design and effectiveness. Journal of the Association for Information Systems 10, 2 (2009), 2.

[43] W HATS A PP. Encryption Overview. https://www.whatsapp.
com/security/WhatsApp-Security-Whitepaper.pdf,
Apr. 2016.

[24] L AURIE , B., L ANGLEY, A., AND K ASPER , E. Certiﬁcate Transparency. RFC 6962 (Experimental), June 2013.
[25] L UND , O. Knowledge construction in typography: the case oflegibility research and the legibility of sans serif typefaces. PhD
thesis, The University of Reading, Department of Typography &Graphic Communication., 1999.

[44] Y LONEN , T., AND L ONVICK , C. The Secure Shell Transport
Layer Protocol. RFC 4253, Jan. 2006. Updated by RFC 6668.
[45] Z IMMERMANN , P., J OHNSTON , A., AND C ALLAS , J. ZRTP:Media Path Key Agreement for Unicast Secure RTP. RFC 6189
(Informational), Apr. 2011.

[26] M C G RAW, K. O., AND W ONG , S. A common language effectsize statistic. Psychological bulletin 111, 2 (1992), 361.
[27] M ELARA , M. S., B LANKSTEIN , A., B ONNEAU , J., F REED MAN , M. J., AND F ELTEN , E. W.
CONIKS: A PrivacyPreserving Consistent Key Service for Secure End-to-End Communication. Tech. Rep. 2014/1004, Cryptology ePrint Archive,
December 2014.
[28] [ MESSAGING ] M AILING -L IST A RCHIVE.
Usability of
Public-Key Fingerprints.
https://moderncrypto.org/
mail-archive/messaging/2014/000004.html, 2014.
[29] M ILLER , G. A. The magical number seven, plus or minus two:some limits on our capacity for processing information. Psychological review 63, 2 (1956), 81.
[30] NAMECOIN P ROJECT. Namecoin. http://namecoin.info,
Nov. 2014.
[31] O GDEN , C. K., ET AL . System of Basic English. Self-published(1934).
[32] O LEMBO , M. M., K ILIAN , T., S TOCKHARDT, S., H ÜLSING ,A., AND VOLKAMER , M. Developing and testing a visual hashscheme. In EISMC (2013), pp. 91–100.
[33] [ OPENPGP ] IETF M AIL A RCHIVE.
Fingerprints.
https://mailarchive.ietf.org/arch/msg/openpgp/
2C9gTsxTgh29W8VX8x70OYZqfUY, 2015.
[34] P ERCIVAL , C. Stronger key derivation via sequential memoryhard functions. Self-published (2009).
[35] P ERRIG , A., AND S ONG , D. Hash visualization: A new technique to improve real-world security. In International Workshop
on Cryptographic Techniques and E-Commerce (1999), pp. 131–138.
[36] P INGEL , I., I RVING , A., GENERALMANAGER , W IKINAUT, TIN LOAF, FARB , M., AND JPOPPLEWELL . Fingeprint exchange issue #826 - whispersystems/textsecure - github.
[37] P LASMOID. Fuzzy Fingerprints: Attacking Vulnerabilities inthe Human Brain. http://www.thc.org/papers/ffp.html,
Oct. 2003.

15
USENIX Association

25th USENIX Security Symposium 207

A

Appendix
I am conﬁdent that I can make comparisons using this method without making mistakes

Alphanumeric
Numeric
Unrelated Words
Generated Sentences

14%
7%
7%
5%

71%
82%
80%
87%

I found this method difﬁcult to use
Alphanumeric
Numeric
Unrelated Words
Generated Sentences

49%
67%
61%
71%

Alphanumeric
Numeric
Unrelated Words
Generated Sentences

5%
8%
8%
12%

Alphanumeric
Numeric
Unrelated Words
Generated Sentences

27%
13%
12%
8%

36%
18%
21%
14%

I think making comparisons using this method would help me keep my communications secure
82%
76%
75%
67%

I was able to do the comparisons very quickly with this method
54%
71%
71%
80%

Overall, I liked this method
Alphanumeric
Numeric
Unrelated Words
Generated Sentences

22%
12%
14%
11%

55%
64%
67%
69%

The comparisons were easy for me with this method
Alphanumeric
Numeric
Unrelated Words
Generated Sentences

25%
11%
11%
6%

58%
76%
72%
83%

100

Response

50

0

50

Percentage

Strongly Disagree

Disagree

Neutral

Agree

100

Strongly Agree

Figure 6: Survey results for all statement ratings

16
208 25th USENIX Security Symposium

USENIX Association