How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Elie Bursztein, Steven Bethard, Celine Fabry, John C. Mitchell, Dan Jurafsky
elie@cs.stanford.edu, bethard@stanford.edu, celinefabry@gmail.com, mitchell@cs.stanford.edu, jurafsky@stanford.edu
Stanford University
Abstract—Captchas are designed to be easy for humans but
hard for machines. However, most recent research has focused
only on making them hard for machines. In this paper, we
present what is to the best of our knowledge the first large
scale evaluation of captchas from the human perspective, withthe goal of assessing how much friction captchas present to theaverage user.
For the purpose of this study we have asked workers from
Amazon’s Mechanical Turk and an underground captchabreaking service to solve more than 318 000 captchas issued
from the 21 most popular captcha schemes (13 images schemes
and 8 audio scheme).
Analysis of the resulting data reveals that captchas are often
difficult for humans, with audio captchas being particularly
problematic. We also find some demographic trends indicating,for example, that non-native speakers of English are slower ingeneral and less accurate on English-centric captcha schemes.
Evidence from a week’s worth of eBay captchas (14,000,000
samples) suggests that the solving accuracies found in ourstudy are close to real-world values, and that improving
audio captchas should become a priority, as nearly 1% of all
captchas are delivered as audio rather than images. Finally ourstudy also reveals that it is more effective for an attacker touse Mechanical Turk to solve captchas than an undergroundservice.

I. I NTRODUCTION
Completely Automated Public Turing tests to tell Computers and Humans Apart (CAPTCHAs) are widely used by
websites to distinguish abusive programs from real humanusers. Captchas typically present a user with a simple test
like reading digits or listening to speech and then ask
the user to type in what they saw or heard. The image
or sound is usually distorted in various ways to make it
difficult for a machine to perform the test. When successful,captchas can prevent a wide variety of abuses, such as invalid
account creation and spam comments on blogs and forums.
Captchas are intended to be easy for humans to perform, anddifficult for machines to perform. While there has been muchdiscussion of making captchas difficult for machines (e.g.
[2], [4], [13]), to the best of our knowledge there has been
no large scale study assessing how well captchas achieve theformer goal: making it easy for humans to pass the test.
We address this problem by collecting captcha samples
from each of the 13 most used image schemes and 8 mostused audio schemes, for a total of over 318,000 captchas.

We then ask humans to solve these and analyze theirperformance.
Our current annotation efforts have resulted in over 5000
captchas for each image scheme and 3500 captchas for eachaudio scheme, each annotated by at least three different
humans from Amazon’s Mechanical Turk. We also had an
additional 1000 captchas from each image scheme annotatedthree times by an underground service which promises tomanually solve captchas in bulk. Based on an analysis ofthese captchas, we make a number of important findings:• Despite their goals, captchas are often hard for humans.
When we presented image captchas to three different
humans, all three agreed only 71% of the time on
average.
• Audio captchas are much harder than image captchas.
We found perfect agreement by three humans only 31%of the time for audio captchas.
• Some captcha schemes are clearly harder for humans
than others. For example, three humans agreed on 93%
of authorize image captchas, but only 35% of mail.ru
image captchas.
We obtained statistics from eBay regarding 14 million
eBay captchas delivered over a week. This additional data
corroborates our Mechanical Turk and underground captcha
service statistics and underscores the importance of audio
captchas:• Our Mechanical Turk assessment of eBay image
captchas is lower than eBay’s measured success rate:
our data shows 93.0% accuracy, compared to eBay’s
measured success rate of 98.5% on 14,000,000 eBaysite captchas.
• Evaluating the utility of audio captchas is important asthey account for almost 1% of all captchas delivered.
We also analyze human variations along a number ofdemographic lines and find some interesting trends:• Non-native speakers of English take longer to solve
captchas, and are less accurate on captchas that include
English words.
• Humans become slightly slower and slightly moreaccurate with age.
• Ph.D.s are the best at solving audio captchas.
Finally our study shows that for attackers, it is more

efficient to use Mechanical Turk to solve captchas than theunderground service, as it is cheaper and more accurate.
All these findings contribute to a better understanding of
the human side of captcha tests. The remainder of the paper
is organized as follows: In Sec II, we discuss our study
methodology. In Sec. III, we introduce the 13 image and 8
audio schemes we analyzed. In Sec. IV. we provide usage
statistics for eBay on a 14 million captcha corpus gathered
over a 7 day period. In Sec. V, we present the results of ourstudy. In Sec VI, we discuss how the user demographics
affect the captcha solving process. In sec. VII, we present
some additional related work. Finally, we conclude and give
future directions in sec. VIII.
II. S TUDY M ETHODOLOGY
We designed our study for two purposes: to collect information on the speed and accuracy with which humans solve
captchas, and to collect information about a broad range
of design and demographic factors that could potentiallyinfluence these results. To build our captcha corpus, wecollected eleven thousand captchas from the 21 most used
schemes: 13 image schemes and 8 audio schemes. In total
we scraped more than 90 000 captchas, as discussed in
section III below. For human subjects on which to test thesecaptchas, we relied on two sources: Amazon’s Mechanical
Turk and a underground captcha-breaking service called
Bypass-captcha.
A. Amazon’s Mechanical Turk
Amazon’s Mechanical Turk (AMT) is an online marketplace from Amazon where requesters can find workers
to solve Human Intelligence Tasks (HITs). This service isdesigned to tackle problems that are difficult for a machine
to solve but should be easy for humans. Essentially, the AMT
service is intended as a way of crowd-sourcing interesting
problems, and has been used extensively to collect annotateddata on a variety of tasks, including image classificationand filtering of porn for websites. Since AMT provides
easy access to human intelligene, it is the perfect service
to conduct the task of solving captchas, which is supposed
to be easy for humans and hard for computers.
Any task that can be presented as a webpage can becrowd-sourced through AMT, and workers will often perform complicated tasks for relatively small amounts ofmoney (for example, as little as $0.05 for solving oneof our captchas). AMT also has the advantage that theworkers (colloquially, “Turkers”) are real people and canbe asked demographic information such as age, educationlevel, native language, etc., which, as we discuss in section
VI, are important for understand how taxing captchas are on
different people.
In our experiment, we presented Turkers first with asurvey asking the following demographic information:Age
Native language (one from the Wikipedia list1 )• Education (one of: no formal education, high school,
bachelors degree, masters degree, Ph.D)• (If native language is not English) Years studying
English• Industry (one of the United States Bureau of Labor
Standard Occupational Classifications)• Country of birth• Country of residence• Years using the internet• Frequency of internet use (daily, weekly, monthly or
yearly)After filling out this survey, Turkers were then presented
with 39 image captchas or 24 audio captchas, one at a time,and asked to type in their answers. We built a task scheduler
to ensure that three Turkers (see Sec. V) saw each captcha
even though some Turkers gave up on some tasks. As aresult, we ended up effectively having more that 318 000
captchas annotated by Turkers. In particular we had a very
high give up rate, around 50%, for audio captchas as they
are tedious to solve.
Our task scheduler presented the captchas to Turkers in
two different ways to make sure the order did not influence
the results of the study:• Random Order: Fully randomly, where any captcha
from any scheme could follow any other.
• Blocks of Three: In blocks of three captchas fromthe same scheme, where the schemes were orderedrandomly.
For each captcha, we recorded the time a Turker took tosolve it and their final response2 . Turkers were then paid
anywhere from $0.02 to $0.50 for solving their full set of
39 image or 24 audio captchas.
••

B. Underground captcha-Breaking Service
We also investigated using a underground captchabreaking service, captcha-bypass.com. This service promisesthat captchas submitted to them will be solved by “qualified
specialists” for $0.005 per captcha. They used to provide
an web service that can be accessed via an application
programming interface (API) available for .Net, C++. PHP
and Java. Since it is a web based service, HTTP call canalso be used to interact with it from any language.
For the purpose of this study we used the PHP API
to collect data on accuracy and solving time. Of course,
the demographic information available through AMT is notavailable through this service. However, we performed thisexperiment because to the best of our knowledge, there
1 http://en.wikipedia.org/wiki/List of languages by number of native
speakers
2 We also recorded information about their computing environment,
including their browser and operating system, with the hope of using thisinformation in furture researchhave been no previous studies on how efficient undergroundservices are at solving captchas, and this experiment cantherefore shed some light on an unexplored corner of theblack market. Overall we submitted 39000 captchas to thisservice.

Baidu.

Baidu.com is the most popular Chinese search engine. Image
captchas from baidu.com consist of four black digits anduppercase letters on a white background. To obfuscate
the text, a wavy line is drawn across the characters andcharacters are individually tilted and placed so as to overlap
one another.
captchas.net.

Front page of the underground service we used.
III. CAPTCHA C OLLECTION
To run our study on humans, we first needed to collecta large number of captchas representative of what people
encounter on the web. We consulted the Alexa list of mostused websites3 and identified the top sites which presented
captchas as part of their account registration process. We
also collected captchas from sites which provide captchas toother sites, e.g. recaptcha.net and captchas.net. We looked
both for image captchas and for audio captchas, which aresometimes provided as an alternative to image captchas as anaccessibility measure. For each scheme, we collected 11,000
captchas. Tables I and II compare some of the features of
the captchas from each of these schemes, and the following
sections give a little more detail about the sites and theircaptchas.
A. Image captchas
Authorize.

Authorize.net is a gateway through which other sites can accept credit cards and electronic check payments ala Paypal.
Image captchas from authorize.net consist of five black digits
and lowercase letters on a white background. To obfuscate
the text, the character sequence as a whole is squeezed andtilted to varying degrees, and both the characters and thebackground are spotted lightly with gray dots.
3 http://www.alexa.com/topsites

captchas.net provides captchas to other sites such as
www.blog.de and www.blog.co.uk. Image captchas from
captchas.net consist of a white background with six black
lowercase letters. To obfuscate the text, individual characters
are randomly rotated and shifted up or down and spottedwith small white dots, and the background is spotted thickly
with small black dots.
digg.com.

Digg.com is a site where users can post web links and voteon the links posted by others that they find most interesting.
Image captchas from digg.com consist of five black digits
and letters (upper and lowercase) on a white and gray stripedbackground. To obfuscate the text, individual characters are
randomly rotated and shifted up or down, and a dense crosshatch of thin black and gray lines is drawn across the image.
eBay.

Ebay.com is the world’s largest online marketplace, whereusers can buy and sell almost anything. Image captchas from

Scheme
Min Len
Max LenChar set
Word

Auth.
5
5
a0
no

Baidu
4
4
0A
no

capt.
6
6
a
no

Digg
5
5
a
no

eBay
6
6
0
no

Ggle
5
10
a
pseudomail.ru
6
6
0aA
no

MS
8
8
0A
no

Recap.
5
20
0aA- !
yes

Skyrock
5
6
a0
no

Slash.
68ayes

Blizzard
6
8
a0
no

Y!
5
8
0aA
no

Table II MAGE CAPTCHA FEATURES

ebay.com consist of six letters in a dark color on a whitebackground. To obfuscate the text, individual characters are
randomly tilted, shifted up and down and placed so as tooverlap one another.

Google.

Microsoft.

Live.com is the server for Windows Live IDs, the user
accounts for Hotmail, MSN Messenger, Xbox LIVE, andother Microsoft sites. Image captchas from live.com consistof eight dark blue digits and uppercase letters on a graybackground. To obfuscate the text, characters are squeezed,
tilted and moved so that they touch each other, and thesequence is arranged in a wave.
Recaptcha.

Google.com is a popular search engine that provides manyother services, such as webmail. Image captchas fromgoogle.com consist of a white background with red, green
or blue lettering that forms a pseudo-word – a sequence ofcharacters that could probably be an English word, but isn’t– of four to ten lowercase letters. To obfuscate the text,
characters are squeezed, tilted and moved so that they toucheach other, and the character sequence is arranged in a wave.

Mail.ru.4

Recaptcha.net provides image captchas to a large numberof high profile sites, such as Facebook, Ticketmaster, andCraigslist. Captchas from recaptcha.net consist of two words(roughly 5-20 characters, according to the captcha answers
we collected) in black on a white background. To obfuscate
the text, the words are drawn from scanned books, where optical character recognizers failed on at least one of the words.
Additionally, the characters of both words are squeezed intoa wave-like shape.
Skyrock.

Mail.ru is the biggest free Russian webmail provider. Image
captchas from mail.ru consist of six blue outlines of lettersand numbers on a white background. To obfuscate the text,
characters are tilted, bent, moved up and down, and the entire
background is covered with hundreds of other outlines ofletters and numbers.
4 The Mail.ru captcha is scaled at 0.5

Skyrock.com is a social network and blogging site that is
popular in many French speaking countries. Image captchas
from skyrock.com consist of five to six dark colored digits
and lowercase letters on a lighter colored background. To
obfuscate the text, the characters are squeezed and thesequence is arranged in a wave.

Slashdot.

B. Audio captchas
When available for the sites presented above, we alsocollected their audio captchas. As a result we also studied
the following 8 audio captcha schemes.
Authorize.

Slashdot.org is a website with user-submitted and editorevaluated current affairs news, accompanied by forum-stylecomments. Image captchas from slashdot.org consist of asingle English word of six to eight lower case letters inblack on a white background. To obfuscate the text, somecharacters are filled while others are only outlines, a numberof zig-zag lines are drawn over all the letters, and small black
dots spot the background.

Blizzard.5

WorldOfWarcraft.com is the website for the popular online
game World of Warcraft (WoW), run by Blizzard Entertainment. Image captchas from worldofwarcraft.com consistof six to eight bright colored letters on a darker patternedbackground. To obfuscate the text, characters are slightly
tilted, squeezed and shifted up and down.

Audio captchas on authorize.net consist of a female voice
speaking aloud each of the five letters or digits of the image
captcha. The voice clearly articulates each character, andthere is minimal distortion. The example waveform andspectrogram are 3 second clips including the letters Q, 5
and V. These show that a good pause appears between
each spoken character, and that the vowel formants (thethick black waves in the middle of the spectrogram, which
are good indicators of both vowels and the surrounding
consonants) are clearly visible in each word.

Digg.

Yahoo.6

Yahoo.com is a popular search engine, portal and webmail
provider. Image captchas from yahoo.com consist of five toeight black digits and upper or lowercase letters on a whitebackground. To obfuscate the text, characters are squeezed,
bent and moved so that they touch each other, and thecharacter sequence is arranged in a wave.

5 The Blizzard captcha is scaled at 0.75
6 The Yahoo captcha is scaled at 0.75

Audio captchas on digg.com consist of a female voice
speaking aloud five letters. There is heavy white noisein the background, and sometimes an empty but louder
segment is played between letters. The example waveformand spectrogram are 3 second clips including the letters N,
L and an empty louder segment. The overall darkness of thespectrogram shows the heavy white noise which somewhat
obscures the vowel formants.

SchemeMin lenMax lenSpeaker
CharsetAvg. duration
Sample rate
Beep
RepeatAuthorize
5
5Female0-9a-z
5.0
8000
no
no

Digg
5
5
Female
a-z
6.8
8000
no
no

eBay
6
6
Various
0-9
4.4
8000
no
no

Google
5∗
15∗
Male
0-9∗
37.1
8000
yes
yes

Microsoft
1010
Various
0-9
7.1
8000
no
no

Recaptcha88
Various
0-9
25.3
8000
no
no

Slashdot
6
8
Male
Word
3.4
22050
no
no

Yahoo
7
7
Child
0-9
18.0
22050
yes
no

Table II
AUDIO CAPTCHA FEATURES

eBay.

Audio captchas on ebay.com consist of the same six digits
from the image captcha being spoken aloud, each by a different speaker in a different setting. The example waveform andspectrogram are 3 second clips containing the digits 6, 0, 7,
7 and 7 - note that the digits in these captchas are deliveredmuch faster than those of authorize.net or digg.com. The
waveform shows the variability of the various digits due todifferent speakers and different background noise levels, andthe spectrogram shows that the vowel formants are short andoften obscured by the noise.

interjected. The example waveform and spectrogram are 3
second clips containing the words “zero”, “it” and “oh” atthe beginning, middle and end of the segment. The spectogram shows how similar the formants from the backgroundnoise look to the true vowel formants, and of the three largest
amplitude wave groups in the waveform, only the first andthe last are actual words.
Note that because of the difficulty of this captcha, wewere unable to say with confidence how many digits were
presented, or even that the captcha was supposed to consist
only of digits. Thus, the entries in Table II for Google are
estimated from the answers we collected from our humansubjects.

Microsoft.

Google.

Audio captchas on google.com consist of three beeps, amale voice speaking digits aloud, the phrase “once again”,
and a repeat of the male voice speaking the digits. In the
background, various voices are playing simultaneously, andconfusing decoy words like “now” or “it” are occasionally

Audio captchas on live.com consist of ten digits being
spoken aloud, each by a different speaker over a low quality
recording, with various voices playing simultaneously in the
background. The example waveform and spectrogram are 3
second clips containing the digits 1, 4, 6, 5 and 0 - like theeBay audio captchas, these digits are delivered quite fast
to the user. While all the high amplitude sections of the
waveform correspond to the actual digits, the spectrogramshows that the vowel formants are somewhat obscured by
the background noise.

Recaptcha.

Audio captchas from recaptcha.net consist of eight digits
spoken by different speakers, with voices in the backgroundand occasional confusing words interjected. This is similar tothe live.com presentation, but the digits are delivered much
more slowly - the 3 second clip in the example waveformand spectrogram includes only the digit 6 and a confusing“eee” sound, with the next actual digit following about asecond after the end of this clip7 .

Slashdot.

Audio captchas from slashdot.org consists of a single word
spoken aloud, followed by the same word spelled letter byletter. The speech is generated by a text-to-speech system,with a computer-generated male voice. The example waveform and spectrogram are the entire 3 second clip containing
Leathers, L, E, A, T, H, E, R and S. The spectrogram showsthat the vowel formants are very clear (not surprsing, as they
were computer-generated), though the speed of the letterspelling speech in these captchas is among the fastest of all
the audio captchas we surveyed.

7 We used the default version supplied by the PHP API, but the recaptcha
webpage suggests they also have another audio captcha scheme based on
spoken words

Yahoo.

Audio captchas from yahoo.com consist of three beeps andthen a child’s voice speaking seven digits with various otherchild voices in the background. The example waveformand spectrogram are a 3 second clip containing the digits
7 and 8. The digits are the largest amplitude sections
on the waveform, though the spectrogram shows that thebackground voices look very much like the “real” speech.
IV. R EAL W ORLD U SAGE : E BAY DATA
Before testing humans on our corpus of captchas, we firstgathered some information about how captchas are usedin the wild. With our collaborators at eBay, we gatheredstatistics on how captchas were used on their site over thecourse of a week in 2009, as shown in Table IV. Over thecourse of a week, eBay provided nearly 14 000 000 captchas
to its users. Of these, over 200 000 were failed, suggestingthat an average eBay user answers their captchas correctly
98.5% of the time. Thus, in our study, we would expect tosee our subjects with captcha solving accuracies in roughly
this range.
Another interesting aspect of the eBay statistics is theusage of audio captchas. Of the 14 000 000 captchas, morethan 100 000 were audio captchas, indicating that 0.77% of
the time, users prefer to have audio captchas over image
captchas. This is actually a surprisingly large percentage
compared to our expectations, and shows the importanceof studying human understanding of both audio and image
captchas.
V. E XPERIMENTAL R ESULTS
From our corpus of captchas, we presented 5000 image
captchas and 3500 audio captchas to Turkers, and 1000
image captchas to the underground service following ourstudy methodology above. To allow for some analysis ofhuman agreement on captcha answers, we had three subjects
annotate each of the captchas, for both Turkers and theunderground service. As a result we have annotated 195 000
image captcha and 84 400 audio captcha by the Turker and
39 000 captchas by the underground service. Overall thisstudy is based on more than 318 000 captcha.

Day 12079120
15592
31709
0.75%
1.52512

Total captcha
Audio captcha
FailureAudio ratio
Failure ratio

Day 22041526
15476
30179
0.76%
1.47826Day 31902752
14370
28475
0.75%
1.49652

Day 41825314
14482
28484
0.79%
1.56050Day 51825314
14482
28484
0.79%
1.56050Day 62178870
16412
33516
0.75%
1.56050Day 72104628
16578
32564
0.79%
1.54726

Table IIIE BAY CAPTCHA STATISTICS FOR A WEEK IN NOVEMBER 200960

Number of seconds

50

when captchas are not frequently presented, the 20 or moreseconds taken by audio captchas may present a substantial
annoyance for users.

Known correct
Overall
Known incorrect

40

30

20
21.62

10
8.94

0

22.44

25.45

28.35

28.70

22.25

10.13

9.8

Im
a

ge

Tu
rk

Im
a

ge

Au
By
pa

ss

dio

Tu
rk

Figure 1. Average solving time for image and audio captchas, for Turkers
and the underground service.

We focus on two primary research questions when analyzing the resulting data: how much inconvenience (userfriction) does a typical captcha present to a user, and how
different captcha schemes affect users with different backgrounds. The following sections explore these two questions
in detail.
A. Captcha Friction
Captchas are meant to be a quick and easy verification thata user is in fact a human. Captchas that are time consuming
or difficult for humans represent friction or inconvenience
for the user. We consider some forms of captcha friction in
the following sections.
1) Solving Time: One simple measure of friction is thetotal time a user spends on a given captcha, and Figure 1
shows these statistics.
Overall, we see that image captchas take Turkers about
9.8 seconds to view and solve, while audio captchas take
Turkers about 28.4 seconds to hear and solve8 . While 5-10
seconds is probably an acceptable inconvenience for a user
8 The underground service takes around 22.44 seconds to solve image
captchas, but we can only measure the turnaround time through their API,
which may include some overhead for routing, etc.

In the cases where we believe we know the correct
answer for a captcha, we can compare the time it takes
a user to give the correct answer with the time it takes
them to give an incorrect answer. Across the Turkers, theunderground service, and both image and audio captchas,
we see that correct answers are given more quickly thanincorrect answers, though this difference is most pronounced
for audio captchas, where correct answers are 6.5 seconds
faster than incorrect ones. This is another argument formaking sure that the captcha scheme used is sufficiently easy
for humans, because the more they fail, the longer they’ll
be spending on the captchas.
It is worth noting that for all these mean solving times, thestandard deviations are quite large. This is even true after
we remove outliers greater than three standard deviations
from the mean, as we do for all of our graphs. This is thestandard flaw of taking timing measurements on the internet– people are always free to stop midway through and go
do something else for a while if they like (even though theTurkers were explicitly requested not to).
Figures 2 and 3 show histograms of solving times for ourusers, demonstrating the very long tail of this distribution,
but also showing that there is a clear clustering of most users
in the 5-15 second range.
Figures 2 and 3 also show how the solving times differ
for the different schemes. For image captchas, we seethat mail.ru and Microsoft captchas take the most timefor Turkers, with means around 13 seconds. The fastest
image captchas were from Authorize, Baidu and eBay whichtake on average around 7 seconds each. In audio captchas,
Google, reCaptcha and Yahoo captchas were the most time
consuming, with means over 25 seconds, while Authorize,
eBay and Slashdot all averaged 12 seconds or less. Notethat these timings closely track the average duration of eachscheme as shown in Table II, indicating that audio captcha
solving time is dominated by the time spent listening to
the captcha. These results suggest that careful selection of acaptcha scheme can substantially reduce the friction of thecaptcha system on the user.

authorize
baidu
captchas.net
digg
ebay
googlemailrumsliverecaptcha
skyrockslashdot
blizzard
yahoo

200
0
200
0
100
0
200
0
200
0
100
0
50
0
100

70%
60%
50%
40%
71.01

0
100

20%
10%

Figure 4.
answers

0
200

67.17

30%

0%

0
100

1 distinct answer
2 distinct answer
3 distinct answer

31.18

28.09

23.22
5.77

Image Turk

35.24 33.6

4.74

Image Bypass

Audio Turk

Percents of image and audio captchas given 1, 2 or 3 distinct

0
100
0
100
0

1

2

3

4

5

6

Figure 2.

78

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Solving times for image captcha schemes

Authorize
Digg
eBay
GoogleMicrosoft
Recaptcha
Slashdot
Yahoo

50
0
50

0
50
0
50
0
50
0
50
0
50

0
50

0

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Figure 3.

Solving times for audio captcha schemes

2) Solving Agreement: Another source of friction worth
considering is the inconvenience of having to solve anothercaptcha if a human can’t guess the correct answer to the firstone. Since we collected the captchas rather than generatingthem ourselves, we do not know the correct answer for each
captcha, so we cannot determine with certainty when a user
gets a captcha right or wrong. However, we can get anapproximation of this by looking at the number of distinct
answers given for each captcha. On a captcha that is easy forhumans, all three subjects annotating the captcha will agree
on the answer, while on a difficult captcha, subjects will
disagree and more than one distinct answer will be given.
Thus, the greater the number of distinct answers, the greater
the disagreement and the harder the captcha.
Figure 4 shows the percent of image and audio captchasthat were given 1, 2 or 3 distinct answers by our subjects.
Both the Turkers and the underground service reach similar
numbers for overall agreement on image captchas, with all
three subjects agreeing on a single answer for around 70% of
all captchas. Only about 5% of the image captchas were sobad that we got a different answer from each of the subjects.
Audio captchas are a total different story: All subjects
agree on only 31.2% of these captchas, and on 33.6%
everyone had a different answer. These results imply thatmany audio captchas are just too difficult for humans.
As with solving time, we also see differences by scheme
when looking at number of distinct answers. Figures 5, 6
and 7 show percents of distinct answers for each scheme.
Authorize.net has the easiest image captchas for humans,with three subjects agreeing on the answer over 93% of thetime. On the other end of the spectrum is the extremely difficult mail.ru image captchas, which have perfect agreement
among subjects less than 35% of the time. On the audio side,
Google, Microsoft, and Digg audio captchas are by far themost difficult, with all subjects producing the same answer
only about 1% of the time. Some of this difficulty with audio
captchas may be because we give no instructions on whatkinds of words or letters to expect, but this is consistent withfor example the Google interface, which at the time of thiswriting was just a single button beside the image captcha
text box that used Javascript to play the recording9 . Again,
we see that captcha friction on users can be substantiallyreduced by selecting a different captcha scheme.
9 The alternate reCaptcha audio captchas, not tested in this study, which
do not include intentional distortion may better agreement on these.

Yahoo5.2632

Yahoo
Blizzard

0.87449

0.89514

87.65
87.11

24.3887.89741.9013Digg

24.78786.37282.271

Digg

12.6151.012978.49532.11341.65539.232

Authorize

59.94919.11316.31980.9780%0.23095
6.51780%36.41738.797eBay
66.71819.0312.7027Baidu

4.18750.40965

41.00434.60825.3857.9385

captchas.net

95.403

Google53.24815.8272.474510.823
1.526433.633Mail.ru

Microsoft

38.56542.747

13.119eBay

66.62
25.512
7.867868.11718.689Google39.47432.486

Recaptcha

11.995Microsoft

28.04

Slashdot

85.49426.132

Recaptcha

Authorize

68.705

13.6325.7506

SlashdotSkyrock

26.032

1 answer
2 answer
3 answer

1 answer
2 answer
3 answer

29.19736.43934.363

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

93.251

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Figure 7. Percents of audio captchas given 1, 2 or 3 distinct answers by
Turkers for each scheme

Figure 5. Percents of image captchas given 1, 2 or 3 distinct answers by
Turkers for each scheme

1 answer
3.8934Yahoo

3.8462

Blizzard

19.3061.47785.6893Mail.ru

79.55720.0482.7311eBay

4.3678Baidu

0.691240%70.667

16.59780.67230.54314.139Captchas.net

68.49
37.44
42.51224

5.2036

Digg

40.57637.69425.8215.3333

Google46.85518.966

Recaptcha

Authorize

71.63533.83921.729Microsoft

3 answer

70.49224.519SlashdotSkyrock

2 answer

25.61564.25342.00843.85221.83973.79313.82520%

85.48440%3) Optimistic Solving Accuracy: As a final measure ofcaptcha friction, we wanted to calculate how often a humancan expect to pass the captcha challenge. Again, we don’tknow all the answers for the captchas we collected, but wecan make an approximation: if the three subjects produced
the same answer, we assume that answer is correct. For two
distinct answers we make the optimistic assumption that
2 out of 3 answers are correct and when the three users
disagree make the optimistic assumption that 1 out of the 3answers is correct.

60%

80%

Figure 6. Percents of image captchas given 1, 2 or 3 distinct answers by
the underground service for each scheme

Because our accuracy measurement is not perfect, looking
at the absolute values may be misleading, but differences insolving accuracy in different scenarios should still reflect
real differences.
Figure 8 shows the optimistic solving accuracy for image
and audio captchas through both Mechanical Turk and theunderground service. Even with our optimistic accuracy, theunderground service achieves only 84% accuracy on image
captchas, while Turkers achieve 87%. We see again thataudio captchas are the most difficult, with Turkers solving
them correctly only 52% of the time. Figure 8 also compares
the optimistic solving accuracy across different schemes.
Among image captchas, both Turkers and the undergroundservice are worst on the mail.ru captchas, achieving 61%
and 70% accuracy respectively. The easiest captchas are theauthorize.net captchas, where Turkers achieve 98% accuracy
and the underground service achieves 95% accuracy. Notethat these results mostly parallel what we saw in the analysis
of solving time and solving agreement, where for example,

Image Turker

Image Bypass

0.52

ALL

Audio Turker

0.84
0.87
0.68

Yahoo

0.89
0.88

Blizzard

0.89
0.68

Slashdot

0.76

0.87

Skyrock

0.93
0.95
0.47

Recaptcha
Microsoft

0.72
0.75

0.38

0.61

0.7

0.35

0.88
0.86
0.63

eBay
Digg

0.88

0.8

Mail.ru
Google

0.95

0.93
0.93

0.38

0.86

Captchas.net

0.77

Baidu

0.92

0.84
0.90
0.93

0.59

Authorize
0.4

0.5

0.6

0.95
0.98

0.7

0.8

0.9

1.0

SchemeAuthorize
Baidu
Captchas.net
Digg
eBay
Googlemail.ru
Microsoft
Recaptcha
Skyrock
Slashdot
Blizzard
Yahoo
Authorize audio
Digg audio
eBay audio
Google audio
Microsoft audio
Recaptcha audio
Slashdot audio
Yahoo audio

Time
6.8
7.1
8.2
8.2
7.3
9.7
12.8
13
11.9
7.9
7.7
9.3
10.6
11.9
14.8
11.8
35.2
16.6
30.1
11.7
25

Accuracy
0.98
0.93
0.84
0.92
0.93
0.86
0.7
0.8
0.75
0.95
0.87
0.95
0.88
0.59
0.38
0.63
0.35
0.38
0.47
0.68
0.68

Expected time
6.9
7.6
9.8
8.9
7.8
11.3
18.3
16.3
15.8
8.3
8.8
9.8
12
20.2
39
18.8
100.6
43.8
64.1
17.2
36.8

Table IV
E XPECTED SOLVING TIME FOR EACH SCHEME

Figure 8. Optimistic solving accuracy for image and audio captchas,overall and for each scheme

live.com and mail.ru are also among the captcha schemeswith the slowest responses and the worst agreement.
For eBay image captchas, we see an accuracy of 93%,
which is a little less than the 98.5% found in the statistics
collected by eBay. This is probably because on the eBaysite the user may ask for a new captcha before solving
it if they think the current one is too hard. That ourapproximate measurements are at least within a few points ofthe eBay measurement suggests that our solving accuracies
are reasonable.
Figure 8 shows optimistic solving accuracy for all audio the schemes, in blue. Google is the hardest of theseschemes, with Turkers achieving only 35% accuracy, while
slashdot.com and yahoo.com are the easiest schemes, withTurkers achieving 68% accuracy for both of them. Overall,these numbers track closely our earlier results looking at
solving agreement, and suggest the need for further research
to investigate what makes an audio captcha easy or hard.
4) Expected Solving Time: The previous sections haveshowed a number of ways of measuring captcha friction onusers. One way of unifying these all into a single measureis through expected solving time. The expected solving timeof a captcha is the total amount of time a user should expect
to spend, including not just the time to respond to the initialcaptcha, but also any time required to respond to additional
captchas if the first is failed. Expected time can be measured
by the following infinite summation, where t is the time ittakes to answer a single captcha, and a is the user’s solving
accuracy:

est(t, a) = t + (1 − a)(t + (1 − a)(t + (1 − a)(t + . . .)))
= t + t(1 − a) + t(1 − a)2 + . . .
t=aEssentially, we always pay the cost of the initial captcha,
and then we pay a fractional cost for additional captchas thatis weighted by our chance of answering incorrectly. As an
example of expected solving time in action, a user solving a
Microsoft audio captcha, which takes on average 13 seconds,
and on which users have optimistic solving accuracy of 0.8, the expected solving time is actually 16.3 seconds 25%
longer than the single captcha time would suggest. Table IVshows optimistic solving accuracies for all captcha schemes.
VI. CAPTCHAS AND U SER BACKGROUND
Having demonstrated that captchas are often quite difficultfor humans, we turn to the question of why. Certainly
the various distortion methods used to to increase captcha
difficulty for computers play a role, but in this study wefocus on the characteristics of the people, not the captchas,that predict captcha difficulty. Thus, we rely primarily on the
demographic data we collected from Turkers to investigate
these questions.
Overall we had more than 11800 demographic surveys
completed by Turkers. Since we authorized Turkers tocomplete up to five image and audio tasks for us, it is likelythat some of these surveys are duplicates. However evenin the worst case, we had more than 1100 different peopleanswering our questions.

Native speakers

708

686
727

658
647

700

13.7
11.5
13.5

63
70
46
26
56
36
42
26
58
35
26
18
11
10
21
2
6
1
2
16
15
2
2
1

103

Mail.ru

18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 66 67 68 71 72

Figure 9.

12.8
9.4

Microsoft

38
37

100

8.3
7.0

Recaptcha
268
182
177
133
230
106
113
137
91
132
76
104

200

6.2

Skyrock

318
361
388
364
351

429
205
239

Number of users

400

0

8.3
8.3

Slashdot

500

300

9.4
9.7

Blizzard

600

11.1
10.4

Google

8.0
7.5

eBay

6.7
8.6

Digg

Mechanical Turk worker age distribution

7.2
8.7

captchas.net

7.0
7.4
6.3
7.1
5.9

Baidu
Authorize
Other
Russian
BalochiPortuguese
Hebrew
PunjabiVietnamese
Bikol
CebuanoArabicMacedonian
Dutch
FrenchGermanGujarati
SloveneMarathi
Mandarin
BengaliKannadaSpanishRomanian
TeluguHindi/Urdu
MalayalamEnglish
Tamil

149
12
13
15
15
15
15
16
17
19
21
21
23
28
30
33
39
51
52
64
71
95

0s

2s

4s

6s

8s

10 s

12 s

14 s

Figure 12. Average image captcha solving time for native and non-native
speakers of English.

331
578
625
2791
3502

0

Non-native speakers

11.1

Yahoo

800

500

Figure 10.

1000

1500

2000

2500

3000

3500

Mechanical Turk worker native language

PhD
1%

No Formal
2%

Master
16%

High
School26%

Bachelor
55%

Figure 11.

Education repartition.

4000

First, it is useful to get a general picture of the Turkers
that worked for us. Figure 9 shows the age distribution of
our workers - the average age was around 29 years, andalmost half of our workers were 25 or younger. Figure 10
shows the native language distribution - Tamil and English
accounted for 40% and 32% of the languages, respectively,
leaving only 28% distributed to all other languages. Finally,
Table 11 shows the distribution of education levels showingthat over 72% of our workers had a bachelor’s degree orhigher. Overall, our Turkers represent a young, educated,primarily Tamil and English speaking population.
Our first question was about the effect of native language:
can non-native speakers of English perform as well as
native speakers on captchas? Figures 12 and 13 show thesolving time and solving accuracy for native and non-native
speakers, broken down by image captcha scheme. Overall
on image captchas, native speakers are substantially fasterand slightly more accurate. Looking across the schemes, theschemes that use real words, like recaptcha, and pseudowords, like Google, are solved far faster (up to 30%) by
native English speakers. This last point is important as
it suggests that captcha schemes that rely on extensive
experience with a single language can be biased against users
who are not native speakers of that language.
Considering audio captchas, we still see that non-native
speakers of English are usually somewhat slower than native speakers, though for reCaptcha the times are roughly
comparable. As for image captchas, solving accuracy islower across the board for non-native speakers. Moreover,the language bias is again illustrated by the the Slashdot

Native speakers

Non-Native speakers

0.881
0.874

Yahoo

0.954
0.956

Skyrock
0.738

Recaptcha

0.67
0.71
0.65

Slashdot

0.73

0.772

0.45

Recaptcha

0.804
0.794

Microsoft
0.700
0.704

0.50
0.37

Microsoft

0.39

0.861
0.873

Google
eBay
Digg

0.935
0.935
0.919
0.925

Google

0.927
0.928

Digg

0.35
0.35
0.62

eBay

0.65

0.837
0.848

captchas.net
Baidu

0.976
0.979

Authorize
0.70

0.75

0.80

0.85

0.90

0.95

0.37
0.39
0.57

Authorize

0.63

1.00
0.2

Figure 13. Average image captcha optimistic solving accuracy for nativeand non-native speakers of English.

Native speakers

Non-native speakers

Yahoo

0.867
0.890

Slashdot

Mail.ru

Native speakers

0.946
0.955

Blizzard

0.3

0.4

0.5

0.6

0.7

Figure 15. Average audio captcha optimistic solving accuracy for nativeand non-native speakers of English.

Non-native speakers

27.27

Yahoo

25.00
18.39

Slashdot

20s

11.70

18s

30.63

Recaptcha

16s

30.12

14s

19.48

Microsoft

12s

16.63

10s

33.21

Google

35.20
14.49

eBay

6s
4s

11.84

20

30

40

50

60

70

18.89

Digg

14.83

Figure 16.

Solving time for Turkers of different ages.

15.41

Authorize
0s

8s

11.94

5s

10 s

15 s

20 s

25 s

30 s

35 s

Figure 14. Average audio captcha solving time for native and non-native
speakers of English.

1.00%
0.95%
0.90%
0.85%

audio captchas, which are based on spelling English words:
the solving time for non-native speakers is 18.39 seconds,
7.4 seconds longer (57%) than the expected solving time
for native speakers. Thus, some significant work needs tobe done to make audio captchas reasonable for globally
deployed web applications.
We also investigated questions of aging: how does performance differ for old and young users? Figures 16 and17 show that there is a lot of variation in both solving

0.80%
0.75%
0.70%
0.65%
20

Figure 17.

30

40

50

60

70

Optimistic solving accuracy for Turkers of different ages.

60
Solving time for image
Solving time for audio
50

seconds

40

30

20

10

19.75

0

23.67

19.44

9.6

8.49

9.36

Not formalHigh School

Bachelor

23.25
9.16

21.33
7.64

Masterslower.
Of course, we are interested not just in speed, but alsoin accuracy, and so Figure 19 shows optimistic solving
accuracy for Turkers with different levels of education.
Overall the correlation between having a higher education
and being more efficient at solving captchas does not seemto hold in our data. However, we are relying on self reported
education levels in this work, and it is possible that there is
therefore noise in our data as some Turkers may perceive
reporting a higher education level as an opportunity to get
offered better tasks on Mechanical Turk.

Phd

VII. A DDITIONAL R ELATED W ORK
Figure 18.

Solving time for Turkers of different education levels.

Image captcha

Audio captcha

0.8

0.7

0.6

0.5

0.88

0.87

0.88

0.87

0.85

0.4
0.54

0.52

Not formal High School

Bachelor

0.51

0.51

0.54

0.3

0.2
MasterPhd

Figure 19. Optimistic solving accuracy for Turkers of different educationlevels.

time and solving accuracy for users of all ages. However
there are small trends visible in these graphs - each year,
users slow down by about 0.01 seconds, and become moreaccurate by about 0.01%. These findings are in line withsome psychological research on aging [15] where older
people are found to be more accurate than younger subjects
who demonstrate greater speed but more errors.
Finally, we looked into the effect of education: do users
with more or less education have an advantage in solving
captchas? Figure 18 shows solving time for Turkers withdifferent levels of education. There is a small decrease insolving time of image captchas as higher levels of education
are obtained, starting at 9.6 seconds for Turkers with noformal education, and dropping to 7.64 seconds for Turkers
with a Ph.D. With audio captchas, on the other hand, moreeducation doesn’t seem to make people faster: Turkers with
only a high school education were faster than Ph.D.s, andTurkers with bachelors or masters degrees were substantially

The closest work to our is that of [3] where they looked
at usability issues in presenting audio captchas to humans.
Though they only used 10 captchas from each of 10 sites,
they also found audio captchas to be more time consuming
and difficult than image captchas, and introduced a new userinterface to make audio captchas more accessible.
The first discussion of the captcha idea appears in [22],though the term CAPTCHA was coined in [24]. Text/image
based captchas have been studied extensively [6], [18], [19]and there is a long record of successful attempts at breaking
captchas of popular sites [8]. For example in March 2008, a
method to break 60% of MSN visual captchas was disclosed[26]. One of the most famous visual captcha breakers is
PWNtcha [16]. In [14], Hernandez-Castro and al use a sidechannel attack to break labeling captcha. In [12], Golle
use learning attack to break the Asirra scheme. Tam andhis colleagues [23] built a breaker for audio captchas fromgoogle.com, digg.com and an older version of recaptcha.net.
In [4], Cain et. al. studied the relation between captchas
and network security. In [25], Yan et. al. used naive patternrecognition to break image captchas. In [1], Athanasopoulos
et. al. used animation as captchas. A comparison of humanand computer efficiency to recognize single characters was
explored in [5]. Many ways of building captchas havebeen proposed [7], [9]–[11], [17]–[19] Finally, many othertechniques have been used to break captchas [20], [21], [27]VIII. C ONCLUSION
We have presented a large scale study of how muchtrouble captchas present for humans. We collected 5000
captchas from each of 13 most widely used image captcha
schemes and 3500 captchas from the 8 most widely usedaudio captcha schemes, and had them each judged bymultiple human subjects from Amazon’s Mechanical Turk
and an underground captcha-breaking service. Overall, we
found that captchas are often harder than they ought to be,with image captchas having an average solving time of 9.8
seconds and three-person agreement of 71.0%, and audio
captchas being much harder, with an average solving timeof 28.4 seconds, and three-person agreement of 31.2%. We
also found substantial variation in captcha difficulty acrossschemes, with authorize.net image captchas being among
the easiest, and google.com audio captchas being among
the hardest. We observed that the workers from Mechanical
Turk were quicker and more accurate than those from theunderground service, and were also willing to solve captchas
for smaller amounts of money.
Using the data collected from Amazon’s Mechanical Turk,
we identified a number of demographic factors that havesome influence on the difficulty of a captcha to a user.
Non-native speakers of English were slower, though they
were generally just as accurate unless the captcha required
recognition of English words. We also saw small trends indicating that older users were slower but more accurate. These
results invite future research to more deeply investigate how
individual differences influence captcha difficulty.
R EFERENCES[13] C.J. Hernandez-Castro and A.: Ribagorda. Pitfalls in captcha
design and implementation: the math captcha, a case study.
http://dx.doi.org/10.1016/j.cose.2009.06.006, 2009.
[14] C.J. Hernandez-Castro, A. Ribagorda, and Y. Saez. Sidechannel attack on labeling captchas. http://arxiv.org/abs/0908.
1185, 2009.
[15] Terence M. Hines and Michael I. Posner. Slow but sure: A
chronometric analysis of the process of aging. In The 84th
Annual Convention of the American Psychological Association, 1976.
[16] Sam Hocevar. Pwntcha captcha decoder. web site, http://sam.
zoy.org/pwntcha.
[17] M.E. Hoque, D.J. Russomanno, and M. Yeasin. 2d captchas
from 3d models. In IEEE SoutheastCon 2006, 2006.

[1] E. Athanasopoulos and S. Antonatos. Enhanced captchas:
Using animation to tell humans and computers apart. In IFIP
International Federation for Information Processing, 2006.

[18] P Simard K Chellapilla, K Larson and M Czerwinski. Building segmentation based human- friendly human interaction
proofs. In Springer-Verlag, editor, 2nd Int’l Workshop on
Human Interaction Proofs, 2005.

[2] H.S. Baird and T. Riopka. Scattertype: a reading captcha
resistant to segmentation attack. In IS & T/SPIE Document
Recognition & Retrieval Conference, 2005.

[19] P Simard K Chellapilla, K Larson and M Czerwinski. Designing human friendly human interaction proofs. In ACM,
editor, CHI05, 2005.

[3] Jeffrey P. Bigham and Anna C. Cavender. Evaluating existingaudio captchas and an interface optimized for non-visualuse. In ACM Conference on Human Factors in Computing
Systems, 2009.

[20] G. Mori and J. Malik. Recognizing objects in adversarial
clutter: Breaking a visual captcha. In CVPR 2003, pages
134–144, 2003.

[4] A. Caine and U. Hengartner. The AI Hardness of CAPTCHAs
does not imply Robust Network Security, volume 238.
[5] K. Chellapilla, K. Larson, P.Y. Simard, and M. Czerwinski.
Computers beat humans at single character recognition inreading based human interaction proofs (hips). In CEAS,
2005.
[6] K Chellapilla and P Simard. Using machine learning to breakvisual human interaction proofs. In MIT Press, editor, Neural
Information Processing Systems (NIPS), 2004.
[7] M. Chew and H.S. Baird. Baffletext: a human interactive
proof. In 10th SPIE/IS&T Doc. Recog. Retr. Conf, DRR 2003,
2003.
[8] Dancho Danchev. Microsoft’s captcha successfully broken.
blog post http://blogs.zdnet.com/security/?p=1232, May 2008.
[9] R. Datta. Imagination: A robust image-based captcha generation system. In ACM Multimedia Conf., 2005.
[10] Anne Eisenberg.
New puzzles that tell humans frommachines.
http://www.nytimes.com/2009/05/24/business/
24novelties.html? r=1&ref=technology, May 2009.
[11] J. Elson, J.R. Douceur, J. Howell, and J. Saul. Asirra: A
captcha that exploits interest-aligned manual image categorization. In 4th ACM CCS, 2007.
[12] P. Golle. Machine learning attacks against the asirra captcha.
In ACM CCS 2008, 2008.

[21] G. Moy. Distortion estimation techniques in solving visual
captchas. In CVPR 2004, 2004.
[22] Moni Naor. Verification of a human in the loop or identification via the turing test. Available electronically: http://www.
wisdom.weizmann.ac.il/∼naor/PAPERS/human.ps, 1997.
[23] Simsa Tam, J., S. J., Hyde, and L. Von Ahn. Breaking audio captchas. http://www.captcha.net/Breaking Audio
CAPTCHAs.pdf.
[24] L. von Ahn, M. Blum, N. J. Hopper, and J. Langford.
Captcha: Using hard ai problems for security. In Sringer,
editor, Eurocrypt, 2003.
[25] J. Yan and A.S.E. Ahmad. Breaking visual captchas withnaive pattern recognition algorithms. In ACSAC 2007, 2007.
[26] Jeff Yan and Ahmad Salah El Ahmad. A low-cost attack on
a microsoft captcha. Ex confidential draft http://homepages.
cs.ncl.ac.uk/jeff.yan/msn draft.pdf, 2008.
[27] H. Yeen. Breaking captchas without using ocr. http://www.
puremango.co.uk/2005/11/breaking captcha 115/.