B reak in g Aud i o CAPTCHAs
Jennifer Tam
Computer Science Department
Carnegie Mellon University
5000 Forbes Ave, Pittsburgh 15217
jdtam@cs.cmu.edu

Jiri Simsa
Computer Science Department
Carnegie Mellon University
5000 Forbes Ave, Pittsburgh 15217
jsimsa@cs.cmu.edu

Sean HydeElectrical and Computer Engineering
Carnegie Mellon University
5000 Forbes Ave, Pittsburgh 15217
sean.a.hyde@gmail.com

Luis Von Ahn
Computer Science Department
Carnegie Mellon University
5000 Forbes Ave, Pittsburgh 15217
biglou@cs.cmu.edu

Abstract
CAP TCHAs are computer-generated tests that humans can pass but current
computer systems cannot. CAP TCHAs provide a method for automaticallydistinguishing a human from a computer program, and therefore can protect
Web services from abuse by so-called “bots.” Most CAP TCHAs consist of
distorted images, usually text, for which a user must provide somedescription. Unfortunately, visual CAP TCHAs limit access to the millionsof visually impaired people using the Web. Audio CAP TCHAs werecreated to solve this accessibility issue; however, the security of audio
CAP TCHAs was never formally tested. Some visual CAP TCHAs havebeen broken using machine learning techniques, and we propose using
similar ideas to test the security of audio CAP TCHAs. Audio CAP TCHAs
are generally composed of a set of words to be identified, layered on top ofnoise. We analyzed the security of current audio CAP TCHAs from popular
Web sites by using AdaBoost, SVM, and k-NN, and achieved correct
solutions for test samples with accuracy up to 71%. Such accuracy isenough to consider these CAPTCHAs broken. Training several different
machine learning algorithms on different types of audio CAP TCHAs
allowed us to analyze the strengths and weaknesses of the algorithms sothat we could suggest a design for a more robust audio CAPTCHA.

1

Int rod uct i o n

CAP TCHAs [1] are automated tests designed to tell computers and humans apart bypresenting users with a problem that humans can solve but current computer programs
cannot. Because CAPTCHAs can distinguish between humans and computers with highprobability, they are used for many different security applications: they prevent bots fromvoting continuously in online polls, automatically registering for millions of spam email
accounts, automatically purchasing tickets to buy out an event, etc. Once a CAP TCHA is
broken (i.e., computer programs can successfully pass the test), bots can impersonate
humans and gain access to services that they should not. Therefore, it is important for
CAP TCHAs to be secure.
To pass the typical visual CAP TCHA, a user must correctly type the characters displayed in
an image of distorted text. Many visual CAP TCHAs have been broken with machine1learning techniques [2]-[3], though some remain secure against such attacks. Because
visually impaired users who surf the Web using screen-reading programs cannot see this typeof CAPTCHA, audio CAP TCHAs were created. Typical audio CAP TCHAs consist of one
or several speakers saying letters or digits at randomly spaced intervals. A user mustcorrectly identify the digits or characters spoken in the audio file to pass the CAP TCHA. To
make this test difficult for current computer systems, specifically automatic speech
recognition (ASR) programs, background noise is injected into the audio files.
Since no official evaluation of existing audio CAP TCHAs has been reported, we tested the
security of audio CAP TCHAs used by many popular Web sites by running machine learning
experiments designed to break them. In the next section, we provide an overview of theliterature related to our project. Section 3 describes our methods for creating training data,and section 4 describes how we create classifiers that can recognize letters, digits, and noise.
In section 5, we discuss how we evaluated our methods on widely used audio CAP TCHAs
and we give our results. In particular, we show that the audio CAP TCHAs used by sites
such as Google and Digg are susceptible to machine learning attacks. Section 6 mentions the
proposed design of a new more secure audio CAP TCHA based on our findings.

2

Lit erat u r e r ev i ew

To break the audio CAP TCHAs, we derive features from the CAP TCHA audio and use
several machine learning techniques to perform ASR on segments of the CAPTCHA. There
are many popular techniques for extracting features from speech. The three techniques we use
are mel-frequency cepstral coefficients (MFCC), perceptual linear prediction (PLP), andrelative spectral transform-PLP (RAS TA-PLP). MFCC is one of the most popular speechfeature representations used. Similar to a fast Fourier transform (FF T), MFCC transforms an
audio file into frequency bands, but (unlike FF T) MFCC uses mel-frequency bands, which
are better for approximating the range of frequencies humans hear. PLP was designed to
extract speaker-independent features from speech [4]. Therefore, by using PLP and a variant
such as RAS TA-PL P, we were able to train our classifiers to recognize letters and digits
independently of who spoke them. Since many different people recorded the digits used in
one of the types of audio CAP TCHAs we tested, PLP and RAS TA-PLP were needed to
extract the features that were most useful for solving them.
In [4]-[5], the authors conducted experiments on recognizing isolated digits in the presenceof noise using both PLP and RAS TA-PL P. However, the noise used consisted of telephoneor microphone static caused by recording in different locations. The audio CAP TCHAs weuse contain this type of noise, as well as added vocal noise and/or music, which is supposed
to make the automated recognition process much harder.
The authors of [3] emphasize how many visual CAP TCHAs can be broken by successfully
splitting the task into two smaller tasks: segmentation and recognition. We follow a similar
approach in that we first automatically split the audio into segments, and then we classify
these segments as noise or words.
In early March 2008, concurrent to our work, the blog of Wintercore Labs [6] claimed tohave successfully broken the Google audio CAP TCHA. After reading their Web article andviewing the video of how they solve the CAP TCHAs, we are unconvinced that the process
is entirely automatic, and it is unclear what their exact pass rate is. Because we are unable tofind any formal technical analysis of this program, we can neither be sure of its accuracy nor
the extent of its automation.

3

Cr e a t i o n o f tra i n i n g dat a

Since automated programs can attempt to pass a CAPTCHA repeatedly, a CAPTCHA isessentially broken when a program can pass it more than a non-trivial fraction of the time;e.g., a 5% pass rate is enough.
Our approach to breaking the audio CAP TCHAs began by first splitting the audio files into
segments of noise or words: for our experiments, the words were spoken letters or digits. We
used manual transcriptions of the audio CAP TCHAs to get information regarding thelocation of each spoken word within the audio file. We were able to label our segments
accurately by using this information.

2

We gathered 1,000 audio CAP TCHAs from each of the following Web sites: google.com,
digg.com, and an older version of the audio CAP TCHA in recaptcha.net. Each of the
CAP TCHAs was annotated with the information regarding letter/digit locations provided by
the manual transcriptions. For each type of CAPTCHA, we randomly selected 900 samples
for training and used the remaining 100 for testing.
Using the digit/letter location information provided in the manual CAP TCHA
transcriptions, each training CAP TCHA is divided into segments of noise, the letters a-z, orthe digits 0-9, and labeled as such. We ignore the annotation information of the
CAP TCHAs we use for testing, and therefore we cannot identify the size of those segments.
Instead, each test CAP TCHA is divided into a number of fixed-size segments. The segmentswith the highest energy peaks are then classified using machine learning techniques (Figure1). Since the size of a feature vector extracted from a segment generally depends on the sizeof the segment, using fixed-size segments allows each segment to be described with a feature
vector of the same length. We chose the window size by listening to a few training
segments and adjusted accordingly to ensure that the segment contained the entire
digit/letter. There is undoubtedly a more optimal way of selecting the window size,
however, we were still able to break the three CAP TCHAs we tested with our method.

Figure 1: A test audio CAP TCHA with the fixed-size segments containing the highest
energy peaks highlighted.
The information provided in the manual transcriptions of the audio CAP TCHAs contains alist of the time intervals within which words are spoken. However, these intervals are ofvariable size and the word might be spoken anywhere within this interval. To provide fixedsize segments for training, we developed the following heuristic. First, divide each file into
variable-size segments using the time intervals provided and label each segment accordingly.
Then, within each segment, detect the highest energy peak and return its fixed-sizeneighborhood labeled with the current segment’s label. This heuristic achieved nearly perfectlabeling accuracy for the training set. Rare mistakes occurred when the highest energy peakof a digit or letter segment corresponded to noise rather than to a digit or letter.
To summarize this subsection, an audio file is transformed into a set of fixed-size segments
labeled as noise, a digit between 0 and 9, or a letter between a and z. These segments arethen used for training. Classifiers are trained for one type of CAPTCHA at a time.

4

C l a s s i f i e r co n s t ru ct i o n

From the training data we extracted five sets of features using twelve MFCCs and twelfth-

3

order spectral (SPEC) and cepstral (CEPS) coefficients from PLP and RAS TA-PL P. The
Matlab functions for extracting these features were provided online at [7] and as part of the
Voicebox package. We use AdaBoost, SVM, and k-NN algorithms to implement automateddigit and letter recognition. We detail our implementation of each algorithm in thefollowing subsections.
4 .1

AdaBoost

Using decision stumps as weak classifiers for AdaBoost, anywhere from 11 to 37 ensemble
classifiers are built. The number of classifiers built depends on which type of CAPTCHA weare solving. Each classifier trains on all the segments associated with that type of
CAP TCHA, and for the purpose of building a single classifier, segments are labeled byeither -1 (negative example) or +1 (positive example). Using cross-validation, we choose touse 50 iterations for our AdaBoost algorithm. A segment can then be classified as aparticular letter, digit, or noise according to the ensemble classifier that outputs the number
closest to 1.
4 .2

S u p p o rt v e ct o r m a c h i n e

To conduct digit recognition with SVM, we used the C++ implementations of libSVM [8]version 2.85 with C-SMV and RBF kernel. First, all feature values are scaled to the range of-1 to 1 as suggested by [8]. The scale parameters are stored so that test samples can bescaled accordingly. Then, a single multiclass classifier is created for each set of featuresusing all the segments for a particular type of CAPTCHA. We use cross-validation and gridsearch to discover the optimal slack penalty (C=32) and kernel parameter (γ=0.011).
4 .3

k - n e a re st n e i g h b o r ( k - N N )We use k-NN as our final method for classifying digits. For each type of CAP TCHA, fivedifferent classifiers are created by using all of the training data and the five sets of featuresassociated with that particular type of CAP TCHA. Again we use cross-validation to discover
the optimal parameter, in this case k=1. We use Euclidian distance as our distance metric.

5

Ass e s sm e n t of cu rre n t a ud i o CAPTCHAs

Our method for solving CAP TCHAs iteratively extracts an audio segment from a
CAP TCHA, inputs the segment to one of our digit or letter recognizers, and outputs thelabel for that segment. We continue this process until the maximum solution size is reachedor there are no unlabeled segments left. Some of the CAPTCHAs we evaluated have
solutions that vary in length. Our method ensures that we get solutions of varying lengththat are never longer than the maximum solution length. A segment to be classified is
identified by taking the neighborhood of the highest energy peak of an as yet unlabeled partof the CAP TCHA.
Once a prediction of the solution to the CAPTCHA is computed, it is compared to the true
solution. Given that at least one of the audio CAP TCHAs allows users to make a mistake inone of the digits (e.g., reCAPTCHA), we compute the pass rate for each of the different typesof CAPTCHAs with all of the following conditions:•

The prediction matches the true solution exactly.

•

Inserting one digit into the prediction would make it match the solution exactly.

•

Replacing one digit in the prediction would make it match the solution exactly.

•

Removing one digit from the prediction would make it match the solution exactly.

However, since we are only sure that these conditions apply to reCAPTCHA audio
CAP TCHAs, we also calculate the percentage of exact solution matches in our results foreach type of audio CAP TCHA. These results are described in the following subsections.
5 .1

Goog leGoogle audio CAP TCHAs consist of one speaker saying random digits 0-9, the phrase“once again,” followed by the exact same recorded sequence of digits originally presented.

4

The background noise consists of human voices speaking backwards at varying volumes. A
solution can range in length from five to eight words. We set our classifier to find the 12
loudest segments and classify these segments as digits or noise. Because the phrase “onceagain” marks the halfway point of the CAPTCHA, we preprocessed the audio to only serve
this half of the CAP TCHA to our classifiers. It is important to note, however, that theclassifiers were always able to identify the segment containing “once again,” and thesesegments were identified before all other segments. Therefore, if necessary, we could have
had our system cut the file in half after first labeling this segment.
For AdaBoost, we create 12 classifiers: one classifier for each digit, one for noise, and one
for the phrase “once again.” Our results ( Table 1) show that at best we achieved a 90% passrate using the “one mistake” passing conditions and a 66% exact solution match rate. Using
SVM and the “one mistake” passing conditions, at best we achieve a 92% pass rate and a
67% exact solution match. For k-NN, the “one mistake” pass rate is 62% and the exact
solution match rate is 26%.
Table 1: Google audio CAP TCHA results: Maximum 67% accuracy was achieved by SVM.
Classifiers Used

Features Used

AdaBoost

5 .2

SVM

k-NN

One
mistake

exact
match

one
mistake

exact
match

one
mistake

exact
match

MFCC

88%

61%

92%

67%

30%

1%

PLPSPEC

90%

66%

90%

67%

60%

26%

PLPCEPS

90%

66%

92%

67%

62%

23%

RAS TAPLPSPEC

88%

48%

90%

61%

29%

1%

RAS TAPLPCEPS

90%

63%

92%

67%

33%

2%

Digg

Digg CAP TCHAs also consist of one speaker, in this case saying a random combination ofletters and digits. The background noise consists of static or what sounds like tricklingwater and is not continuous throughout the entire file. We noticed in our training data thatthe following characters were never present in a solution: 0, 1, 2, 5, 7, 9, i, o, z. Since theDigg audio CAPTCHA is also the verbal transcription of the visual CAP TCHA, we believethat these characters are excluded to avoid confusion between digits and letters that are
similar in appearance. The solution length varies between three and six words. Using
AdaBoost, we create 28 classifiers: one classifier for each digit or letter that appears in our
training data and one classifier for noise. Perhaps because we had fewer segments to train
with and there was a far higher proportion of noise segments, AdaBoost failed to produce any
correct solutions. We believe that the overwhelming number of negative training examples
versus the small number of positive training samples used to create each decision stumpseverely affected AdaBoost’s ability to classify audio segments correctly.
A histogram of the training samples is provided in Figure 2 to illustrate the amount oftraining data available for each character. When using SVM, the best feature set passed with
96% using “one mistake” passing conditions and passed with 71% when matching thesolution exactly. For k-NN, the best feature set produced a 90% “one mistake” pass rate anda 49% exact solution match. Full results can be found in Table 2.

5

Table 2: Digg audio CAP TCHA results: Maximum 71% accuracy was achieved by SVM.
Classifiers Used

Features Used

AdaBoost

SVM

k-NN

one
mistake

exact
match

one
mistake

exact
match

one
mistake

exact
match

MFCC

-

-

96%

71%

89%

49%

PLPSPEC

-

-

94%

65%

90%

47%

PLPCEPS

-

-

96%

71%

64%

17%

RAS TAPLPSPEC

-

-

17%

3%

67%

17%

RAS TAPLPCEPS

-

-

96%

71%

82%

34%

1000
900
800

# of Segments

700
600
500
400
300
200
100

y

x

v

w

t

u

r

s

p

q

n

m

l

j

k

h

f

g

e

c

d

a

b

8

6

3

4

noise

0

Segment Label

Figure 2: Digg CAP TCHA training data distribution.
5 .3

reC A P T C H A

The older version of reCAPTCHA’s audio CAP TCHAs we tested consist of several speakers
who speak random digits. The background noise consists of human voices speaking
backwards at varying volumes. The solution is always eight digits long. For AdaBoost, wecreate 11 classifiers: one classifier for each digit and one classifier for noise. Because we6know that the reCAPTCHA passing conditions are the “one mistake” passing conditions,
SVM produces our best pass rate of 58%. Our best exact match rate is 45% ( Table 3).
Table 3: reCAPTCHA audio CAP TCHA results: Maximum 45% accuracy was achieved by
SVM.
Classifiers Used

Features Used

AdaBoost

6

SVM

k-NN

one
mistake

exact
match

one
mistake

exact
match

one
mistake

exact
match

MFCC

18%

6%

56%

43%

22%

11%

PLPSPEC

27%

10%

58%

39%

43%

25%

PLPCEPS

23%

10%

56%

45%

29%

14%

RAS TAPLPSPEC

9%

3%

36%

18%

24%

4%

RAS TAPLPCEPS

9%

3%

46%

30%

32%

12%

Prop ert i e s of w ea k ver s u s st ro n g CAPT CHAs

From our results, we note that the easiest CAP TCHAs to break were from Digg. Google
had the next strongest CAP TCHAs followed by the strongest from reCAPTCHA. Although
the Digg CAP TCHAs have the largest vocabulary, giving us less training data per label, thesame woman recorded them all. More importantly, the same type of noise is usedthroughout the entire CAPTCHA. The noise sounds like running water and static whichsounds very different from the human voice and does not produce the same energy spikes
needed to locate segments, therefore making segmentation quite easy. The CAP TCHAs from
Google and reCAPTCHA used other human voices for background noise, making
segmentation much more difficult. Although Google used a smaller vocabulary than Digg
and also only used one speaker, Google’s background noise made the CAP TCHA moredifficult to solve. After listening to a few of Google’s CAP TCHAs, we noticed that
although the background noise consisted of human voices, the same background noise wasrepeated. reCAP TCHA had similar noise to Google, but they had a larger selection of noise
thus making it harder to learn. reCAP TCHA also has the longest solution length making it
more difficult to get perfectly correct. Finally, reCAPTCHA used many different speakers
causing it to be the strongest CAP TCHA of the three we tested. In conclusion, an audio
CAP TCHA that consists of a finite vocabulary and background noise should have multiple
speakers and noise similar to the speakers.

7
Recomm e n d at i o n s f o r creat i n g st ro n g e r aud i o
CAPTCHAs
Due to our success in solving audio CAP TCHAs, we have decided to start developing new
audio CAP TCHAs that our methods, and machine learning methods in general, will be lesslikely to solve. From our experiments, we note that CAP TCHAs containing longer
solutions and multiple speakers tend to be more difficult to solve. Also, because ourmethods depend on the amount of training data we have, having a large vocabulary would
make it more difficult to collect enough training data. Already since obtaining these results,
reCAPTCHA.net has updated their audio CAP TCHA to contain more distortions and a7larger vocabulary: the digits 0 through 99. In designing a new audio CAP TCHA we are also
concerned with the human pass rate. The current human pass rate for the reCAPTCHA audio
CAP TCHAs is only 70%. To develop an audio CAP TCHA with an improved human passrate, we plan to take advantage of the human mind’s ability to understand distorted audio
through context clues. By listening to a phrase instead of to random isolated words, humans
are better able to decipher distorted utterances because they are familiar with the phrase or
can use contextual clues to decipher the distorted audio. Using this idea, the audio for our
new audio CAP TCHA will be taken from old-time radio programs in which the poor qualityof the audio makes transcription by ASR systems difficult. Users will be presented with anaudio clip consisting of a 4-6 word phrase. Half of the CAPTCHA consists of words, which
validate a user to be human, while the other half of the words need to be transcribed. This is
the same idea behind the visual reCAP TCHA that is currently digitizing text on which OCR
fails. We expect that this new audio CAP TCHA will be more secure than the current version
and easier for humans to pass. Initial experiments using this idea show this to be true [9].

8

Co n c l u s i o n

We have succeeded in “breaking” three different types of widely used audio CAP TCHAs,
even though these were developed with the purpose of defeating attacks by machine learningtechniques. We believe our results can be improved by selecting optimal segment sizes, butthat is unnecessary given our already high success rate. For our experiments, segment sizes
were not chosen in a special way; occasionally yielding results in which a segment only
contained half of a word, causing our prediction to contain that particular word twice. We
also believe that the AdaBoost results can be improved, particularly for the Digg audio
CAP TCHAs, by ensuring that the number of negative training samples is closer to thenumber of positive training samples. We have shown that our approach is successful and canbe used with many different audio CAP TCHAs that contain small finite vocabularies.
A ck n o w l e d g m e n t sThis work was partially supported by generous gifts from the Heinz Endowment, by anequipment grant from Intel Corporation, and by the Army Research Office through grant
number DAAD19-02-1-0389 to CyLab at Carnegie Mellon University. Luis von Ahn waspartially supported by a Microsoft Research New Faculty Fellowship and a MacArthur
Fellowship. Jennifer Tam was partially supported by a Google Anita Borg Scholarship.
R e f e re n c e s[1] L. von Ahn, M. Blum, and J. Langford. “Telling Humans and Computers Apart Automaticall y,”
Communication s of the ACM, vol. 47, no. 2, pp. 57-60, Feb. 2004.
[2] G. Mori and J. Malik. “Recognizing Objects in Adversarial Clutter: Breaking a Visual
CAPTCHA,” In Computer Vision and Pattern Recognition CVPR'03, June 2003.
[3] K. Chellapilla, and P. Simard, “ U sing Machine Learning to Break Visual Human Interactio n
Proofs (HIP s),” Advances in Neural Information P rocessing Systems 17, Neural Info rmat io n
P rocessing Systems (NIPS'2004), MIT Press.
[4] H. Hermansk y, “ Perceptual Linear Predictive (PL P) Analysi s of Speech,” J. Acoust. Soc. Am.,
vol. 87, no. 4, pp. 1738-1752, Apr. 1990.
[5] H. Hermansk y, N. Morgan, A. Bayya, and P. Kohn. “RASTA-PL P Speech Analysi sTechnique,” In P roc. IEEE Int’l Conf. Acoustics, Speech & Signal Processing, vol. 1, pp. 121124, San Francisco, 1992.
[6] R. Santamarta. “Breaking Gmail ’s Audio Captcha,” http: //blog.wintercore.com/ ?p=11, 2008.
[7] D. Ell is. “ P L P and RASTA (and MFCC, and inversion) in Matlab using melfcc.m andinvmelfcc.m,” http:/ /ww w.ee.columbia.edu/~dpwe/resources/matlab/rastamat/, 2006.
[8] C. Chang and C. Lin. LIBSVM: a library for support vector machines, 2001. Software available
at http: //ww w.csie.ntu.edu.tw/~cjlin/l ibsvm[9] A. Schlaikjer. “ A Dual-Use Speech CA PTCHA: Aiding Visually Impaired Web Users while
Providing Transcriptions of Audio Streams,” Technical Report CMU-LTI-07-014, Carnegie
Mellon Universi t y. November 2007.

8