The busuu Efficacy Study
FINAL REPORTRESEARCH TEAMROUMEN VESSELINOV1,2, PhD
Economics Department
Queens College,
City University of New Yorkroumen.vesselinov@qc.cuny.edu

JOHN GREGO, PhD
Statistics Department
University of South Carolina
grego@stat.sc.edu

May 2016

1
2

Corresponding author.
This report represents the individual opinion of the authors and not necessarily of the two institutions.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 1EXECUTIVE SUMMARY
This study was independently conducted by the Research Team from February 2016 to April
2016. A random representative sample of 196 busuu users was drawn. The participants took
one college placement Spanish language test and one oral proficiency test, then studied
Spanish with busuu for two months and took the same tests again. Both tests were proctored.
The improvement in language abilities was measured as the difference between the final andthe initial language test results. The efficacy of busuu was measured as written proficiency
improvement per one hour of study and the proportion of users who improved their oral
proficiency.
MAIN RESULTS
Written Proficiency Gain:• Overall 84% of the participants improved their written proficiency.
• busuu users need on average 22.5 hours of study in a two-month period
to cover the requirements for one college semester of Spanish.
Oral Proficiency Gain:• Over 75% of busuu users increased their oral proficiency by at least one level.
SUPPLEMENTARY RESULTS
Written Proficiency:• The efficacy of busuu is a gain of about 12 test points per one hour of study.
For beginners the gain is 13.6 points and for advanced users it is 3 points.
• About 42% of the participants moved up at least one college semester level.
Of those, 26% moved up one semester, 10% - two semesters, and 6% - three semesters.
Oral Proficiency:• Over 75% of the participants increased their oral proficiency by at least one level.
Of those, 44% moved up one level, 25% - two levels, and 7% - more than two levels.
User Satisfaction:• The majority of users thought that busuu was easy to use (86%),
helpful (84%), enjoyable (78%), and they were satisfied with it (74%).
• busuu received a positive Net Promoter Score of 8.4 from users.
• busuu efficacy was not affected by gender, race, age, education, native language,
device used, etc.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 2

CONTENTS

1.

Introduction ....................................................................................................................... 3

2.

Research Design ................................................................................................................ 5

3.

Sample Description ........................................................................................................... 8

4.

Language Improvement and Study Time ..................................................................... 15

5.

Main Results .................................................................................................................... 18

6.

User Satisfaction .............................................................................................................. 25

7.

Limitations of the Study ................................................................................................. 26

8.

Conclusion........................................................................................................................ 27

9.

Cited Literature............................................................................................................... 29

10.

Appendix .......................................................................................................................... 30

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 3

1. Introduction
There are many language learning apps available today. There are different claims from all ofthem about how effective or how fast one can learn a foreign language. It is very difficult toselect the right app just based on their claims. Our Research Team started evaluating language
learning software back in 2008. Since then we have evaluated several learning software
providers and this study adds one more evaluation and tests one more claim from a language
learning app.

There is a growing interest in evaluating the efficacy (or effectiveness) of language learningapps. New users, investors, analysts and academics are eager to learn what they can expect togain by using a particular software and which software is most effective. Our research teamhas already conducted several studies attempting to directly evaluate the efficacy, attitude
and motivation of some popular language learning software packages, namely Rosetta Stone®,
Aurolog® and Berlitz®, Duolingo® and a new language app (Vesselinov 2008, Vesselinov et al.
2009a, 2009b, Vesselinov & Grego, 2012 & 2016). Since the 2012 study we regularly receive
inquiries from the US and all over the world: e.g. a school district administrator in New Yorkand in China, a foundation in India related to school excellence, major investment groups,
individual users, etc. All of them want the same thing: they need help to decide which
language app they should use. Other things being equal (e.g. price, appearance, ease to use,etc.) they needed independent evaluation of the efficacy of the apps and the more specific themeasure is, the better.

With this study we are trying to evaluate the efficacy of a well-known language software
product: busuu3. The company was founded in 2008 by two European entrepreneurs whobelieved that existing online language learning programs were missing an important social
element. busuu was designed to combine self-paced language study with a supportive social
network of language learners around the world. Students learn vocabulary and grammar in3

www.busuu.com

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 4thematic lessons, and then put the language they have learned into practice through writing
and speaking exercises which are marked by native speaker members of the community.

As of early 2016, the busuu community numbers over 60 million (per www.busuu.com)
language learners around the world; there are courses in 12 languages, delivered throughweb, iOS and Android apps. Up to 100,000 new learners join the site each day according to thecompany.

Some of busuu’s key features include:• Interactive vocabulary and grammar lessons with audio, translation and multiple practice
exercises;• Audio recordings of each vocabulary item, plus example sentences and dialogues to place
vocabulary and grammar in context;• Voice recording exercises to drill pronunciation and allow students to insert their voice into a
dialogue and get feedback from native speakers;• Translations of key vocabulary, instructions and grammar tips into thirteen languages;• Writing exercises which receive instant corrections from native speakers in the busuu
community;• busuu-talk (web only) which allows students to find language partners and practice speaking
or text chatting with them;This study was funded by busuu but the data collection and the analysis were carried out
independently by the Research Team. The two language tests used in the study were designed
and developed by two external independent testing companies.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 5

2. Research Design
The random sample for this study was selected from existing or new busuu users who lived inor around London, UK or New York, US. The geographical restrictions were imposed because
the test for oral proficiency must be proctored. There were some additional requirements forthe potential participants. They had to be:-

Willing to study Spanish using only busuu for two months, and come to the testing location
for two sets of language tests;-

At least 18 years of age;-

Not of Hispanic origin;-

Not advanced learners of Spanish.

The last requirement was due to the fact that the written language placement test used in the
study has placement in college Semester 4+ as its highest evaluation group and it has limited
abilities for very advanced users. The oral proficiency test has no limitations because theevaluation is done by two independent raters.

The recommended goal for the participants in the study was to use busuu for at least 16 hours
during the two-month study, or two hours per week. Based on our experience with previous
studies we imposed a threshold of at least two hours of study for the written test. People withless than two hours of study were not allowed to complete the study because there was not a
sufficient effort for measurable progress. For the oral proficiency test the requirement was atleast 16 hours of study.

The Spanish language was selected as one of the more popular languages and also because of
the existence of previous research on Spanish for other language learning apps. The length of
the study was approximately 8 weeks and it was conducted between the months of February
2016 and April 2016. People who successfully completed the study were given a lifetime freesubscription to the premium edition of busuu for themselves and one friend of theirs. The
participants in the oral proficiency tests received an official certificate for their level. No
monetary or other incentives were offered to the participants.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 6The main instrument for evaluating the level of knowledge of Spanish was the Web Based
Computer Adaptive Placement Exam4 (WebCAPE test). It is an established university
placement test and it is offered in ESL, Spanish, French, German, Russian and Chinese. It was
created by Brigham Young University and is maintained by the Perpetual Technology Group. A
more detailed description of the test can be found at their website5.

The Spanish WebCAPE test has a very high validity correlation coefficient (0.91) and very highreliability (test-retest) value of 0.81. The test is adaptive so the time for taking the test varies
with an average time of 20-25 minutes. The WebCAPE test gives a score (in points) and basedon that score places the students in different level groups (college semesters).
Table 1. Spanish WebCAPE Test Cut-off Points
WebCAPE Test Points
Below 270
270-345346-428
Above 428

College Semester Placement
Semester 1
Semester 2
Semester 3
Semester 4+

The WebCAPE results alone cannot give a clear picture about the efficacy of the language
learning app because they do not account for the time spent studying. That is why we arerelying on a direct and objective measure of efficacy which is defined as follows:

Efficacy Effect Improvement of language skills Final-Initial WebCAPE test score=
EffortStudy time
Hours of study

Efficacy=Improvement per one hour of study
This measure includes both the amount of progress made by each study participant and theamount of their effort. It is a fair measure of efficacy and also a direct and objective measure

4

Spanish WebCAPE Computer-Adaptive Placement Exam by Jerry Larson and Kim Smith, online version Charles
Bush. ©1998, 2004 Humanities Technology and Research Support Center, Brigham Young University.
5
http://www.perpetualworks.com/webcape/overview

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 7of efficacy. Direct, because it includes directly the effect and the effort. Objective, because theeffect is measured by an independent college placement test (instead of our own test) and theeffort is measured by the time recorded on computer servers (instead of self-report).

The second test used in the study was the Oral Proficiency Interview by Computer® (OPIc)6
created by Language Testing International (LTI). LTI is the exclusive licensee of the American
Council on the Teaching of Foreign languages (ACTFL). The online test is proctored and therecording of the test is reviewed and evaluated by two independent raters and an official
certificate of oral proficiency is issued by ACTFL.
Table 2. OPIc Ratings
UR
NL
NM
NH
IL
IM
IH

Un-Ratable
Novice Low
Novice Mid
Novice HighIntermediate Low
Intermediate Mid
Intermediate High

AL
AM
AH
S

Advanced Low
Advanced Mid
Advanced HighSuperior

The specific definition of the levels are presented on the company’s webpage7.

67

http://www.languagetesting.com/oral-proficiency-interview-by-computer-opic
http://d2k4mc04236t2s.cloudfront.net/wp-content/uploads/2013/07/ACTFL-Proficiency-Guidelines-2012.pdf

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 8

3. Sample Description
The entire sample selection process is graphically represented in the Appendix, Figure A1.
E-mail messages were sent out to busuu clients with an invitation to participate in the
research study. If they accepted the invitation they were asked to complete the online Entry
Survey with some demographic questions and questions about their knowledge of Spanish. In
all 2,716 people viewed the invitation page and of those 743 successfully completed the Entry
Survey. This was the initial pool of respondents in the study.

Initial Pool (N=743)
The initial pool of potential participants consisted of people from London (N=394) and New
York (N=349) and about half of them (48%) were female. Interestingly, 40% of the respondents
in London were female, versus almost 60% in New York. The mean age was 36 years and they
were well educated: 28% had a graduate degree and 59% had a BA or some college education.
Only 13% had just High School or less. The racial composition was 18% Black/African American,
13% Asian, 59% White and 10% Other Race, including multiracial categories. Most of thepeople were employed either full time or part-time (77%), 10% were unemployed and 13%
were students.

For 78% of the initial pool English was their native language and the remainder (22%) included
about 50 languages: Akan, Arabic, Azerbaijani, Cantonese, Cebuano, Chinese, Czech, Dutch,
Dutch/Flemish, Filipino, French, Georgian, German, Greek, Gujarati, Haitian Creole, Hebrew,
Hindi, Italian, Japanese, Korean, Lithuanian, Malay, Malayalam, Mandarin, Mauritian Creole,
Nepali, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Sesotho, Shona, Sinhala,
Slovak, Slovenian, Swedish, Tagalog, Tamil, Telugu, Turkish, Urdu, Uzbek, Yiddish.
Almost 97% described themselves as Novice users or Never Studied Spanish. A small
proportion of them (5.2%) were of Hispanic origin and about 17% of the respondents’ spouse,
partner, or close friends spoke Spanish. A very small proportion (6.3%) of their parents,
grandparents, or great-grandparents spoke Spanish.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 9The primary reason for studying Spanish was personal interest (61%), followed by business orwork (17%), travel (15%), school (2%), and other reasons (5%). For other reasons the
respondents mentioned: “all of the above”, “mixture”, “girlfriend/boyfriend/friend speaks
Spanish”, “want more education”, “planning to move to Spain”, “to talk with Spanish family
members”, “to read Don Quixote”, etc.

About 80% of the initial pool had studied a foreign language before (mostly at school orcollege) and about half (45%) knew a different foreign language.
About half (55%) of the participants planned to use a desktop or laptop/tablet for studying
Spanish with the rest (45%) using their smartphones.

Pool of Eligible Participants (N=634)
From the Initial Pool (N=743) we excluded the following ineligible participants:


People who were younger than 18 years of age.

People of Hispanic origin.

People with advanced or fluent Spanish.

People who did not live in London or New York area.

Altogether 109 people were ineligible for this study and the final pool of eligible participants
for sample selection was N=634.

The pool of eligible potential participants had a mean age of 36 years, from 18 years old to 79
years old, with 48% female users. Racial composition: 16% Black/African American, 13% Asian,
60% White, and 10% Other Race. The pool of eligible users was very well educated with only
about 12% with just a High School diploma or less. About 78% were employed full time or parttime, 13% were students, and 10% were unemployed. For 78% of them English was theirnative language and 46% of the pool knew at least one foreign language.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 10

Initial Random Sample (N=196)
The research design determined a sample size of N=200 based on our previous studies’ effectsize results, drop-out rate and financial considerations. The people in the initial sample wererandomly selected from the pool of eligible participants. They completed the baseline
WebCAPE placement test in Spanish and the oral proficiency OPIc test (96 people in London
and 100 people in New York, 4 people did not show up for the tests). Both tests were
proctored.
The initial random sample had a mean age of 36 years, from 19 years old to 69 years old, with
52% female users (40% female users in London and 63% in New York). The racial composition
was 20% Black/African American, 15% Asian and 55% White, and 10% Other Race. The users
were very well educated with 32% with a graduate degree and 60% with some college orcollege degree. About 78% of them were employed either full time or part time, 13% werestudents, and 10% were unemployed. For 89% of them English was their native language (99%in London and 80% in New York) and almost 41% of the sample knew at least one foreign
language.

People planning to use desktop/laptop/tablet made up over half the sample (58%) with theother group planning to use their smartphones. Personal interest was the primary reason(65%) for studying Spanish, followed by travel (15%), business/work (13%), school (3%) andother reason (5%). About 14% of participants’ spouse/close friend and only 1% ofparents/grandparents knew Spanish.
Table 3. Initial Random Sample: Age and Gender Distribution (N=196)
Age
18-20 years old
21-30 years old
31-40 years old
Over 40 years old
Total

Female (N)
3
45
20
33
101

Male (N)
7
32
28
28
95

Total (N)
10
77
48
61
196

Percent
5.1
39.3
24.5
31.1
100.0

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 11After the selection the study participants were asked to come to New York and London offices
where the initial tests were proctored. All 196 participants took the initial written online
WebCAPE test and the OPIc oral proficiency test in Spanish.

The written proficiency of the initial study sample was as follows:
Table 4. Initial WebCAPE Semester Placement (N=196)
College Semester
First
SecondThird
Fourth+Total

People (N)
170
20
6
0
196

Percent
86.7
10.2
3.1
0
100.0

The majority (87%) of the participants were evaluated as novice/beginner users of Spanish andthey were placed in First Semester of Spanish. About 13% of the participants were placed in
Second or Third Semester of Spanish. The mean WebCAPE score was 120 (std8=118)
corresponding to First college semester of Spanish.
The oral proficiency of the initial sample was as follows:
Table 5. Initial Oral Proficiency (OPIc) (N=196)
Proficiency level
1 Un-Ratable
2 Novice Low
3 Novice Mid
4 Novice High
5 Intermediate Low
6 Intermediate Mid
Total

People (N)
16
133
35
6
4
2
196

Percent
8.2
67.9
17.9
3.1
2.0
1.0
100.0

The majority of people (76%) had the two lowest levels of oral proficiency and about 18%
were at Novice Mid level. A handful of people (N=12) had up to Intermediate Mid level oral
proficiency.

8Standard Deviation.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 12

Final Study Sample (N=144)
The study continued for approximately 8 weeks, starting in February 2016 and ending in April
2016. During the study the Research Team sent weekly e-mail reminders to the participants
with information detailing the amount of time they had used busuu each week.

At the end of the study we reviewed the time use of the participants. For the written
proficiency test (WebCAPE) based on our previous studies (Vesselinov & Grego, 2012, 2016)
the threshold was established at two hours of study. For the oral proficiency test based on therequirements by ACTFL for test-retest (2-3 months) the threshold was established at 16 hoursof study. In other words, participants with at least two hours of study could take the written
test and complete the study. If they had less than 16 hours of study they were not eligible totake the oral test. People with 16 or more hours of study could take both written and oral
tests.

Based on these criteria from the initial sample the following people were excluded:

People who did not satisfy the study time requirements.



People who did not take the final tests.

People who did not do oral practice (for the oral test only).

People who used additional learning tools during the study.

All participants were instructed at the beginning of the study that they were allowed to use
only busuu to study Spanish for the duration of the study. In the exit survey several people
stated that they had regularly used other tools like other language apps, language classes, etc.
and these people were excluded from the study. Other people had occasionally used internet
dictionaries and similar websites and they were allowed to stay in the study.

A small portion (N=12) of the initial sample declared at the end of the study that they could
not do oral practice mostly for technical reasons (e.g. microphone was not working) and thesepeople were excluded from the final oral tests.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 13

About 8% of the participants had less than two hours of study and they were excluded fromthe study. About 43% had from two hours to 15.9 hours and they were eligible to take thewritten proficiency test but not the oral test. Finally, about 49% had 16 hours of study or more
and they were eligible to take both written and oral proficiency tests. The mean study time forthe final study sample was about 18.6 hours.

The final study sample for written proficiency consisted of 144 people with at least two hours
or more of busuu use and valid initial and final WebCAPE tests. The final subsample for oral
proficiency was part of the 144 people sample and consisted of 61 people with about 16 hoursof study or more and valid initial and final OPIc tests. The mean study time for the oral test
sample was about 24 hours.

The final study sample (N=144) had a mean age of 36.6 years, from 19 years old to 69 years
old, with 49.3% female users. Racial composition: 18% Black/African American, 13% Asian,
58% White and 10% of other race. The users were very well educated with 33% holding a
graduate degree and 56% with BA or some college. About 76% of them were employed fulltime or part time, 14% were students, and 10% unemployed.

For 92% of them English was their native language and the rest included: Akan, Cebuano,
Chinese, French, Italian, Malayalam, Polish, and Russian. About 40% of the sample knew atleast one other foreign language (not Spanish) and they were novice users of Spanish or had
never studied Spanish.

About 13% of the respondents’ spouse, partner, or close friends spoke Spanish. None of theirparents, grandparents, or great-grandparents spoke Spanish.
The primary reason for studying Spanish was personal interest (67%), followed by travel (17%),business or work (9%), school (3%), and other reasons (5%).

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 14

Table 6. Final Study Sample: Age and Gender Distribution (N=144)
Age
18 to 20 years old
21-30 years old
31-40 years old
Over 40 years old
Total

Female (N)
3
32
12
24
71

Male (N)
6
21
23
23
73

Total (N)
9
53
35
47
144

Total (%)
6.3
36.8
24.3
32.6
100.0

People from the final sample planned to use different devices to study Spanish with busuu.
The majority of them (60%) planned to use a desktop/laptop/tablet computer with the rest
(40%) planning to use smartphones. Some people used more than one device and/or different
operating systems to access busuu; in the exit survey about 47% said that they had used adesktop/laptop computer, 27% had used a tablet, 37% had used an Android smartphone, and
33% had used an Apple iOS smartphone.

Final Study Sample vs Not Completed
From the initial random sample (N=196) 52 people did not complete the study for different
reasons: people who did not satisfy the study time requirements; people who did not take thefinal tests; people who did not do oral practice (for the oral test only) and people who usedadditional learning tools during the study.

We compared the two groups, the final sample of 144 people and the 52 people who did notcomplete the study by gender, race, age, education, employment status, initial knowledge of
Spanish (initial WebCAPE score and OPIc) and reason for studying Spanish. There were no
statistically significant differences (at 1% error) which means that people who did notcomplete the study were not very different from the ones that did and they did not introduce
a bias.

We also compared the sample composition for the two study sites: London and New York.
There were no statistically significant differences (at 1% error) between the two sites on race,

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 15age, education, employment status, initial knowledge of Spanish (initial WebCAPE score andOPIc) and reason for studying Spanish. The London sample had significantly fewer female
participants (40%) compared to 60% in New York but from our previous studies (Vesselinov &Grego, 2012, 2016) we know that gender is not a statistically significant factor in studying
languages with language apps. That is why we have pooled the data from the two sites andreported the results from the combined data.

4. Language Improvement and Study Time
Study Time
The study time was measured objectively by the actual server time on a weekly basis and thetime was reported to the participants regularly via e-mail in order to encourage them to keep
studying. The average study time for the final study sample (N=144) was about 18.6 hours, ora little over two hours a week.
Figure 1. Study Time Distribution (N=144)

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 16

WebCAPE Test Results
All participants took a proctored initial WebCAPE test before the start of the study and thenagain at the end of the study. The progress or improvement was measured as the difference
between the final test score and the initial one.

Table 7. Language Improvement Written Proficiency (N=144)
WebCAPE Test Points
Statistics

Initial WebCAPE

Final WebCAPE

Mean (std)Median
95% Confidence Interval9

124.8 (122.4)
93.0
104.6 – 145.0

269.6 (151.4)
265.5
244.6 – 294.5

Improvement(Final-Initial)
144.8 (160.3)
125.5
118.4 – 171.2

The overall average improvement of 144.8 WebCAPE test points was statistically significant
with a 95% Confidence Interval from 118 to 171 points. This also means that the improvementin the written proficiency for the final sample was statistically significant (at 5% error). Overall
84% of all participants improved their written proficiency (increased their WebCAPE score)with 95% Confidence Interval10 of 77% to 89%.

There were 23 cases (16%) where study participants did not improve their WebCAPE result or
had a lower result at the end of the study compared to their initial level. There are twoplausible explanations for this fact. First, some of them were more advanced learners of
Spanish (second semester) and gaining points at this higher level is generally more difficult andrequires more time. Second, some of them studied irregularly with more effort and study timein the beginning of the study and less towards the end of the study. These users were notexcluded from the sample so the results can be generalized for all types of users and not only
for diligent, hardworking users who study regularly.

9

We also bootstrapped (N=10,000) the confidence intervals but the results remained practically the same.
95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

10

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 17

College Semester Placement
Progress can be measured by movement from one semester level to a higher semester level
and the results are presented below.
Table 8. WebCAPE Semester Placement (N=144)
College Semester
First
SecondThird
Fourth+Total

Initial Test%
People (N)
122
84.7
17
11.8
5
3.5
144

100

Final Test
People (N)
%
76
34
19
15
144

52.8
23.6
13.2
10.4
100

People at First Semester level decreased from 85% to 53% and the proportion of people in
Second to Fourth+ Semester level increased notably.

Oral Proficiency
The oral proficiency results are presented below.
Table 9. Oral Proficiency Placement (N=61)
Level
1 Un-Ratable
2 Novice Low
3 Novice Mid
4 Novice High
5 Intermediate Low
6 Intermediate Mid
7 Intermediate High
8 Advanced Low
Total

Initial Test
People (N)
6
30
19
2
3
1

%
9.8
49.2
31.1
3.3
4.9
1.6

61

100

Final Test
People (N)
2
9
17
21
7
3
1
1
61

%
3.3
14.8
27.9
34.4
11.5
4.9
1.6
1.6
100

In the oral proficiency area the results are even stronger. At the beginning of the study thetruly novice users (Un-Ratable and Novice Low) were almost 60% of the sample while at theend their numbers decreased to about 18%. This is a completely different level of oral
proficiency.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 18

5. Main Results
Written Proficiency
Table 10. Written Proficiency Improvement (N=144)
Level (Semester Change)-1 Negative change
0 Same/No Change
1 One Semester Up
2 Two Semesters Up
3 Three Semesters Up
Total

Improved
People (N)
%
2
1.4
82
56.9
37
25.7
14
9.7
9
6.3
144
100

Study Time
Mean (Hours)
17.4
16.8
21.4
21.2
19.6
18.6

Overall about 42% of the participants moved up at least one semester. About 26% moved up
one semester, 10% moved up two semesters and 6% moved up three semesters. About 57%
stayed in the same semester they started in and two people moved down a semester. As theresults indicate, the people who had invested the lowest amount of effort and study time
were unsurprisingly the ones who did not improve their written proficiency in semester level.

The problem with this measure is that first, it does not account for the effort (study time) andsecond, moving up a semester is dependent on the exact initial level. For example, if a person
has initially 269 test points (First semester), only 1 point progress is needed to move to Second
semester. Another person can start with 10 points (First semester), then gain 200 points andthe new level (210 points) is still First Semester.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 19

The main efficacy measures are presented below.
Table 11. Main Result. Efficacy of busuu (N=144)

Statistics
Mean
95% Confidence Interval

Efficacy
Improvement per one
hour of study
WebCAPE Test Points
12.0
8.0 – 16.012

Time to cover the requirementsfor one semester of college
Spanish
Hours
22.51117.0 – 34.013

On average busuu users will gain 12 WebCAPE test points per one hour of study with 95%
Confidence Interval of 8 to 16 test points per hour.

The main measure of busuu efficacy is the improvement per one hour of study. In addition if
we divide the required cut-off point (270) for WebCAPE Second Semester placement by theefficacy mean we can construct a new measure representing the time needed to cover therequirements for one college semester of Spanish. This is the one measure of efficacy that iseasy to understand and given the nature of the WebCAPE placement test, can be used forcomparison with other language apps.

In other words, on average, busuu users will need 22.5 hours of study to cover therequirements for one college semester of Spanish with transformed lower and upper limits of
17 hours to 34 hours of study.

11

The threshold of 270 points divided by the mean efficacy (12 points).
We also bootstrapped (N=10,000) the confidence interval but the result remained practically the same.
13
The threshold of 270 points divided by the lower limit (8) and the upper limit (16) of the 95% CI.
12

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 20

Efficacy and the Initial Level of Knowledge of Spanish
Table 12. Efficacy by Initial Level of Language Ability (N=144)
Initial Level
College Semester
First
SecondThird
Total

People
N
122
17
5
144

Efficacy
Mean (std)13.6 (25.7)
2.9 (6.1)
2.6 (3.3)
12.0 (24.0)

The overall efficacy is 12 WebCAPE points per one hour of study but novice users of Spanish
managed a bigger gain of 13.6 points per hour of study. For the second and third semesterlevels the improvement was more modest at about 3 points per hour.

Factors for Written Proficiency
We investigated the impact of some factors on the efficacy measure, namely age, gender,
education, race, employment, device used, reason for studying Spanish, presence of people
around the participant who spoke Spanish (spouse, friend, parents, grandparents, etc.), native
language, and knowing another foreign language.

None of the available factors had a statistically significant effect on the efficacy. In someinstances the number of cases by subgroups was too low to expect enough statistical power
for the test of hypotheses.

This result can be interpreted as a positive finding because it means that the busuu app works
similarly for people with different gender, age, race, employment status, native language, etc.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 21

Oral Proficiency
Table 13. Oral Proficiency Improvement (Initial to Final Level) (N=61)
Progress in Levels
0 Same Level
1 One Level Up
2 Two Levels Up
3 Three Levels Up
4 Four Levels Up
5 Five Levels Up
Total

Progress
People (N)
15
27
15
3
0
1
61

%
24.6
44.3
24.6
4.9
0
1.6
100

Study Time
Mean (Hours)
20.7
25.3
21.2
22.7
n/a
47.5
23.4

Overall 75.4% of the participants improved their oral proficiency by at least one level. The 95%
Confidence Interval14 is 63% to 85%. Almost a half (44.3%) improved by one level, a quarter(24.6%) improved by two levels and four people (6.5%) improved by more than two levels. Not
surprisingly people who did not improve their oral proficiency were the people with the lowest
amount of study time.

Given the study time requirements for the oral test subsample it looks as though 16 hours
within two months of study, or at least two hours a week, was sufficient time for most people
to achieve significant progress in their oral proficiency. The majority of people improved by
one or two levels which seems like a reasonable expectation. The ACTFL requirement to have
2-3 months between test-retest looks appropriate. This study just added the additional
requirement that in practice this means two months of study with at least two hours of study
a week.

14

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 22

Table 14. Oral Proficiency Improvement by Initial Level (N=61)
PeopleInitial Level
N
0 Un-Ratable
1 Novice Low
2 Novice Mid
3 Novice High
4 Intermediate Low
5 Intermediate Mid
Total

6
30
19
2
3
1
61

Improved(One or more levels)% Improved
83.3
66.7
84.2
100
66.7
100
75.4

Conclusions from the oral proficiency tests were less clear than the written proficiency tests.
People who started at rock bottom level as Un-Ratable and people at Novice Mid level
improved the most. People at Novice Low level improved the least probably because this level
can have a very diverse population; from people who literally know just a few words to people
who know much more but not in a consistent way and they cannot speak Spanish in complete
sentences. The more advanced levels had very few cases to make meaningful conclusions. So
it looks like the oral proficiency improvement does not depend on the initial level in a clear
fashion.

Factors for Oral Proficiency
We investigated the impact of some quantifiable factors on the oral proficiency measure,namely age, gender, education, race, employment, device used, reason for studying Spanish,presence of people around the participant who spoke Spanish (spouse, friend, parents,
grandparents, etc.), native language, and knowing another foreign language. None of theavailable factors had a statistically significant effect on the oral proficiency.

As with the factors for written proficiency, this result can be interpreted as a positive finding
because it means that the busuu app works the same way for people with different gender,
age, race, employment status, native language, device used etc. regarding their oral
proficiency.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 23

Oral and Written Proficiency RelationshipThe correspondence between the initial written proficiency and the improvement in the oral
proficiency is presented below.

Table 15. Oral Proficiency Improvement by Initial WebCAPE Level (N=61)(Predictive Relation)
PeopleInitial Written Proficiency(WebCAPE Level)N

1 First Semester
2 Second Semester
3 Third Semester
Total

44
13
4
61

Improved Oral Proficiency(One or more levels)
%
72.7
76.9
100.0
75.4

The above relationship seems to confirm the common sense logic: people who had better
initial written proficiency tended to have better oral proficiency gains at the end of the study.
This conclusion is in a predictive fashion since if we know the initial written proficiency we canpredict the direction of the final oral gain: the higher the initial written proficiency the bigger
the gain in oral proficiency. This is just a direction of the effect because it is not a statisticallysignificant result due to the small sample size of some of the groups.
The relationship between the final written proficiency and the oral gain is presented below.

Table 16. Oral Proficiency Improvement by Final WebCAPE Level (N=61)(Concurrent Relationship)
Final Written Proficiency(WebCAPE Level)
1 First Semester
2 Second Semester
3 Third Semester
4 Fourth+ Semester
Total

People
N
20
20
12
9
61

Improved Oral Proficiency(One or more levels)
%
65.0
80.0
75.0
88.9
75.4

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 24

The interpretation is that people who at the end of the study still had the lowest level (First
Semester) of written proficiency tended to have the lowest oral proficiency gain. Again, this isjust the direction of the effect; because of the small sample size the result is not statisticallysignificant.

The Best Results: People Who Gained in Both Written and Oral Proficiency
Overall 84% of the participants improved their written proficiency (gain in WebCAPE points)
and 75% improved their oral proficiency by at least one level. The relationship between thetwo is presented below.

Table 17. Oral and Written Proficiency Improvement (N=61)
Improved Written Proficiency(Gain in WebCAPE score)

Improved Oral Proficiency(One or more levels)
No
Yes

No

0 (0%)

6 (9.8%)Yes

15 (24.6%)

40 (65.6%)

The interpretation of this relationship is as follows: all people who took both the written andoral tests twice gained either in oral or written proficiency. In other words, the study showsthat if a busuu user does at least two hours of study a week for two months s/he will improvein written or oral proficiency, or both. Almost 66% of the people improved both their written
and oral proficiency. About 25% improved only their written proficiency and about 10%
improved only their oral proficiency.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 25

6. User Satisfaction
After the study the participants were asked for their opinion about busuu, specifically how
easy it was to use, how helpful, enjoyable, and satisfactory.
Table 18. User Satisfaction (N=95)
Do you agree with thefollowing statement?
“busuu was easy to use”
“busuu was helpful instudying Spanish”“I enjoyed learning Spanish
with busuu”“I am satisfied with busuu”

Strongly Disagree/
Disagree
5.3

Neither Disagree
nor Agree
8.4

Percent
Agree/
Strongly Agree
86.3

6.3

9.5

84.2

5.3

16.8

77.9

11.6

14.7

73.7

After two months of study, the majority of users (74% to 86%) agreed with the positive
statements that: busuu was easy to use, helpful, they enjoyed learning with busuu and weresatisfied with it.

In the exit survey a special question was included: “How likely are you to recommend busuu toa colleague or friend?” with 11 possible answers, from 0 “Very unlikely” to 10 “Very likely”.
The answers to this question were used to compute the so called Net Promoter Score (NPS).
This is “a management tool that can be used to gauge the loyalty of a firm's customer
relationships” (Wikipedia). It was developed by Reichheld (2003) and it categorizes users inthree categories: “Promoters” (answers 9, 10), “Passives” (answers 7, 8), and “Detractors”(answers 0-6). NPS is equal to the difference between “Promoters” and “Detractors” and ingeneral it can vary from -100 (all detractors) to + 100 (all promoters). As a rule positive NPS isgood news for the company and the higher the score the better indicator for the company.
From our exit survey the “Promoters” were 37.9% and the “Detractors” were 29.5% and“Passives” were 32.6%. The busuu NPS was +8.4 which is a positive result.

All of the respondents in the exit survey declared that they will continue to use busuu after the
study ends.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 26

7. Limitations of the Study
On the positive side, this was the first study for our Research Team in which the language tests
were proctored (100% of the initial tests and most of the final tests) and both written and oral
proficiency were evaluated with an objective measure of the study time. Neither of the tests
was tailored to any specific learning tool, including busuu. On the one hand, some participantsin the study complained that the tests sometimes contained words or expressions that were
not part of their regular course with busuu. On the other hand, people insisted that they hadlearned a lot more than the tests asked for. The tests are valuable as an independent tool forevaluation which allows us to compare efficacy across different apps, however they do notprovide a complete measure of the exact progress of users.

There are some limitations of the study, mostly related to the instruments and technologicallimitations. The WebCAPE written test measures the progress of beginner/novice users of
Spanish well, but it is not suitable to measure the progress of very advanced users. Also, morestudy time is required for advanced users because it takes longer to achieve mastery of higher
language levels. Participants who started at rock bottom as true beginners (WebCAPE scoreclose to 0) gained much faster per study hour than people who started at the level of a secondor third college semester of Spanish.

The Research team sent e-mail messages every week with individualized information about
the study time for the previous week. This seemed to stimulate the study process. In normal
settings when people work individually on their studies, this stimulation is not available. Many
participants suggested adding a clock and time tracker to the software so they can be aware of
how much time they spend studying. The average study time was about two hours of study aweek but for some of the participants this was too much. The results of the study should bevalid in a setting where the users study regularly for about two hours a week for two months.
The study results could be generalized for studying Spanish with busuu. For other languages
more studies are necessary to confirm these findings, although there is no obvious reason in
the literature that the results should be markedly different.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 27

There are few other studies with a direct objective measure of efficacy available to comparewith the results of this study. More help is needed from users, investors, and analysts torequire the creators of language learning apps to provide independent efficacy measures.

8. Conclusion
The busuu efficacy study is based on a final random sample of 144 people, 18 years of age orolder, residing in or near London, UK or New York, US. They were not of Hispanic origin andthey were novice/beginner users of Spanish.

Overall 84% of the participants improved their written proficiency (gained WebCAPE points).
The main goal of measuring the efficacy of busuu was achieved with this study. The results
show that, on average, one hour of study with busuu alone leads to an improvement of 12
points on the college placement test WebCAPE. There is a lot of variability of the efficacy andthe 95% confidence interval is between 8 and 16 points per hour.

In other words, a busuu user would need on average 22.5 hours to complete the requirementsfor one college semester of Spanish. The transformed upper and lower limits are between 17
and 34 hours of study.

The main factor for the progress is the initial level of language knowledge of the participants.
The novice/beginner users (First semester) gain faster, with an average of 13.6 points per one
hour of study and the more advanced users (Second and Third semester) gain on average 3
points per one hour of study.

Using busuu for two months (two hours of study a week) improved the oral proficiency of 75%of the users. The 95% Confidence Interval is 63% to 85%.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 28

There are only a handful of known studies with direct objective measures of efficacy oflanguage learning apps. Among them the efficacy of busuu is the best so far. The creators ofother language apps should be encouraged to provide efficacy measures so users andinvestors can make more educated choices.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 29

9. Cited Literature
Agresti, A., B. Coull, 1998, Approximation is better than “exact” for interval estimation of
binomial proportions, American Statistician, 52, pp. 119–126.
Reichheld, F., 2003, "One Number You Need to Grow", Harvard Business Review, 2003
December.
Vesselinov, R. & J. Grego, 2016, Efficacy of New Language App, forthcoming.
Vesselinov, R. & J. Grego, 2012, Duolingo Effectiveness Study,
http://static.duolingo.com/s3/DuolingoReport_Final.pdf
Vesselinov, R., J. Grego, B. Habing, A. Lutz, 2009a, Measuring the Attitude and Motivation of
Rosetta Stone ® Users, http://vesselinov.com/RV_Language_Motivation.pdf .
Vesselinov, R., J. Grego, B. Habing, A. Lutz, 2009b, Comparative Analysis of Motivation ofDifferent Language Learning Software, http://vesselinov.com/RV_ComparativeLanguage.pdf .
Vesselinov, R., 2008, Measuring the Effectiveness of Rosetta Stone®,
http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.

The busuu Efficacy Study

Vesselinov & Grego 2016

Page 30

10. Appendix
Figure A1. Sample Selection Tree

2716 people
Viewed invitation
page

1973 people

743 people(London 394, New York 349)
Initial PoolDid not
complete Entry
SurveyCompleted Entry Survey
634 people(London 344, New York 290)
Eligible Pool

109 people
Ineligible

Eligible

196 people(London 96, New York 100)
Initial Random Sample

438 people
Not selected or
refusedCompleted two initial tests

Final Sample

Did Not Complete

144 people(London 78, New York 66)

52 people

Completed two sets of tests andsatisfied study time requirements.

Did use additional learning tools;
Did not satisfy the study time requirements;
Did not complete the final tests;
Did not do oral practice (oral test only).