The italki Efficacy Study
FINAL REPORTRESEARCH TEAMROUMEN VESSELINOV1,2, PhD
Economics Department
Queens College,
City University of New Yorkroumen.vesselinov@qc.cuny.edu

JOHN GREGO2, PhD
Statistics Department
University of South Carolina
grego@stat.sc.edu

JANUARY 2018

1
2

Corresponding author.
This report represents the individual opinion of the authors and not necessarily of the two institutions.

The italki Efficacy Study

Vesselinov & Grego 2018

THIS PAGE LEFT INTENTIONALLY BLANKPage 1

The italki Efficacy Study

Vesselinov & Grego 2018

Page 2EXECUTIVE SUMMARY
This study was independently conducted by the Research Team from August 2017 toNovember 2017. A random representative sample of 125 italki users was drawn. The
participants took one college placement Spanish language test and one oral proficiency test,
then studied Spanish with italki for two months and took the same tests again. The
improvement in language abilities was measured as the difference between the final and theinitial language test results. The efficacy of italki was measured as written proficiency
improvement per one hour of study and the proportion of users who improved their oral
proficiency. The final sample size was 102.
MAIN RESULTS
Written Proficiency Gain:• Overall 78% of the participants improved their written proficiency.
• italki users need on average 19 hours of study in a two-month period
to cover the requirements for one college semester of Spanish.
Oral Proficiency Gain:• About 72% of the participants increased their oral proficiency by at least one level.

DETAILED RESULTS
Written Proficiency:• The efficacy of italki is a gain of about 14 test points per one hour of study.
For beginners, the gain is 21 points and for advanced users it is 6-7 points.
• About 51% of the participants moved up at least one college semester level.
Of those, 32% moved up one semester, 16% - two semesters, and 3% - three semesters.
Oral Proficiency:• About 72% of the participants increased their oral proficiency by at least one level.
Of those, about 49% moved up one level, 19% - two levels, and 4% - more than two levels.
User Satisfaction:• The clear majority of users thought that italki was easy to use (89%),
helpful (98%), enjoyable (98%), and they were satisfied with it (92%).
• italki received a positive Net Promoter Score of 52.8 from users.
• italki efficacy was not affected by gender, age, education, native language, etc.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 3

CONTENTS

1.

Introduction ......................................................................................................................... 4

2.

Research Design .................................................................................................................. 6

3.

Sample Description ............................................................................................................. 9

4.

Language Improvement and Study Time .......................................................................... 15

5.

Main Results ..................................................................................................................... 19

6.

User Satisfaction ............................................................................................................... 25

7.

Limitations of the Study.................................................................................................... 26

8.

Conclusion ........................................................................................................................ 27

9.

Appendix ........................................................................................................................... 29

The italki Efficacy Study

Vesselinov & Grego 2018

Page 4

1. Introduction
There is a growing interest in the world in using language software packages or language
applications (apps) by individuals, organizations and school districts. On the other hand, there aremany language apps and their number is growing. There is an urgent need for users, investors, andeducators to have better and comparable scientific evaluation about the different language apps’
abilities and efficacy. Users cannot rely on the self-serving marketing campaigns by the respective
companies. There are many claims like “Learn a foreign language in 10 days!” It is not clear how muchyou can learn for 10 days, or how many hours you have to study during these 10 days. It is very
unrealistic and unfounded expectation but more importantly usually this is just a claim without any
scientific evidence.

Since 2008 our research team has been working in the area of evaluating language learningapps. We have already conducted eight studies attempting to directly evaluate the efficacy, attitude
and motivation of some popular language learning apps like, Rosetta Stone, Duolingo, Busuu, Babbel,
and Hello English (Vesselinov 2009, Vesselinov et al. 2009a, 2009b, Vesselinov & Grego, 2012, 2015,2016a, 2016b, 2017).

In this study the Research Team is trying to evaluate the efficacy of a well-known language
learning product: italki3 which helps language learners connect with language teachers from all over
the world. Whether users are learning Chinese for business, Spanish for travel, or Arabic for personal
interest, italki connects people with professional teachers or tutors to help them become fluent.
Many people study a foreign language in a traditional way for years, and are still unable tospeak the language. Most of these people have very few opportunities to communicate with a personfrom that country. By connecting learners with online teachers, learners can become fluent fasterand have an authentic cultural experience.

3

www.italki.com Note. The name of the company is spelled with lower case “i”, italki.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 5

italki is one of the well-known leaders in online language education, and according to thecompany, it has over 3 million students and 5000+ teachers of over 100 world languages. italkiconnects students with teachers around the world for 1-on-1 language lessons.
italki was founded by Kevin Chen (US) and Yongyue Jiang (China) in 2007. The company isregistered in Hong Kong, with a development office in Shanghai, China.

This study was funded by italki but the data collection and the analysis were carried out
independently by the Research Team. The two language tests used in the study were
designed, developed and managed by two external independent testing companies.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 6

2. Research Design
A random sample for this study was selected from existing italki users. There were somerequirements for the potential participants.
-

Willing to study Spanish using only italki for two months, and take two sets of language
tests;-At least 18 years of age;-

Not advanced learners of Spanish.

The last requirement was due to the fact that the written language placement test used in the
study has placement in college Semester 4+ as its highest evaluation group and it has limited
capabilities to test very advanced users. The oral proficiency test has no limitations because
the evaluation is done by independent raters.

The recommended goal for the participants in the study was to use italki for at least 9 hours
during the two-month study. Based on our experience with previous studies we imposed athreshold of at least two hours of study for the written test. People with less than two hours ofstudy were not allowed to complete the study because there was not a sufficient effort formeasurable progress. For the oral proficiency test, the requirement was at least 9 hours ofstudy. The testing company requires 2-3 months of study for test-retest but does not specify
how many hours of study is required.

Spanish language was selected as one of the more popular languages and because of theexistence of previous research on Spanish for other language learning apps. The length of thestudy was approximately 8 weeks and it was conducted between the months of August 2017
and November 2017. People who successfully completed the study were given a 30% refund(up to $100) of the amount they paid for lessons during the study.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 7The main instrument for evaluating the level of knowledge of Spanish was the Web Based
Computer Adaptive Placement Exam4 (WebCAPE test). It is an established university
placement test and it is offered in ESL, Spanish, French, German, Russian and Chinese. It was
created by Brigham Young University and is maintained by the Perpetual Technology Group. A
more detailed description of the test can be found at their website5.

The Spanish WebCAPE test has a very high validity correlation coefficient (0.91) and very highreliability (test-retest) value of 0.81. The test is adaptive so the time for taking the test varies
with an average time of 20-25 minutes. The WebCAPE test gives a score (in points) and basedon that score places the students in different level groups (college semesters).
Table 1. Spanish WebCAPE Test Cut-off Points
WebCAPE Test Points
Below 270
270-345346-428
Above 428

College Semester Placement
Semester 1
Semester 2
Semester 3
Semester 4+

The WebCAPE results alone cannot give a clear picture about the efficacy of the language
learning app because they do not account for the time spent studying. That is why we arerelying on a direct and objective measure of efficacy which is defined as follows:

Efficacy Effect Improvement of language skills Final-Initial WebCAPE test score=
EffortStudy time
Hours of study

Efficacy=Improvement per one hour of study
This measure includes both the amount of progress made by each study participant and theamount of their effort. It is a fair measure of efficacy and also a direct and objective measure

4

Spanish WebCAPE Computer-Adaptive Placement Exam by Jerry Larson and Kim Smith, online version Charles
Bush. ©1998, 2004 Humanities Technology and Research Support Center, Brigham Young University.
5
http://www.perpetualworks.com/webcape/overview

The italki Efficacy Study

Vesselinov & Grego 2018

Page 8of efficacy. Direct, because it includes directly the effect and the effort. Objective, because theeffect is measured by an independent college placement test (instead of our own test) and theeffort is measured by the actual time for italki lessons.

The second test used in the study was the Oral Proficiency Interview by Computer® (OPIc)6
created by Language Testing International (LTI). LTI is the exclusive licensee of the American
Council on the Teaching of Foreign languages (ACTFL). The online test is recorded and therecording of the test is reviewed and evaluated by independent raters.
Table 2. OPIc Ratings
UR
NL
NM
NH
IL
IM
IH

Un-Ratable
Novice Low
Novice Mid
Novice HighIntermediate Low
Intermediate Mid
Intermediate High

AL
AM
AH
S

Advanced Low
Advanced Mid
Advanced HighSuperior

The specific definition of the levels is presented on the company’s webpage7.

67

http://www.languagetesting.com/oral-proficiency-interview-by-computer-opic
http://d2k4mc04236t2s.cloudfront.net/wp-content/uploads/2013/07/ACTFL-Proficiency-Guidelines-2012.pdf

The italki Efficacy Study

Vesselinov & Grego 2018

Page 9

3. Sample Description
The entire sample selection process is graphically represented in the Appendix, Figure A1.
E-mail messages were sent out to italki clients with an invitation to participate in the researchstudy. If they accepted the invitation they were asked to complete the online Entry Survey
with some demographic questions and questions about their knowledge of Spanish. In all 1631
people viewed the invitation page and of those 388 successfully completed the Entry Survey.
This was the initial pool of respondents in the study.

Initial Pool (N=388)
The initial pool of potential participants consisted of people from the US (N=222) and outside
the US (N=166) and almost half of them (48.8%) were female. The mean age was 36.3 years
and they were well educated: 37% had a graduate (MA, PhD) degree and 45% had a BA orequivalent college degree. Only 8.5% had just High School or less and about 9% has somecollege but no degree. Most of the people were employed, either full time or part-time (72%),
4% were unemployed, 11% were students, 6% retired and 8% had other employment.

For 75% of the initial pool English was their native language and the remainder (25%) included
29 other languages: Arabic, Bengali, Chinese, Croatian, Dutch, Filipino, Finnish, French,
German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Macedonian,
Mandarin, Mongolian, Polish, Portuguese, Russian, Slovak, Swedish, Telugu, Turkish, Urdu,
Vietnamese.
Almost 77% described themselves as Beginner users or Never Studied Spanish. About 14% of
the respondents’ spouse, partner, or close friends spoke Spanish. A very small proportion (5%)
of their parents, grandparents, or great-grandparents spoke Spanish.

The primary reason for studying Spanish was personal interest (63%), followed by travel (18%),
business or work (15%), etc.
More than 60% of the initial pool had studied a foreign language before (mostly at school orcollege).

The italki Efficacy Study

Vesselinov & Grego 2018

Page 10

Pool of Eligible Participants (N=284)
From the Initial Pool (N=388) we excluded the following ineligible participants:•

People who were younger than 18 years of age.

•

People with advanced or fluent Spanish.

•

People who lived in a Spanish speaking country.

Altogether 104 people were ineligible for this study and the final pool of eligible participants
for sample selection was N=284. Of them 163 were from the US and the rest (N=121) werefrom outside the US.
The pool of eligible potential participants had a mean age of 36 years, from 18 years old to 73
years old, with 49.6% female users. The pool of eligible users was very well educated with only
about 5% with just a High School diploma or less and 38% had a masters or PhD. About 46%
had a BA or equivalent. About 74% were employed full time or part time, 9% were students,and 5% were unemployed. For 78% of them, English was their native language and 56% of thepool knew at least one foreign language.

Initial Random Sample (N=125)
The research design suggested an initial sample size of N=150 based on our previous studies’
effect size results and drop-out rate. The people in the initial sample were randomly selectedfrom the pool of eligible participants. They completed the baseline WebCAPE placement testin Spanish and the oral proficiency OPIc test. Some people (n=25) could not complete theinitial tests due to technical problems with OPIc and other reasons.
The initial random sample included 125 people with a mean age of 38 years, from 19 years old
to 67 years old, with 52.8% female users. The initial sample participants were from the US(N=78) and outside the US (N=47).
More than half (52.5%) of the initial sample were female, the mean age was 38 years and they
were well educated: 43% had a graduate (MA, PhD) degree and 43% had a BA or equivalentcollege degree. Only 2% had just High School or less and 12% had some college but no degree.
Most of the people were employed either full time or part-time (78%), 8% were unemployed,
4% were students, 6% were retired and 8% had other employment.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 11

For 78% of the initial sample, English was their native language and the remainder (22%)
included 14 other languages: Chinese, Dutch, Filipino, French, German, Greek, Hungarian,Portuguese, Russian, Swedish, Telugu, Turkish, Urdu, and Vietnamese.
All of them described themselves as Beginner users or Never Studied Spanish. About 9% of the
respondents’ spouse, partner, or close friends spoke Spanish. A very small proportion (3%) of
their parents, grandparents, or great-grandparents spoke Spanish.
The primary reason for studying Spanish was personal interest (63%), followed by travel (18%),
business or work (16%), and 3% other reasons.
More than 63% of the initial sample had studied a foreign language before (mostly at school orin college).

Table 3. Initial Random Sample: Age and Gender Distribution (N=125)Age
18-20 years old
21-30 years old
31-40 years old
Over 40 years old
Total

Female (N)
1
24
20
21
66

Male (N)
0
22
15
22
59

Total (N)
1
46
35
43
125

Percent
0.8
36.8
28.0
34.4
100.0

The written proficiency of the initial study sample was as follows:
Table 4. Initial WebCAPE Semester Placement (N=125)
College Semester
First
SecondThird
Fourth+Total

People (N)
66
34
25
0
125

Percent
52.8
27.2
20.0
0
100.0

The majority (53%) of the participants were evaluated as novice/beginner users of Spanish andthey were placed in First Semester of Spanish. About 27% of the participants were placed in

The italki Efficacy Study

Vesselinov & Grego 2018

Page 12

Second Semester and 19% in Third Semester of Spanish. The mean WebCAPE score was 237(std8=121), corresponding to First college semester of Spanish.
The oral proficiency of the initial sample was as follows:
Table 5. Initial Oral Proficiency (OPIc) (N=125)
Proficiency level0 Un-Ratable
1 Novice Low
2 Novice Mid
3 Novice High
4 Intermediate Low
5 Intermediate Mid
6. Intermediate High
Total

People (N)
9
29
34
33
12
7
1
125

Percent
7.2
23.2
27.2
26.4
9.6
5.6
0.8
100.0

The majority of the participants (84%) scored at the Novice level of oral proficiency. A handfulof people (N=20) scored at the intermediate level. The unrated cases were basically people
who cannot speak any Spanish and kept silent or spoke English on their oral test.

8Standard Deviation.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 13

Final Study Sample (N=102)
The work on the study started in August 2017 and ended in November 2017 with about 8
weeks dedicated to study with italki with time for tests in the beginning and at the end of thestudy. During the study, the Research Team sent weekly e-mail reminders to the participants
with information detailing the amount of time they had used italki each week.

At the end of the study, we reviewed the time use of the participants. For the written
proficiency test (WebCAPE) based on our previous studies (Vesselinov & Grego, 2012, 2016)
the threshold was established at about two hours of study. For the oral proficiency test thethreshold was established at about 9 hours of study. In other words, participants with at least
two hours of study could take the written test and complete the study. If they had less than 9hours of study, they were not eligible to take the oral test. People with 9 or more hours ofstudy could take both written and oral tests.
Based on these criteria from the initial sample the following people were excluded:•

People who did not satisfy the study time requirements.

•

People who did not take the final tests.

All participants were instructed at the beginning of the study that they were allowed to use
only italki services to study Spanish for the duration of the study. Italki users study with atutor or teacher and in many cases their study plan included the use of other tools. The
restriction for this study was that the participants cannot use other language tools
independently of their tutors or teachers’ recommendations.
Nine people (7%) had less than two hours of study and they were excluded from the study.
Altogether 102 people took the final written test and 81 people took the final oral test. The
mean study time for the final study sample was about 13 hours.
The final study sample for written proficiency consisted of 102 people with at least two hours
or more of italki use and valid initial and final WebCAPE tests. The final subsample for oral
proficiency was part of the 102-person sample and consisted of 81 people with about 9 hoursof study or more and valid initial and final OPIc tests. The mean study time for the oral test
sample was about 16 hours.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 14

The final study sample (N=102) had a mean age of 39.5 years, from 19 years old to 67 years
old, with 53% female users. The users were very well educated with 44% holding a graduate
degree and 44% with a BA or equivalent, one person with less than HS, and 11% with somecollege but no degree. About 78% of them were employed full time or part time, 4% werestudents, 4% unemployed, 6% retired and 8% had other employment.

For 80% of the participants, English was their native language and the rest included: Chinese,
Dutch, Filipino, French, German, Greek, Hungarian, Nepalese, Russian, Swedish, Turkish, Urdu,
and Vietnamese. About 58% of the sample knew at least one other foreign language (notSpanish).

About 8% of the respondents’ spouse, partner, or close friends spoke Spanish. A small portion(2%) of their parents, grandparents, or great-grandparents spoke Spanish.
The primary reason for studying Spanish was personal interest (65%), followed by travel (18%),
business or work (14%), and other reasons (3%).
Table 6. Final Study Sample: Age and Gender Distribution (N=102)
Age
18 to 20 years old
21-30 years old
31-40 years old
Over 40 years old
Total

Female (N)
1
18
17
18
54

Male (N)
0
14
12
22
48

Total (N)
1
32
29
40
102

Total (%)
1.0
31.4
28.4
39.2
100.0

People from the final sample used different devices to study Spanish with italki. The majorityof them (92%) used a desktop/laptop computer. Other devices were also used (categories are
not mutually exclusive): tablet (11%), Android smartphone (10%), Apple iPhone (18%) andother devices (9%).

The italki Efficacy Study

Vesselinov & Grego 2018

Page 15

Final Study Sample vs Not Completed
From the initial random sample (N=125) 23 people (18.4%) did not complete the study for
different reasons: people who did not satisfy the study time requirements or people who did
not take the final tests.

We compared the two groups, the final sample of 102 people and the 23 people who did notcomplete the study by gender, age group, education, employment status, initial knowledge of
Spanish (initial WebCAPE score and OPIc) and reason for studying Spanish. There were no
statistically significant differences (p=.01) which means that people who did not complete thestudy were not very different from the ones that did, and their exclusion did not introduce anevident bias.

We also compared the sample composition for the US versus all other countries. There wereno statistically significant differences (p=.01) between the two subsamples on gender, agegroups, education, employment status, initial knowledge of Spanish (initial WebCAPE scoreand OPIc) and reason for studying Spanish.

4. Language Improvement and Study Time
Study Time
The study time was measured objectively by the actual purchased lesson time with an italki
tutor or teacher. The average study time for the final study sample (N=102) was about 13
hours, or one hour and 40 minutes a week. The average study time for the oral test eligible
participants (N=81) was about 16 hours, or two hours a week.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 16

WebCAPE Test Results
All participants took an initial WebCAPE test before the start of the study and then again at theend of the study. The progress or improvement was measured as the difference between the
final test score and the initial one.

Table 7. Language Improvement Written Proficiency (N=102)
WebCAPE Test Points
Statistics

Initial WebCAPE

Final WebCAPE

Mean (std)Median
95% Confidence Interval9

236.9 (123.8)
268.5
212.6 – 261.2

336.8 (163.7)
334.5
304.6 – 368.9

Improvement(Final-Initial)
99.9 (149.5)
74.5
70.5 – 129.2

The overall average improvement of about 100 points WebCAPE test points was statisticallysignificant with a 95% confidence interval from 71 to 129 points. This also means that the
improvement in the written proficiency for the final sample was statistically significant (at 5%
error). Overall 78% of all participants improved their written proficiency (increased their
WebCAPE score) with a 95% confidence interval10 of 68% to 85%.

There were 23 cases (22%) where study participants did not improve their WebCAPE result or
had a lower result at the end of the study compared to their initial level.
There are two plausible explanations for this fact. First, some of them were more advanced
learners of Spanish and gaining points at this higher level is generally more difficult andrequires more time. Second, some of them studied irregularly with more effort and study timein the beginning of the study and less towards the end of the study. These users were notexcluded from the sample, so the results can be generalized for all types of users.

9

We also bootstrapped (N=10,000) the confidence intervals but the results remained practically the same.
95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

10

The italki Efficacy Study

Vesselinov & Grego 2018

Page 17

College Semester Placement
Progress can be measured by movement from one semester level to a higher semester level
and the results are presented below.
Table 8A. WebCAPE Semester Placement (N=102)
College Semester
First
SecondThird
Fourth+Total

Initial Test%
People (N)
54
52.9
28
27.5
20
19.6
102

Final Test
People (N)
%
30
29
25
18
102

100

29.4
28.4
24.5
17.6
100

People at First Semester level decreased from 53% to 29% and the proportion of people in
Second and Third Semester level increased as well. There was nobody in 4th semester level inthe beginning compared to about 18% at the end of the study.
Another way of presenting the results is direct comparison in the following table.
Table 8B. Direct Comparison of WebCAPE Semester Placement (N=102)

INITIAL

Semester

FINAL
1

2

3

4

Total

1

27

15

9

3

54

2

2

9

10

7

28

3

1

5

6

8

20

Total

30

29

25

18

102

The italki Efficacy Study

Vesselinov & Grego 2018

Page 18

Oral Proficiency
The oral proficiency results are presented below.
Table 9. Oral Proficiency Placement (N=81)
Level
0 Un-Ratable
1 Novice Low
2 Novice Mid
3 Novice High
4 Intermediate Low
5 Intermediate Mid
6 Intermediate High
Total

Initial Test
People (N)
%
6
26
27
25
11
6
1
81

5.9
25.5
26.5
24.5
10.8
5.9
1.0
100

Final Test
People (N)
%
0
3
10
31
21
15
2
81

0
3.7
12.3
38.3
25.9
17.3
2.5
100

In the oral proficiency area, the results are strong. At the beginning of the study the truly
novice users (Un-Ratable and Novice Low) were almost 32% of the sample while at the end
their numbers decreased to about 4%. This is a completely different level of oral proficiency.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 19

5. Main Results
Written Proficiency
Table 10. Written Proficiency Improvement (N=102)
Level (Semester Change)-1 Negative change/Down
0 Same/No Change
1 One Semester Up
2 Two Semesters Up
3 Three Semesters Up
Total

Improved
People (N)
%
8
7.9
42
41.2
33
32.4
16
15.7
3
2.9
102
100

Overall about 51% of the participants moved up at least one semester. About 32% moved up
one semester, 16% moved up two semesters and 3% moved up three semesters. About 41%
stayed in the same semester they started in and 8 people moved down a semester. The
average study time for the people who decreased their score (n=8) was about 5 hours while
the average time for the people who improved at least by one semester was more than 17hours. People who stayed in the same semester on average studied about 8 hours.
As the results indicate, people who had invested the lowest amount of effort and study time
were unsurprisingly the ones who did not improve or worsened their written proficiency
measured by semester level.

The problem with this measure is that first, it does not account for the effort (study time) andsecond, moving up a semester is dependent on the exact initial level. For example, if a person
has initially 269 test points (First semester), only a 1-point gain is needed to move to Second
semester. Another person can start with 10 points as initial level (First semester), then gain
200 points and the new level (210 points) is still First Semester.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 20

The main efficacy measures are presented below.
Table 11. Main Result. Efficacy of italki (N=102)

Statistics
Mean
95% Confidence Interval

Efficacy
Improvement per one
hour of study
WebCAPE Test Points
14
7 – 2112

Time to cover the requirementsfor one semester of college
SpanishHours
1911
13 – 3813

On average italki users will gain 14 WebCAPE test points per one hour of study with a 95%
confidence interval of 7 to 21 test points per hour.

The main measure of italki efficacy is the improvement per one hour of study. In addition, if
we divide the required cut-off point (270) for WebCAPE Second Semester placement by theefficacy mean we can construct a new measure representing the time needed to cover therequirements for one college semester of Spanish. This is the one measure of efficacy that iseasy to understand and given the nature of the WebCAPE placement test, can be used forcomparison with other language apps.

In other words, on average, italki users will need 19 study hours during a period of two months
to cover the requirements for one college semester of Spanish with transformed lower andupper limits of 13 hours to 38 hours of study.

11

The threshold of 270 points divided by the mean efficacy (14 points).
We also bootstrapped (N=10,000) the confidence interval but the result remained practically the same.
13
The threshold of 270 points divided by the lower limit (7) and the upper limit (21) of the 95% CI.
12

The italki Efficacy Study

Vesselinov & Grego 2018

Page 21

Efficacy and the Initial Level of Knowledge of Spanish
Table 12. Efficacy by Initial Level of Language Ability (N=102)
Initial Level
College Semester
First
SecondThird
Total

People
N
54
28
20
102

Efficacy
Mean (std)20.9 (48.1)
7.6 (18.7)
6.0 (15.4)
14.3 (37.5)

The overall efficacy is 14 WebCAPE points per one hour of study, but novice users of Spanish
managed a bigger gain of 21 points per hour of study. For the second and third semester levels
the improvement was more modest at about 6-7 points per hour.

Factors for Written Proficiency
We investigated the impact of some factors on the efficacy measure, namely age, gender,
education, employment, reason for studying Spanish, presence of people around theparticipant who spoke Spanish (spouse, friend, parents, grandparents, etc.), native language,
and knowing another foreign language.
None of the available factors had a statistically significant effect on the efficacy. In someinstances, the number of cases by subgroups was too low to expect enough statistical power
for the test of hypotheses.
Italki users could chose freely a teacher or a tutor for their study. About 15% of theparticipants chose only tutors for their study, and about 58% chose only teachers, with therest using a mix of teachers and tutors. People who used only teachers for their study had
slightly higher improvement rate (80% vs 74%), but the difference was not statisticallysignificant.

This result can be interpreted as a positive finding because it means that italki works similarly
well for people with different gender, age, employment status, native language, etc.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 22

Oral Proficiency
Table 13. Oral Proficiency Improvement (Initial to Final Level)
Progress in Levels
0 Same Level
1 One Level Up
2 Two Levels Up
3 Three Levels Up
4 Four Levels Up
Total

Progress
People (N)
23
40
15
2
1
81

%
28.4
49.4
18.5
2.5
1.2
100

Overall 71.6% of the participants improved their oral proficiency by at least one level. The 95%
Confidence Interval14 is 61% to 80%. Almost half (49.4%) improved by one level, a fifth (18.5%)
improved by two levels and 3 people (3.7%) improved by more than two levels.
The average study time for people who did not improve (same level) was 13.7 hours; the sametime for people who improved one level was 14.5 hours, and people who improved more than
one level studied on average more than 20 hours.
Not surprisingly people who did not improve their oral proficiency were the people with thelowest amount of study time.

Given the study time requirements for the oral test subsample it looks as though 16 hours
within two months of study, or at least two hours a week, was sufficient time for most people
to achieve significant progress in their oral proficiency. The majority of people improved by
one or two levels which seems like a reasonable expectation. The ACTFL requirement to have
2-3 months between test-retest looks appropriate. This study simply confirmed the additional
requirement that in practice this means two months of study with at least two hours of study
a week.

14

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

The italki Efficacy Study

Vesselinov & Grego 2018

Page 23

Table 14. Oral Proficiency Improvement by Initial Level
PeopleInitial Level
N
0 Un-Ratable
1 Novice Low
2 Novice Mid
3 Novice High
4 Intermediate Low
5 Intermediate Mid
6 Intermediate High
Total

2
17
23
23
11
4
1
81

Improved(One or more levels)% Improved
100
88.2
87.0
65.2
54.5
0
0
71.6

People who started at rock bottom as Un-Ratable and people at Novice level improved themost (65% to 100%). People at higher intermediate levels improved less than novice. The
more advanced levels had very few cases to make meaningful conclusions.

Factors for Oral Proficiency
We investigated the impact of some quantifiable factors on the oral proficiency measure,namely age, gender, education, employment, reason for studying Spanish, presence of people
around the participant who spoke Spanish (spouse, friend, parents, grandparents, etc.), native
language, using teachers/tutors and knowing another foreign language. None of the available
factors had a statistically significant effect on the oral proficiency.

As with the factors for written proficiency, this result can be interpreted as a positive finding
because it means that italki works the same way for people with different gender, age,
employment status, native language, device used etc. regarding their oral proficiency.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 24

The Best Results: People Who Gained in Both Written and Oral Proficiency
Overall 78% of the participants improved their written proficiency (gain in WebCAPE points)
and 72% improved their oral proficiency by at least one level. The relationship between thetwo is presented below.

Table 17. Oral and Written Proficiency Improvement (N=81)
Improved Written Proficiency(Gain in WebCAPE score)

Improved Oral Proficiency(One or more levels)
No
Yes

No

5 (6.2%)

12 (14.8%)

Yes

18 (22.2%)

46 (56.8%)

The interpretation of this relationship is as follows: almost all people (93.8%) who took both
the written and oral tests twice gained either in oral or written proficiency. Almost 57% of thepeople improved both their written and oral proficiency. About 22% improved only theirwritten proficiency and about 15% improved only their oral proficiency.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 25

6. User Satisfaction
After the study the participants were asked for their opinion about italki, specifically how easy
it was to use, how helpful, enjoyable, and satisfactory.
Table 18. User Satisfaction (N=89)
Do you agree with thefollowing statement?
“italki was easy to use”
“italki was helpful in studying
Spanish”“I enjoyed learning Spanish
with italki”“I am satisfied with italki”

Strongly Disagree/
Disagree
6.7

Neither Disagree
nor Agree
4.5

Percent
Agree/
Strongly Agree
88.8

1.1

1.1

97.8

0

2.2

97.8

2.2

5.6

92.1

After two months of study, the vast majority of users (89% to 98%) agreed with the positive
statements that: italki was easy to use, helpful, they enjoyed learning with italki and they weresatisfied with it.

In the exit survey a special question was included: “How likely are you to recommend italki toa colleague or friend?” with 11 possible answers, from 0 “Very unlikely” to 10 “Very likely”.
The answers to this question were used to compute the so called Net Promoter Score (NPS).
This is “a management tool that can be used to gauge the loyalty of a firm's customer
relationships” (Wikipedia). It was developed by Reichheld (2003) and it categorizes users inthree categories: “Promoters” (answers 9, 10), “Passives” (answers 7, 8), and “Detractors”(answers 0-6). NPS is equal to the difference between “Promoters” and “Detractors” and ingeneral it can vary from -100 (all detractors) to + 100 (all promoters). As a rule, positive NPS isgood news for the company and the higher the score the better the indicator for the company.
From our exit survey the “Promoters” were 55.6% and the “Detractors” were 2.8% and“Passives” were 41.7%. The italki NPS was +52.8 which is a quite substantive result.

Almost all respondents (97.8%) in the exit survey declared that they will continue to use italki
after the study ends.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 26

7. Limitations of the Study
On the positive side, neither of the two language tests was tailored to any specific learning
tool, including italki. On the one hand, some participants in the study complained that the
tests sometimes contained words or expressions that were not part of their regular coursewith italki. On the other hand, people insisted that they had learned a lot more than the tests
asked for. The tests are valuable as an independent tool for evaluation which allows us tocompare efficacy across different apps, however they do not provide a complete measure of
the exact progress of users.

There are some limitations of the study, mostly related to the instruments and technologicallimitations. The WebCAPE written test measures the progress of beginner/novice users of
Spanish well, but it is not suitable to measure the progress of very advanced users. Also, morestudy time is required for advanced users because it takes longer to achieve mastery of higher
language levels. Participants who started at rock bottom as true beginners (WebCAPE scoreclose to 0) gained much faster per study hour than people who started at the level of a secondor third college semester of Spanish.

The Research team sent e-mail messages every week with individualized information about
the study time for the previous week. This seemed to stimulate the study process. In normal
settings when people work individually on their studies, this stimulation is not available. Many
participants suggested adding a clock and time tracker to the software, so they can be awareof how much time they spend studying. The average study time was a little less than twohours of study a week but for some of the participants this was too much. The results of thestudy should be valid in a setting where the users study regularly for about two hours a week
for two months.
The study results could be generalized for studying Spanish with italki. For other languages,
more studies are necessary to confirm these findings, although there is no obvious reason in
the literature that the results should be markedly different.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 27

There are few other studies with a direct objective measure of efficacy available to comparewith the results of this study. More help is needed from users, investors, and analysts torequire the creators of language learning apps to provide independent efficacy measures.

8. Conclusion
The italki efficacy study is based on a final random sample of 102 people, 18 years of age orolder.
Overall 78% of the participants improved their written proficiency (gained WebCAPE points).
The main goal of measuring the efficacy of italki was achieved with this study. The results
show that, on average, one hour of study with italki alone leads to an improvement of 14
points on the college placement test WebCAPE. There is a lot of variability in the efficacy andthe 95% confidence interval is between 7 and 21 points per study hour.

In other words, an italki user would need on average 19 hours to complete the requirementsfor one college semester of Spanish. The transformed upper and lower limits are between 13
and 38 hours of study.

The main factor for progress is the initial level of language knowledge of the participants. The
novice/beginner users (First semester) gain faster, with an average of 21 points per one hourof study and the more advanced users (Second and Third semester) gain on average 6-7 points
per one hour of study.

Using italki for two months (about two hours of study a week) improved the oral proficiency of
72% of the users. The 95% Confidence Interval is 61% to 80%.

There are only a handful of known studies with direct objective measures of efficacy oflanguage learning apps. Among them, the efficacy of the written proficiency of italki is the best
so far. The creators of other language apps should be encouraged to provide efficacy
measures so users and investors can make more educated choices.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 28

Cited Literature
1. Agresti, A., B. Coull, 1998, Approximation is better than “exact” for interval estimation of
binomial proportions, American Statistician, 52, pp. 119–126.
2. Reichheld, F., 2003, "One Number You Need to Grow", Harvard Business Review, 2003
December.
3. Vesselinov, R. and J. Grego, 2017, Hello English Efficacy Study.
http://centralsquarefoundation.org/grant/hello-english-efficacy-study/ , or
http://comparelanguageapps.com/documentation/HelloEnglish_2017Study.pdf
4. Vesselinov, R. and J. Grego, 2016b, The Babbel Efficacy Study.
http://comparelanguageapps.com/documentation/Babbel2016study.pdf , or
http://press.babbel.com/en/releases/2016-09-29-Spanish-Study.html
5. Vesselinov, R. and J. Grego, 2016a, The Busuu Efficacy Study.
http://comparelanguageapps.com/documentation/The_busuu_Study2016.pdf , orhttps://blog.busuu.com/wp-content/uploads/2016/05/The_busuu_Study2016.pdf
6. Vesselinov, R. and J. Grego, 2015, Efficacy of New Language App,
http://comparelanguageapps.com/documentation/LA_Final_Report.pdf .
7. Vesselinov, R. and J. Grego, 2012, Duolingo Effectiveness Study.
http://comparelanguageapps.com/documentation/DuolingoReport_Final.pdf , or
http://static.duolingo.com/s3/DuolingoReport_Final.pdf
8. Vesselinov, R., J. Grego, B. Habing, A. Lutz, 2009b, Measuring the Attitude and Motivationof Rosetta Stone® Users.
http://comparelanguageapps.com/documentation/MeasuringTheAttitudeandMotivationofRSUse
rs.pdf
9. Vesselinov, R., J. Grego, B. Habing, A. Lutz, 2009a, Comparative Analysis of Motivation ofDifferent Language Learning Software.
http://comparelanguageapps.com/documentation/ComparativeMotivationAnalysisofDifferentLa
nguageSoftware.pdf
10. Vesselinov, R., 2009, Measuring the Effectiveness of Rosetta Stone®.
http://comparelanguageapps.com/documentation/MeasuringTheAttitudeandMotivationofRSUse
rs.pdf , or http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS5.pdf.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 29

9. Appendix
Figure A1. Sample Selection Tree

1631 people
Viewed invitation
page1243 people

388 people
Initial PoolDid not complete
Entry Survey

Completed Entry Survey

284 people
Eligible Pool104 people

Eligible

Ineligible

125 people
Initial Random Sample

159 people
Not selected

Completed two initial tests

Final Sample

Did Not Complete

102 people

23 people

Completed final tests and satisfied
study time requirements.

Did not satisfy the study timerequirements;
Or, did not complete the final tests.

The italki Efficacy Study

Vesselinov & Grego 2018

Page 30

Table A1. Geographic Distribution Countries (Number of people)
Code

State

38
2
1
3
5
4
6
34
7
11
8
9
10
39
12
13
15
16
14
17
18
19
20
21
23
22
24
26
25
27
28
29
30
33
31

United Arab Emirates
Austria
Australia
BelgiumBrazilBelizeCanadaSwitzerland
China
Germany
Estonia
FinlandFranceUnited Kingdom
GreeceHungary
Indonesia
IrelandIndia
Italy
JamaicaJapan
Kazakhstan
LatviaMali
Malaysia
Netherlands
NorwayNew ZealandPhilippines
PolandPortugal
RussiaSwedenSingaporeCountry
Code
AE
AT
AU
BE
BR
BZ
CA
CH
CN
DE
EE
FI
FR
GB
GR
HU
ID
IE
IN
IT
JM
JP
KZ
LV
ML
MY
NL
NO
NZ
PH
PL
PT
RU
SE
SG

Initial
Pool
1
2
19
1
7
1
10
2
5
7
1
1
8
42
2
1
2
4
4
4
2
2
2
1
1
1
1
1
1
1
2
1
6
1
3

Eligible
Pool
1
2
12
1
3
1
7
1
5
6
1

InitialSampleFinal
Sample

4
1
1

4

5

5

2
1

2
1

6
33
2
1
2
3
4
4
2
2
1
1
1
1
1
1

2
17

1
13

1

1

1
1
1
5
1
3

1

1
1
1

1
1
1

1

1

1
1

1

1
3
1
1

3
1
1

The italki Efficacy Study

Vesselinov & Grego 2018

Page 31

Table A1. Continued and end.Geographic
Code
32
35
36
37
40
41
42
43
44
45
46
47
48
49
Total

State

Country
CodeSlovakia
SK
ThailandTH
TurkeyTR
UkraineUA
USA
US
Spanish-Speaking Countries
Chile
CL
Colombia
CO
Costa RicaCR
Dominican Republic
DO
EcuadorEC
Honduras
HN
MexicoMX
Paraguay
PY
Spain
ES

Initial
Pool
1
1
1
1
222

Eligible
Pool
1
1
1
1
163

InitialSampleFinal
Sample

78

65

284

125

102

1
2
1
1
1
1
2
1
2
388

The italki Efficacy Study

Vesselinov & Grego 2018

Page 32

Table A2. Geographic Distribution: US States (Number of people)
Code

State

ST

2
1
4
3
5
6
7
9
10
11
15
12
13
14
16
17
18
21
20
19
22
23
25
24
26
33
34
27
29
30
31
28
32
35

AlaskaAlabamaArkansas
Arizona
California
Colorado
Connecticut
Delaware
FloridaGeorgia
Iowa
Idaho
Illinois
Indiana
KansasKentuckyLouisiana
Massachusetts
Maryland
Maine
MichiganMinnesota
MissouriMississippi
Montana
North Carolina
North Dakota
Nebraska
New Hampshire
New Jersey
New Mexico
NevadaNew York
Ohio

AK
AL
AR
AZ
CA
CO
CT
DE
FL
GA
IA
ID
IL
IN
KS
KY
LA
MA
MD
ME
MI
MN
MO
MS
MT
NC
ND
NE
NH
NJ
NM
NV
NY
OH

InitialPool

Eligible
Pool

InitialSampleFinal
Sample

1

1

1
38
8
1
3
12
4
2
1
12
1

1
31
4
1
2
11
3
2
1
9

1
18
2
1
1
1
1
2

1
15
1
1
1
1
1
2

8

6

3
1
8
6
1
3
3
1

3

1

1

5
4
1
2
1
1

2
3
1
1

2
2
1
1

5

2

2

2

1
10

9

2

1

25
2

18
1

9

9

The italki Efficacy Study

Vesselinov & Grego 2018

Page 33

Table A2 Continued and end.
Code

State

ST

36
37
38
39
40
41
42
43
44
46
45
47
49
48
50

Oklahoma
OregonPennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Virginia
Vermont
Washington
Wisconsin
West Virginia
Wyoming
District of Columbia
Unknown state (but US)

OK
OR
PA
RI
SC
SD
TN
TX
UT
VA
VT
WA
WI
WV
WY
DC

Total

USA

InitialPool

Eligible
Pool

InitialSampleFinal
Sample

4
5

2
4

1
3

1
3

1

1

3
13
2
9

1
11
2
5

1
3
1
3

2
1
3

4
2

4
2

3
1

3

1
25

18

6

4

222

163

78

65

******************* LAST PAGE OF THE italki FINAL REPORT *******************