Mango Languages Efficacy Study
FINAL REPORTRESEARCH TEAMROUMEN VESSELINOV1,2, PhD
Queens College, City University of New YorkJOHN GREGO2, PhD
University of South Carolina

October 2019

1
2

Corresponding author: roumen.vesselinov@qc.cuny.edu .
This report represents the individual opinion of the authors and not necessarily of their institutions.

Mango Languages Efficacy Study 2019
THIS PAGE LEFT INTENTIONALLY BLANKPage 1

Mango Languages Efficacy Study 2019

Page 2EXECUTIVE SUMMARY
The Research Team independently conducted this study from July to September 2019.
A random representative sample of 95 novice Spanish learners participated in the study.
The participants took one set of Spanish language written and oral proficiency tests in the beginning of
the study, then studied Spanish with Mango Languages for two months and took the same tests again.
The improvement in language proficiency was measured as the difference between the final and theinitial test results. The Efficacy of Mango Languages was measured as improvement per one hour ofstudy.
MAIN RESULTS
Overall Written Proficiency Gain:• Overall 82% of the participants improved their written proficiency3 during the study;• Novice users need on average 15 hours of study in a two-month period to cover
the requirements for one college semester of Spanish;Overall Oral Proficiency Gain:• Overall 77% of all study participants increased their oral proficiency4 during the study;
Oral Proficiency Gain for Participants with 8 Hours of Study or More:• 85% of the participants increased their TNT oral proficiency4;• 74% of the participants increased their TNT-ACTFL estimation5 level;• 72% of the participants increased their TNT-CEFR estimation6 level;
Efficacy:• On average participants gained 18 written proficiency WebCAPE points per one hour of study;• On average participants gained 0.13 points of the TNT oral proficiency test per one hour of study;
User Satisfaction:• The majority of participants thought that Mango Languages was easy to use (95%),
helpful (88%), enjoyable (91%), and satisfying (84%);• Mango Languages received a positive Net Promoter Score of +42 from the participants;• Mango Languages efficacy was not affected by gender, age, education, native language, etc.;• Participants’ motivation was very high with average level of 82%;3

Based on the college placement test WebCAPE.
Based on TrueNorth (TNT) oral proficiency test.
5
TNT estimation of American Council for Teaching Foreign Languages (ACTFL) levels.
6
TNT estimation of Common European Framework of Reference for Languages (CEFR) levels.
4

Mango Languages Efficacy Study 2019
THIS PAGE LEFT INTENTIONALLY BLANKPage 3

Mango Languages Efficacy Study 2019

Page 4

CONTENTSINTRODUCTION .........................................................................................................................6RESEARCH DESIGN ...................................................................................................................7
Study Instruments ..................................................................................................................... 8
STUDY SAMPLE .......................................................................................................................10
Sample Description ................................................................................................................. 13
Initial Language Tests ............................................................................................................. 16
Motivation ............................................................................................................................... 18
Language Profile ..................................................................................................................... 22
Study Time .............................................................................................................................. 24
LANGUAGE IMPROVEMENT .................................................................................................25
Written Proficiency Results .................................................................................................... 25
Oral Proficiency Results ......................................................................................................... 27
EFFICACY ..................................................................................................................................31
Written Proficiency Efficacy .................................................................................................. 31
Oral Proficiency Efficacy ....................................................................................................... 32
FACTORS FOR EFFICACY ......................................................................................................33
USER SATISFACTION ..............................................................................................................35
LIMITATIONS OF THE STUDY...............................................................................................36
CONCLUSION ............................................................................................................................38
CITED LITERATURE ................................................................................................................40
APPENDIX ..................................................................................................................................42
Table A1. Study Participants’ Geographic Distribution: US States ....................................... 42
Table A2. Motivation Scale ................................................................................................... 44
Table A3. Language Profile ................................................................................................... 46

Mango Languages Efficacy Study 2019
THIS PAGE LEFT INTENTIONALLY BLANKPage 5

Mango Languages Efficacy Study 2019

Page 6INTRODUCTION
This is a study designed to evaluate the efficacy of Mango Languages7.
The company describes Mango Languages as follows:“Since 2007, Mango Languages has built an award-winning learning environment around its
core purpose to enrich lives with language and culture. With powerful teaching methodology based on
research in Second Language Acquisition, Mango's learning system prepares learners to speak
confidently in real-life situations. Mango is the leading provider of language software in North
American public libraries and is used by individuals and organizations around the world. Learners canchoose from over 70 world languages and over 20 ESL courses, available on iOS, Android, and theweb. Key features include:• Conversation-based methodology that breaks up conversations into manageable chunks, thenguides learners to build them back up again, challenging them to use what they learn to produce
new phrases and sentences;• Carefully designed content that aligns with internationally recognized proficiency levels andstandards, so that learners can accurately understand their progress;• Language content created and recorded by native speakers for clear, high-quality language input;• Notes to explain difficult language concepts and put language learning in the context of culture;• Color mapping between the learners’ target and native languages to help learners understand thestructure of the target language;• Phonetic transcriptions, slow-paced recordings, and a voice comparison tool to help learners honetheir pronunciation;• Authentic passages for extra practice with reading and listening comprehension.
• A spaced-repetition algorithm that optimizes transferring words and phrases to long-term
memory.”This study was funded by Mango Languages, but the data collection and the analysis werecarried out independently by the Research Team. The language tests used in the study were designed
and developed by external independent testing companies.

7

www.MangoLanguages.com

Mango Languages Efficacy Study 2019

Page 7RESEARCH DESIGN
The random sample for this study was drawn from existing Mango Languages users
residing in the U.S. There were some additional requirements for the potential participants, who
had to:-be willing to study Spanish using only Mango Languages for two months. The minimum
requirement was a total of 2 study hours for the two-month period.

-

take two sets of written and oral proficiency language tests;-

be at least 18 years of age;-

be novice or beginner learners of Spanish.

Sample Size and Power Analysis
We based our power and sample size calculations on the typical results from our previous
studies (Rosetta Stone, italki, Babbel, and Busuu), using SAS 9.48 PROC POWER procedure. We canassume a minimum WebCAPE average gain of 80 points and a standard deviation up to 160 points.
With a 5% statistical significance level (alpha=0.05) and at least 80% power, a sample size of 30
participants of more will be sufficient to test for statistical significance of the WebCAPE gain.
For the oral proficiency we can assume at least 60% improvement based on our previous
studies. A sample size of 30 participants or more will be sufficient to test for statistical significance ofthe oral proficiency improvement as well.
To be consistent with our previous studies, we will perform a separate analysis only forparticipants with 8 study hours of more. In our previous studies about 30%-40% of the participants
reached 8 study hours. In addition, we expect that about 30% of the participants will not complete thestudy for different reasons.
Our initial sample size was set to 150 in order to reflect possible dropouts from the study andthe separate analysis for participants with 8 hours or more.
Spanish language was selected for this study because it is one of the most popular languages
studied by users of language learning apps, and for comparability with previous research on Spanish forother language learning apps (Vesselinov & Grego, 2009-20199). The study lasted approximately 8
weeks and was conducted between July and September 2019. Participants who successfully completed

89https://www.sas.com/
Except Hello English study in 2017.

Mango Languages Efficacy Study 2019

Page 8the study were given one-year of free access to Mango’s All Languages Package: the participant andfive of their friends were given unlimited access to all of Mango’s 70+ languages, as well as other
Mango products and features. Participants with 8 study hours or more additionally received a packageof Mango merchandise (2 pens, 2 notebooks and a water bottle). No other incentives were offered tothe participants.

Study Instruments
Test 1. WebCAPE: Written Proficiency: Vocabulary/Reading/Grammar
We used a college placement test called the Web based Computer Adaptive Placement
Exam (WebCAPE). This is an established university placement test and is offered in English,
Spanish, French, German, Russian and Chinese. It was created by Brigham Young University
and is hosted by Emmersion Learning10.
The WebCAPE has a very high validity correlation coefficient (0.91) and very highreliability (test-retest) value of 0.8611. The test is adaptive so the time for taking the test varies
with an average time of 20-25 minutes. The WebCAPE gives a score (in points) and based onthat score places the students in different group levels (college semesters; see Table 1).
Table 1. Spanish WebCAPE Cut-off Points
WebCAPE Points
Below 27012
270-345346-428
Above 428

College Semester Placement
Semester 1
Semester 2
Semester 3
Semester 4+

The WebCAPE results alone cannot give a clear picture of the efficacy of a language
learning app because they do not account for the time spent studying. We are therefore relying
on a direct and objective measure of efficacy, which is defined as follows:
Efficacy=Improvement per one hour of study.

Efficacy =10

Effect Improvement of language skills Final-Initial WebCAPE test score==
EffortStudy time
Hours of study

Currently WebCAPE is still listed with their old host at https://perpetualworks.com/
https://perpetualworks.com/webcape/details/
12
The same threshold of 270 points (four-semester Spanish) was used for all our previous language studies.
11Mango Languages Efficacy Study 2019

Page 9A similar efficacy measure will be computed for oral proficiency, using the TrueNorth Test
score described below.
The efficacy measure includes both the amount of progress made and the amount ofeffort invested by each study participant. This is a direct and objective measure of efficacy:
direct, because it includes directly the effect and the effort; objective, because the effect ismeasured by an independent college placement test (instead of our own test) and the effort ismeasured by the time recorded on the computer servers (instead of self-report).
Test 2. TrueNorth Test (TNT)13: Oral Proficiency Test
This is a newly developed oral proficiency test based on Elicited Imitation (EI) as atesting method in which participants hear an utterance in the target language and are prompted
to repeat the utterance as accurately as possible.
a. TNT oral proficiency score.
The TNT test gives an incremental score from 0.0 to 10.0 with zero being the lowest
level and 10 – the highest.
TNT also provides estimation of ACTFL14 and CEFR15 levels.
b. TNT-ACTFL estimation of oral proficiency.
The American Council for Teaching Foreign Languages (ACTFL) developed aproficiency scale to assess foreign language abilities. This scale includes four main groups16(Novice, Intermediate, Advanced, and Superior), with the first three divided into levels. The
levels are shown below:
Novice:1. Low 2. Mid 3. HighIntermediate:

4. Low 5. Mid 6. High

Advanced:

7. Low 8. Mid 9. High

10. Superior (no levels)

13

TrueNorth Test, https://truenorthtest.com
TrueNorth Technical Report:
https://truenorthtest.com/wp-content/uploads/2019/09/Spanish-Technical-Report_2019.9.16.pdf
15
Emmersion Learning, Inc.: “Crafted by PhD Psychometricians using patent-pending methods that correlate withstandard scales (CEFR & TOEFL)”, https://truenorthtest.com/about/16
https://www.actfl.org/publications/guidelines-and-manuals/actfl-proficiency-guidelines-201214

Mango Languages Efficacy Study 2019

Page 10c. TNT-CEFR estimation of oral proficiency.
The Common European Framework of Reference for Languages (CEFR)17 is designed
as a global standard for describing language proficiency. It has six levels, A1-A2 for beginner,
B1-B2 for intermediate, and C1-C2 for proficient.

STUDY SAMPLE
In July 2019, emails were sent to existing Mango Languages users with an invitation toparticipate in a Spanish language study for two months. They were directed to an online survey
designed by the Research Team. This survey collected demographic information, and selfevaluation of their language proficiency level. We received complete responses from 467
people which constituted the initial pool for the study. From this pool we excluded 47 people
who lived outside the US, were younger than 18 years of age, or had an intermediate or highlevel of knowledge of Spanish; the remainder formed the eligible pool (N=420) for this study.
We randomly selected 150 people from the eligible pool of participants and 149 of themcompleted the initial language test. They constituted our initial random sample (N=149).
The Mango Languages study continued for approximately two months (8 weeks),
starting in July 2019 and ending in September 2019. During the study, the Research Team sent
weekly e-mail reminders to the participants with information detailing the amount of time they
had used Mango Languages each week.
The final study sample consisted of 95 people who had at least two hours of study andhad valid initial and final sets of tests.
All participants were instructed at the beginning of the study that they could use only
Mango Languages to study Spanish for the duration of the study. In the exit survey eight people
stated that they had regularly used other language apps or spoke and consulted regularly withnative speakers of Spanish and were excluded from the final analyses. Other people had
occasionally used internet dictionaries, YouTube or translation websites and they were includedin the final analyses.

17

https://www.coe.int/en/web/common-european-framework-reference-languages

Mango Languages Efficacy Study 2019

Page 11

Final Study Sample versus Not Completed
From the initial random sample (N=149), 54 people (36.2%) did not complete the study
for two different reasons: 1) they did not take the final tests, or 2) they studied for less than 2hours during the two-month study. This dropout rate is about average in this line of research.
We compared the two groups, the final sample of 95 people and the 54 people who did
not complete the study by gender, age, education, employment status, initial knowledge of
Spanish and reason for studying Spanish. There were no statistically significant differences,which means that participants who did not complete the study were not very different from theones that did.

Mango Languages Efficacy Study 2019

Page 12Figure 1. Sample Selection Tree1312 people
Viewed invitation page
467 people
Initial Pool845 people

Completed Entry
SurveyDid not complete
Entry Survey

420 people
Eligible Pool

47 people

Eligible
149 people
Initial Random
SampleIneligible

271 people
Not selected

Completed initial test
95 people
Final Sample
Had at least 2 hours of study
and completed two sets oftests.

54 people
Did Not Complete
Did not complete the final test or,
had less than 2 hours of study.

Mango Languages Efficacy Study 2019

Page 13

Sample Description
In the final study sample (N=95), 70.2% were female. In our previous studies we havefound similar population phenomena: there are more female users of language apps willing toparticipate in research than male. Our analysis has shown that gender is not a statisticallysignificant factor for language learning improvement. This study sample by design isrepresentative of the population of people who seek to learn a foreign language with language
apps. We know from previous studies that this special population is older, more educated, andhas a higher proportion of female users (60%-70%) than the general population.
The age of participants varied from 21 to 66 years of age, with a mean age of 41.2 years.
The participants were very well educated with a majority of them having at least some college
experience, or an undergraduate or graduate degree.

Figure 2. Age Distribution

Mango Languages Efficacy Study 2019

Page 14

Table 2. Age and Gender DistributionAge
21-30 years old
31-40 years old
Over 40 years old
Total

Female (N)
14
26
26
66

Male (N)461828

Total (N)
18
32
44
94*

Total (%)
19.1
34.0
46.9
100

* Some participants declined to answer specific survey questions, so the number of answers
can be less than 95 in some tables in this report. The number of answers is reported in thetables.
Table 3. Education
Category
2. HS diploma or equivalent
3. Some college but no degree
4. College graduate, BA or equivalent
5. Some graduate school but no degree
6. Master’s degree (MA, MS)
7. PhD/MD/JD
Total

N
5
10
31
7
34
7
94

Percent
5.3
10.6
33.0
7.4
36.2
7.4
100

The majority of the people were employed full time (60.6%).
Table 4. Employment Status
1.
2.
3.
4.
5.
6.
7.
Total

Category
Employed full time
Homemaker
Employed part time
Unemployed
Retired
Other employment
Student

N
57
12
9
4
5
4
3
94

Percent
60.6
12.8
9.6
4.3
5.3
4.3
3.2
100

Eighty-seven percent of the participants were English native speakers and the rest
included native speakers of the following languages: Arabic, Bahasa Malaysia, Bahasa
Indonesia, Kristang, French, Haitian Creole, Hebrew, Mandarin, Russian, Shona, and Tagalog.
All participants in the final sample described themselves as beginner or novice Spanish
learners. About 28% of the respondents had a spouse, partner, or close friends who spoke
Spanish. About 3% had parents, grandparents, or great-grandparents who spoke Spanish.

Mango Languages Efficacy Study 2019

Page 15

About 71% of the final sample had studied a foreign language before (mostly at high school orcollege).
About 24% had lived outside the U.S. in a non-English speaking country for more than
6 months. About 19% of the respondents were raised in a multilingual or non-English speaking
household.
The primary reason for studying Spanish among participants was personal interest(49.5%), followed by business or work (30.5%) and travel (15.8%).
Table 5. Reason for Studying Spanish
Category
1. Personal Interest
2. Business/Work
3. Travel
4. Other
5. School
Total

N
47
29
15
3
1
95

Percent
49.5
30.5
15.8
3.0
1.1
100

Mango Languages Efficacy Study 2019

Page 16

Initial Language Tests
Test 1. Written18 proficiency: WebCAPE
All participants took an initial written proficiency test (WebCAPE) and the results are as
follows.
Figure 3. Initial WebCAPE Score

As expected, a large portion of the participants (45%) scored zero on the first WebCAPE. The
overall median WebCAPE score was 60 (IQR19=161) corresponding to first college semester of
Spanish.
Test 2. Oral Proficiency:
a. TNT Score.
TNT scores can vary from 0.0 to 10.0, and the initial test scores ranged from 0.1 to 5.2. The
overall median value was 2.7 (IQR=1.9) with 18% scoring very close to zero (0.1).

18
19

Reading/Grammar/VocabularyInterquartile Range= Third Quartile – First QuartileMango Languages Efficacy Study 2019

Page 17

Figure 4. Oral Proficiency: TNT Score.

TNT test offers estimates of ACTFL20 and CEFR21 levels. The initial TNT-ACTFL and TNTCEFR estimated levels are shown in Tables 6 and 7, respectively.
b. TNT-ACTFL estimation of oral proficiency.
Table 6. Initial TNT-ACTFL Estimation Levels
Level
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5
6.0

Description
Novice Low
Novice Low-Novice Mid
Novice Mid
Novice Mid – Novice High
Novice High
Novice High – Intermediate Low
Intermediate Low
Intermediate Low - Intermediate Mid
Intermediate Mid
Intermediate Mid - Intermediate HighIntermediate High
Total

20

N
20
12
0
30
0
24
0
8
0
1
0
95

Percent
21.1
12.6
0
31.6
0
25.3
0
8.4
0
1.1
0
100

TrueNorth Technical Report:
https://truenorthtest.com/wp-content/uploads/2019/09/Spanish-Technical-Report_2019.9.16.pdf
21
Emmersion Learning, Inc.: “Crafted by PhD Psychometricians using patent-pending methods that correlate withstandard scales (CEFR & TOEFL)”, https://truenorthtest.com/about/Mango Languages Efficacy Study 2019

Page 18c. TNT-CEFR Estimation of Oral Proficiency
Table 7. Initial TNT-CEFR Levels
Level
1.0
1.5
2.0
2.5
3.0

Description
Beginner A1
Beginner A1 – A2
Beginner A2
Beginner A2 – Intermediate B1
Intermediate B1
Total

N
32
30
24
8
1
95

Percent
33.7
31.6
25.3
8.4
1.1
100

Motivation
All participants completed a motivation scale in the beginning of the study to evaluate
the role of motivation on efficacy.
We adopted a motivation scale approach largely based on the second language (L2)
motivational self-system (Dörnyei, 2005, 2009) which stems largely from the concepts ofpossible selves and self-discrepancy theory. The model proposes that language learners are
guided by visions of ‘second language selves’, one which attracts them toward becoming an
idealized L2 user (ideal L2 self) and one which pushes them from societal obligation or a fearof failure (ought-to L2 self).
We adopted a specific 33 question/6 factor version of L2 Motivational Self System (seeAppendix, Table A2) created by Kong et al. (2018).
Kong et al. (2018) offer the following descriptions of the motivation scale elements:
1. Ideal L2 self: “The ideal L2 self refers to a positive future image of the L2 self. For
example, learners who have developed a vivid ideal L2 self are likely to endeavor to learn an L2
by imagining themselves communicating fluently using the L2 in the future.”
2. Ought-to L2 self: “(This element) pushes people from societal obligation or a fear of
failure.”3. International posture: “It captures a tendency to relate oneself to the international
community rather than any specific L2 group. The key characteristics of international posture
are described as an interest in global issues or international affairs, a willingness to travel, stay,or work abroad, and a readiness to interact with foreigners or foreign cultures.”Mango Languages Efficacy Study 2019

Page 19

4. Competitiveness: “Competitiveness can be described as the desire to excel incomparison to others and contends that a learner constantly compares oneself with one's
idealized self-image or with other learners, feels pressured to out-do other students.”
5. L2 learning Experience or Attitudes: “L2 learning experience is related to thelearners' environment including teachers, peer groups, curriculum, and their attitudes toward L2
learning.”
6. Learners' Intended Effort or Motivated Behavior in L2 Learning: This motivation
element evaluates how much effort users are determined to make and how hard they are readyto study.
Sixty-eight participants completed the motivation survey. As we can see from the tablebelow and Figures 6 to 11 below, the participants’ motivation is very high. The scaledimensions were re-coded, so the maximum motivation is equal to 100.
Table 8. Motivation Levels (N=68)
Motivation Dimensions
1. Ideal Self
2. Ought-to-Self
3. International Posture
4. Competitiveness5. Learning Attitude
6. Intended Effort
Overall Motivation

1st Quartile22
75
49
77
74
80
77
74

(%)
Median23
80
63
83
83
95
87
82

3rd Quartile24
95
80
90
90
100
93
88

The average level of the overall motivation was very high (Median=82%). From themotivation elements the highest level (95%) belongs to “Learning Attitude” which indicatesthat the participants were extremely eager to learn a new language. The element “Ought-toSelf” has the lowest level of all (63%) which suggests that the participants were not very afraidof failure or they were not that susceptible to pressure from societal obligation.

22

First 25% of the sample.
50% middle point.
24
First 75% of the sample.
23

Mango Languages Efficacy Study 2019
Figure 5. Total Motivation Level

Page 20(%)

As noted above, the average initial level of motivation was very high, and most people
were overall highly motivated. The lowest individual motivation level was 63%.

Mango Languages Efficacy Study 2019

Page 21

MOTIVATION LEVELS:
Figure 6. L2 Motivation (Max=100)
M1 “Ideal Self”

Figure 7. L2 Motivation (Max=100)
M2. “Ought-to Self”

Figure 8. L2 Motivation (Max=100)
M3. “International Posture”

Figure 9. L2 Motivation (Max=100)
M4. “Competitiveness”

Figure 10. L2 Motivation (Max=100)
M5. “Learning Attitude”

Figure 11. L2 Motivation (Max=100)
M6. “Intended Effort”

Mango Languages Efficacy Study 2019

Page 22

Language Profile
We asked participants to complete an adapted version of the Bilingual Language Profile(Birdsong et al., 2012). This provides a Global Language Score (GLS) for English and for
additional languages spoken by the participants. GLS is based on separate modules forevaluating language history, language use, language proficiency and language attitudes. GLS
can vary from 0 to 218, and we re-coded it, so the maximum is equal to 100.
For example, a GLS score of 218 (or rescaled as 100) would be appropriate forparticipants born into an English-speaking family, in an English-speaking country, who started
studying English immediately, for whom all classes at school were in English, who speak only
English all the time with family, friends, and at work. Their language history and language useare entirely English- based. They feel totally proficient in English, and they identify themselveswith an English-speaking culture.
Figure 12. GLS Score for English Language (N=68)

Max=100

The median English GLS percent was 97 (IQR=6.6) which corresponds to an initialsample of very strong English native speakers. Eighteen participants felt comfortable enough tocomplete the language profile for another language in addition to English.

Mango Languages Efficacy Study 2019

Page 23

The GLS score is lower if the participants started learning English at older age, attended
school with some instruction in a language other than English, or used another language to
speak with some of their friends and family, etc.
Figure 13. GLS Score for Language other than English (N=18) Max=100

The median GLS level for language other than English was 49 (IQR=50).
Figure 14. English vs Non-English GLS(n=18)

Mango Languages Efficacy Study 2019

Page 24

Participants with two languages fall mainly into two groups (Fig. 14):Cluster A: they have a less than perfect English profile (English GLS 60-80%), but they havestrong profile in another language, i.e. their native language (Non-English GLS 80-100).
Cluster B: they are native speakers of English (English GLS 90-100%) and have someknowledge of a second language (Non-English GLS 20-50%).

Study TimeWe measured the study time objectively by the actual server time on a weekly basis and wereported the time to the participants regularly via e-mail in order to encourage them to keep
studying. The average total study time for the final study sample (N=95) was about nine hours,
or a little over one hour per week. The total study time ranged from two hours to 48 hours.

Figure 15. Study Time Distribution

Hours

Mango Languages Efficacy Study 2019

Page 25LANGUAGE IMPROVEMENT
Written Proficiency Results
All participants took the initial WebCAPE before the start of the study and then again at the endof the study. We measured the progress or improvement as the difference between the final testscore and the initial one.
Figure 16. WebCAPE Gain (n=87)

Table 9. Language Improvement (Reading/Vocabulary/Grammar)
WebCAPE Points (n=87)
Statistics
Mean (std)
95% Confidence Interval

Initial WebCAPE Final WebCAPE
97.2 (106.9)
74.4 – 120.0

229.4 (120.5)
204 – 255

Improvement(Final-Initial)
132.2 (124.0)
104 – 161

The average overall improvement of 132 WebCAPE points was statistically significant with a
95% confidence interval from 104 to 161 points. This means that the improvement in thelanguage proficiency for the final sample was statistically significant (at 5% error). Overall 82%
of all participants improved their written language proficiency with a 95% Confidence interval25
of 72% to 88.4%.

25

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Mango Languages Efficacy Study 2019

Page 26

Only 16 participants out of 87, or 18% did not increase their WebCAPE score. There are twoplausible explanations for this fact. First, some participants scored higher on the initial test andgaining points at this higher level is generally more difficult and requires more time. Second,some participants studied irregularly with more effort and more study time in the beginning of
the study and less time towards the end of the study, so they may have forgotten what theylearned due to lack of continued practice.
College Semester Placement
We can measure progress by movement from an initial semester level to a final semester levelwith the results presented below.
Table 10. WebCAPE Semester Placement
College Semester
First
SecondThird
Fourth+Total

Initial Test%
People (N)
77
88.5
9
10.3
1
1.1
87

Final Test
People (N)
%

100

54
21
7
5
87

62.1
24.1
8.0
5.7
100

Participants at First Semester level decreased from 88.5% to 62.1% and the proportion in
Second to Fourth+ Semester level increased notably.
Table 11. Semester Improvement
Level (Semester Change)
-1
0
1
2
3

Negative change
Same/No Change
One Semester Up
Two Semesters Up
Three Semesters Up
Total

Improved
People (N)
%
3
3.2
55
63.2
18
20.7
8
9.2
3
3.4
87
100

Study Time
Mean (Hours)
Small n
8.7
10.3
15.3
Small n10.0

People who did not improve their semester placement studied the least (8.7 hours) while people
with a two-semester improvement studied the most (15.3 hours).
The problem with the semester improvement measure is that first, it does not account for theeffort (study time) and second, moving up a semester is dependent on the exact initial level. For
example, if a person initially has 269 test points (First semester), only one-point progress isMango Languages Efficacy Study 2019

Page 27

needed to move to Second semester. Another person can start with 10 points (First semester),then gain 200 points and the new level (210 points) is still First Semester. This measure is notvery informative, and it is presented here only for completeness.

Oral Proficiency Results
Test 2. TNT Oral Proficiency
a. TNT score
Figure 17. Oral Proficiency: TNT Score Gain(n=87)

Table 12. Language Improvement. Oral Proficiency by TNT score (n=87)
TNT Test Points
Statistics
Mean (std)
95% confidence interval

Initial TNT

Final TNT

2.4 (1.4)
2.1 – 2.7

3.4 (1.1)
3.2 – 3.7

Improvement(Final-Initial)
1.1 (1.6)
0.7 – 1.4

The average overall oral proficiency improvement of 1.1 TNT test points was statisticallysignificant with a 95% confidence interval from 0.7 to 1.4 points. Overall 77% of allparticipants improved their oral proficiency with 95% confidence interval26 of 67% to 84.6%.

26

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Mango Languages Efficacy Study 2019

Page 28

Oral Proficiency by TNT for Participants with 8 Hours of Study
Language Testing International, the exclusive licensee of ACTFL recommends27 8 weeks as theminimum time between test and retest for Group I languages which includes Spanish. In ourprevious studies we used ACTFL computerized oral proficiency test (OPIc) and we required atleast 8 hours for 8 weeks of study for oral evaluation. In order to make the results of this study
comparable to the previous studies we present similar results here.
Table 13. Oral Proficiency by TNT for People with 8 Hours of Study (n=46)
TNT Test Points
Statistics
Mean (std)
95% confidence interval

Initial TNT

Final TNT

2.2 (1.3)
1.9 – 2.6

3.7 (1.0)
3.4 – 4.0

Improvement(Final-Initial)
1.4 (1.5)
1.0 – 1.9

The average overall oral proficiency improvement of 1.4 TNT test points was statisticallysignificant with a 95% confidence interval from 1.0 to 1.9 points. This means that the
improvement in the oral proficiency for the final sample was statistically significant (at 5%
error). Overall 84.8% of all participants improved their oral proficiency with a 95% confidence
interval28 of 71.4% to 92.6%.

27

https://www.languagetesting.com/how-long-does-it-take

28

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Mango Languages Efficacy Study 2019

Page 29b. TNT-ACTFL Estimation
Comparable to the previous study, we consider oral proficiency by TNT-ACTFL estimation
only for participants with 8 hours of study or more.
Table 14. Oral Proficiency by TNT-ACTFL Estimation for People with 8 Hours of Study
Level Description
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5
6.0

Novice Low
Novice Low-Novice Mid
Novice Mid
Novice Mid – Novice High
Novice High
Novice High – Intermediate Low
Intermediate Low
Intermediate Low - Intermediate Mid
Intermediate Mid
Intermediate Mid - Intermediate HighIntermediate High
Total

N
10
5
0
15
0
14
0
2
0
0
0
46

InitialPercent
21.7
10.9
0
32.6
0
30.4
0
4.3
0
0
0
100

N
0
3
0
4
0
21
0
13
0
5
0

Final
Percent
0
6.5
0
8.7
0
45.7
0
28.3
0
10.9
0
100

Table 15. Oral Proficiency Increase by TNT-ACTFL Estimation (Final – Initial)
Gain
-1
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5

Description
Decreased one level
Stayed the same level
Increased half level
Increased 1 level
Increased 1.5 levels
Increased 2 levels
Increased 2.5 levels
Increased 3 levels
Increased 3.5 levels
Increased 4 levels
Increased 4.5 levels
Total

n
2
10
0
14
2
9
4
2
1
0
2
46

Percent
4.3
21.7
0
30.4
4.3
19.6
8.7
4.3
2.2
0
4.3
100

73.9 percent of the people with at least 8 hours of study increased their TNT-ACTFL estimationlevel by at least with one level and up to 4 levels. The 95% confidence interval29 is from 59.6% to84.4%. About 39% of the participants increased their oral proficiency with two or more levels.

29

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Mango Languages Efficacy Study 2019

Page 30c. TNT-CEFR Estimation
As with the case of TNT-ACTFL estimation, we considered the oral proficiency by TNT-CEFR
estimation only for people with 8 hours of study or more.
Table 16. Oral Proficiency by TNT-CEFR Estimation
Level Description
1.0
1.5
2.0
2.5
3.0

Beginner A1
Beginner A1 – A2
Beginner A2
Beginner A2 – Intermediate B1
Intermediate B1
Total

InitialN
Percent
15
32.6
15
32.6
14
30.4
2
4.3
0
0
46
100

Final
N
Percent
3
6.5
4
8.7
21
45.7
13
28.3
5
10.9
46
100

Table 17. Oral Proficiency by TNT-CEFR Estimation (Final – Initial)
Gain
-0.5
0
0.5
1
1.5
2

Description
Decreased half level
Stayed the same level
Increased half level
Increased 1 level
Increased 1.5 levels
Increased 2 levels
Total

n
2
11
15
13
3
2
46

Percent
4.3
23.9
32.6
28.3
6.5
4.3
100

71.7 percent of the people with 8 study hours or more increased their TNT-CEFR estimationlevel up to 2 levels. The 95% confidence interval30 is from 57.3% to 82.7%.

30

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Mango Languages Efficacy Study 2019

Page 31

EFFICACY
Written Proficiency Efficacy
Table 18. Written Efficacy of Mango Languages (N=87)
Efficacy =Improvement per one
hour of study
WebCAPE Points

Time to cover the placement
requirements for first semesterof college Spanish
Hours

Mean (std)18.3 (22.1)

14.831

95% Confidence Interval

13.6 – 23.0

11.7 – 19.932

Statistics

On average Mango Languages users will gain 18 WebCAPE points per one hour of study with a
95% confidence interval of 14 to 23 test points per one hour of study.

The Mango Languages written efficacy is the improvement per one hour of study. In addition, if
we divide the required cut-off point (270) for WebCAPE Second Semester placement by themean efficacy, we can construct a new measure representing the time needed to cover therequirements for the first college semester of Spanish. This is the one measure of efficacy that iseasy to understand and given the nature of the WebCAPE placement test, can be used forcomparison with other language apps.

Based on this measure, Mango Languages users will need on average about 15 hours of study
during a two-month period to cover the requirements for the first college semester of Spanish.
The transformed lower and upper limits are from 12 to 20 hours of study during a two-month
period.

31
32

The threshold of 270 points divided by the mean efficacy (18.3 points).
The threshold of 270 points divided by the lower limit (13.6) and the upper limit (23.0) of the 95% CI.

Mango Languages Efficacy Study 2019

Page 32

Oral Proficiency Efficacy
Table 19. Oral Efficacy of Mango Languages for Users with 8 Hours of Study (N=46)
Efficacy =Improvement per one
hour of study
TNT Test Points

Time to reach maximum
TNT=10.0

Mean (std)0.13 (0.14)

76.933

95% confidence interval

0.08 – 0.17

58.8 – 12534

Statistics

Hours

Regarding oral proficiency efficacy, it will take Mango Languages users on average 77 hours ofstudy to reach the upper limit of the TNT test. The estimate’s 95% confidence interval of thisestimate is from 59 to 125 hours.

33
34

TNT=10 divided by gain per hour (0.13)
TNT=10 divided by the upper (0.17) and lower (0.08) limits of the 95% confidence intervalMango Languages Efficacy Study 2019

Page 33

FACTORS FOR EFFICACY
Demographic Factors
We investigated the impact of several factors on language improvement (WebCAPE andTNT), namely age, gender, education, employment, device used, native language, knowledge ofanother foreign language, presence of people around the participant who spoke Spanish (spouse,friend, parents, grandparents), and reason for studying Spanish. None of these potential factors
had a statistically significant effect on efficacy (p<.05).
This means that the Mango Languages app works similarly well for people regardless ofgender, age, native language, education, employment status, etc.

Motivation EffectWe evaluated the effect of motivation on the oral and written language improvement. There wasno statistically significant effect of the motivation score. One possible explanation is that thepeople in this study are highly motivated with an average level of motivation of 82 out of 100.
At this high average level additional higher motivation does not have an effect of the results.
Language Profile EffectEnglish GLS does have a modest effect on written proficiency (Pearson r=0.38, p=.002). People
with a better English language profile tend to have better WebCAPE results. The non-English
language GLS does not have an effect on the written proficiency.
Neither English GLS nor non-English GLS had any effect on oral proficiency measured by
TNT.
Initial Written and Oral Proficiency
As expected, this study confirmed the inverse relationship between the initial level of language
proficiency and the gain in new knowledge (Figures 18 and 19). The biggest gain in new
knowledge can be attributed to the novice/beginner users, while more advance learners gain test
points slowly, both in written and oral proficiency. This is especially noticeable for oral
proficiency: the regression line for oral proficiency is steeper than the one for written
proficiency (regression coefficient -0.86 vs -0.64).

Mango Languages Efficacy Study 2019
Figure 18. Initial Written Proficiency and Improvement

Figure 19. Initial Oral Proficiency and Improvement

Page 34

Mango Languages Efficacy Study 2019

Page 35

USER SATISFACTION
After the study the participants were asked for their opinion about the Mango Languages
program, specifically how easy it was to use, how helpful it was for learning Spanish, howenjoyable it was, and how much they were satisfied with it. The 5-point Likert scale wasrecoded into two categories: Strongly Agree/Agree vs Strongly Disagree/ Disagree/Neutral.
Table 20. User Satisfaction (N=86)
Do you agree with the following statement?
“Mango Languages was easy to use”
“Mango Languages was helpful in studying Spanish”“I enjoyed learning Spanish with Mango Languages”“I am satisfied with Mango Languages”

Percent
Agree/Strongly Agree
95.3
88.4
90.7
83.7

After two months of study, the overwhelming majority of participants (84% and above)
agreed with the positive statements that: Mango Languages was easy to use, it was helpful, they
enjoyed learning with Mango Languages, and they were satisfied with it.
Almost all (98.9%) of the respondents in the exit survey declared that they would
continue to use Mango Languages after the end of the study.
In the exit survey, a special question was included: “How likely are you to recommend
Mango Languages to a colleague or friend?” with 11 possible answers, from 0 “Very unlikely”
to 10 “Very likely”. The answers to this question were used to compute the so-called Net
Promoter Score (NPS). This is “a management tool that can be used to gauge the loyalty of afirm's customer relationships” (Wikipedia). It was developed by Reichheld (2003) and it
categorizes users in three categories: “Promoters” (answers 9, 10), “Passives” (answers 7, 8),
and “Detractors” (answers 0-6). NPS is equal to the difference between “Promoters” and“Detractors” and in general it can vary from -100 (all detractors) to + 100 (all promoters). As arule, a positive NPS is good news for the company and the higher the score, the better theindicator for the company.
From our exit survey the “Promoters” were 58.1%, the “Detractors” were 16.3% and“Passives” were 25.6%. The Mango Languages NPS was +41.8.

Mango Languages Efficacy Study 2019

Page 36

LIMITATIONS OF THE STUDYThis is the 12th study35 of the Research Team testing the efficacy and motivation of
language learning apps (Vesselinov, Grego, et al., 2009-2019). The statistical design andmethodology are practically the same for all 12 studies. The only major deviation is themeasurement of study time in the first Rosetta Stone study in 2009. In this study, study time
was self-reported instead of automatically recorded by the server because participants were
given CDs with the software and no objective measure for study time was technologicallyavailable.
The population of adult people who are seeking to study foreign language with language
apps is highly educated with the majority of them having at least a college-level education. Thisis true not only for the U.S.36, but also Europe37 and the rest of the world38. This was confirmed
by all our previous studies39. This population has a higher education level than the general
population. Our current sample for the 2019 Mango Languages efficacy study is representativeof this particular population, but it may not be comparable to the general population.
This study measures the progress of novice/beginner users of Spanish. The study resultscannot be generalized for intermediate or advanced users of Spanish.
The independently developed tests used in this study were not tailored to any specificlearning tool, including Mango Languages. On the one hand, some participants in the study
complained that the test contained words or expressions that were not part of their regularcourse with Mango Languages. On the other hand, people insisted that they had learned a lot
more than the test asked for. The test is valuable as an independent tool for evaluation whichallows us to compare efficacy across different apps, however it does not provide a complete
measure of the full progress of users. So, the evaluation of their progress in language
proficiency is generally conservative.
The Research Team sent e-mail messages every week with individualized information
about the study time for the previous week. This seemed to stimulate the study process. The

35

One study was completed in April of 2019, but the Final Report has not been made public yet.
Rosetta Stone (2009, 2019), Duolingo (2012), italki (2018)37
Babbel (Germany & US), Busuu (UK and US).
38
New Language App, 2015 report, (world sample).
39
Except Hello English (2017) where the participants were of high school age.
36

Mango Languages Efficacy Study 2019

Page 37

results of the study should be valid in a setting where users study regularly for two months,
which may not be the case in typical settings, where users do not receive weekly reminders.
The study results could be generalized for studying Spanish with Mango Languages. For
other languages the results could be markedly different.
There are at least two limitations related to comparing the results of this study or thoseof the eleven previous studies, to a standard college semester of Spanish. First, progress or
success in college is determined usually with one midterm exam and one final exam, plus someform of testing for oral proficiency and homework. The progress is measured very differently ina college setting compared to our 12 studies so far. Second, study time in college is difficult tomeasure scientifically and it is not the same for everybody.
If there is a need to directly compare the progress of language app users and college
language learners, a new study can be designed to satisfy the efficacy definition. To the best of
our knowledge, such a study has not yet been done, and until then, the two sets of results cannot
be compared scientifically. Nevertheless, the WebCAPE is widely and successfully used as aplacement test by colleges and universities.
There are a limited number of studies with measures of efficacy available to comparewith the results of this study. More help is needed from users, investors, and analysts to require
the creators of language learning apps to provide independent efficacy measures. Hopefully inthe near future all language apps will be required to present independent efficacy studies to theircustomers.
Lastly, this study’s results may not be generalizable to other contexts: study periods
shorter than two-months, languages other than Spanish, or for intermediate or advanced learnersof Spanish.

Mango Languages Efficacy Study 2019

Page 38

CONCLUSION
The Mango Languages efficacy study is based on a random sample of 95 people, 18
years of age or older, residing in the U.S. All participants were self-reported novice users of
Spanish. We applied a set of instruments for written and oral proficiency.
This is one of the first research studies of the efficacy of language learning apps efficacy
that uses an incremental instrument to measure improvement in oral proficiency. The use of the
TNT gave us the opportunity to detect any change in oral proficiency in both directions:increase and decrease. In our previous studies, we used ACTFL OPIc, which is a level-based
test and cannot detect incremental change. Another valuable feature of TNT is that it gives
estimates of ACTFL and CEFR levels. The high prices of the commercially available tests withhuman raters is a big obstacle for research studies.
The main goal of measuring the efficacy of Mango Languages was achieved with thisstudy. Overall 82% of the participants improved their written proficiency during the study.
Novice users need on average 15 hours of study in a two-month period to cover therequirements for the first college semester of Spanish.
Overall 77% of all study participants increased their oral proficiency (TNT) during thestudy. The oral proficiency gain for participants with 8 hours of study or more was as follows:
85% of the participants increased their TNT score, 74% of the participants increased their
TNT-ACTFL estimation score, and 72% of the participants increased their TNT-CEFR
estimation score;
The written efficacy was an average gain of 18 WebCAPE points per one hour of study
and the oral proficiency was a gain of 0.13 TNT points per one hour of study.

Mango Languages Efficacy Study 2019
THIS PAGE LEFT INTENTIONALLY BLANKPage 39

Mango Languages Efficacy Study 2019

Page 40

CITED LITERATURE
Agresti, A., Coull, B., 1998, Approximation is better than “exact” for interval estimation of
binomial proportions, American Statistician, 52, pp. 119–126.
Birdsong, D., Gertken, L., & Amengual, M. Bilingual Language Profile: An Easy-to-Use
Instrument to Assess Bilingualism. COERLL, University of Texas at Austin.
Web. 20 Jan. 2012. https://sites.la.utexas.edu/bilingual/ .
Dörnyei, Z., 2005. The psychology of the language learners. Mahwah, NJ: Lawrence Erlbaum.
Dörnyei, Z., 2009. The L2 motivational self-system. In Z. Dörnyei, & E. Ushioda (Eds.),
Motivation, language identity and the L2 self (pp. 9e42). Bristol, UK: Multilingual
Matters.
Kong, J., Han, J., Kim, S., Park, H., Kim, Y., Park, Hy.
L2 Motivational Self System, international posture and competitiveness of Korean CTL
and LCTL college learners: A structural equation modeling approach,
System, Volume 72, February 2018, Pages 178-189
Reichheld, F., 2003, "One Number You Need to Grow", Harvard Business Review, 2003
December.
Vesselinov, R., Grego, J., Sacco, S., Tasseva-Kurktchieva, M., 2019,
The 2019 Rosetta Stone Efficacy Study .
http://comparelanguageapps.com/documentation/The2019_RS_FinalReport.pdf
Vesselinov, R. and Grego, J., 2018, italki Efficacy Study.
http://blog.italki.com/wp-content/uploads/2017/12/italki2018FinalReport.pdf or
http://comparelanguageapps.com/documentation/italki2018FinalReport.pdf
Vesselinov, R. and Grego, J., 2017, Hello English Efficacy Study.
http://centralsquarefoundation.org/grant/hello-english-efficacy-study/ , or
http://comparelanguageapps.com/documentation/HelloEnglish_2017Study.pdf
Vesselinov, R. and Grego, J., 2016b, The Babbel Efficacy Study.
http://comparelanguageapps.com/documentation/Babbel2016study.pdf , or
http://press.babbel.com/en/releases/2016-09-29-Spanish-Study.html
Vesselinov, R. and Grego, J., 2016, The Busuu Efficacy Study.
http://comparelanguageapps.com/documentation/The_busuu_Study2016.pdf , orhttps://blog.busuu.com/wp-content/uploads/2016/05/The_busuu_Study2016.pdf

Mango Languages Efficacy Study 2019

Page 41

Vesselinov, R. and Grego, J., 2015, Efficacy of New Language App,
http://comparelanguageapps.com/documentation/LA_Final_Report.pdf .
Vesselinov, R. and Grego, J., 2012, Duolingo Effectiveness Study.
http://comparelanguageapps.com/documentation/DuolingoReport_Final.pdf, or
http://static.duolingo.com/s3/DuolingoReport_Final.pdf
Vesselinov, R., Grego, J., Habing, B., Lutz, A., 2009a, Measuring the Attitude and Motivationof Rosetta Stone Users.
http://comparelanguageapps.com/documentation/MeasuringTheAttitudeandMotivationofRS
Users.pdf
Vesselinov, R., Grego, J., Habing, B., Lutz, A., 2009b, Comparative Analysis of Motivation ofDifferent Language Learning Software.
http://comparelanguageapps.com/documentation/ComparativeMotivationAnalysisofDifferentLanguageSoftware.pdf
Vesselinov, R., 2009, Measuring the Effectiveness of Rosetta Stone.
http://comparelanguageapps.com/documentation/MeasuringTheAttitudeandMotivationofRS
Users.pdf, or
http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.

Mango Languages Efficacy Study 2019

Page 42APPENDIX
Table A1. Study Participants’ Geographic Distribution: US States
Number of people

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

State

ST

Alabama
AlaskaArizona
Arkansas
California
Colorado
Connecticut
Delaware
FloridaGeorgia
Hawaii
Idaho
Illinois
Indiana
Iowa
KansasKentuckyLouisiana
Maine
Maryland
Massachusetts
MichiganMinnesota
Mississippi
MissouriMontana
Nebraska
NevadaNew Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio

AL
AK
AZ
AR
CA
CO
CT
DE
FL
GA
HI
ID
IL
IN
IA
KS
KY
LA
ME
MD
MA
MI
MN
MS
MO
MT
NE
NV
NH
NJ
NM
NY
NC
ND
OH

Initial
Pool
1
1
9
2
60
13
3
2
30
16
1

Eligible
Pool
1
1
9

InitialSample
1
1
3
1
14
5
3
1
12
4

Final
Sample
1
1
3
1
9
5
1
1
6
1

17
2
2
1
3
3

16
2
2
1
3
2

6
1
1
1
2

5
1

10
14
16
8

9
14
16
8

4
6
4
1

4
3
3
1

6

6

2

2

1
3
2
8
2
20
14
2
12

1
3
2
7
2
20
13
2
11

2
3

2
1

8
6
1
7

7
4

56
12
3
1
30
15
1

1
1

3

Mango Languages Efficacy Study 2019

Page 43

Table A1. Continued

36
37
38
39
40
41
42
43
44
46
45
47
49
48
50

Total

State

ST

Oklahoma
OregonPennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Virginia
Vermont
Washington
Wisconsin
West Virginia
Wyoming
District of Columbia
Unknown state (but US)
Outside US
All

OK
OR
PA
RI
SC
SD
TN
TX
UT
VA
VT
WA
WI
WV
WY
DC

Initial
Pool
8
6
12

Eligible
Pool
7
5
11

InitialSample1
3
4

Final
Sample

2

1

5
47
8
19
2
11
5

5
47
7
18
2
10
5

3
14
1
8
1
2
3

2
9
1
5
1

8
29
21
467

7
26

1
8

5

420

149

95

3
1

1

Mango Languages Efficacy Study 2019

Page 44

Table A2. Motivation Scale
Developed by Kong et al., 2018.
A. Ideal L2 self (4 items)
1. I can imagine myself living abroad and having a discussion in Spanish.
2. I can imagine myself speaking Spanish with international friends or colleagues.
3. I can imagine myself speaking Spanish as if I were a native speaker of Spanish.
4. Whenever I think of my future career/life, I imagine myself using Spanish.

B. Ought-to L2 self (7 items)
1. I study Spanish because close friends of mine think it is important.
2. Learning Spanish is necessary because people surrounding me expect me to do so.
3. I consider learning Spanish important because the people I respect think that I should do it.
4. Studying Spanish is important to me in order to gain the approval of mypeers/teachers/family/boss.
5. It will have a negative impact on my life if I don't’ learn Spanish.
6. Studying Spanish is important to me because an educated person is supposed to be able tospeak it.
7. Studying Spanish is important to me because other people will respect me more if I haveknowledge of it.

C. International posture (6 items)
1. I want to make friends with foreigners visiting U.S.
2. I would feel somewhat uncomfortable if a foreigner moved in next door. (reverse-coded)
3. I want to participate in a volunteer activity to help foreigners living in the surrounding
community.
4. I am interested in an international career/living abroad.
5. I often read and watch news about foreign countries.
6. I have thoughts that I want to share with people from other parts of the world.

Mango Languages Efficacy Study 2019

Page 45

D. Competitiveness (6 items)
1. I want to survive in the future.
2. I don't want to be an illiterate person.
3. I want to succeed in life.
4. Other people will consider me an elite if I have a good command of Spanish.
5. I don't want to place behind any of my friends.
6. I want to have a head start on other people.

E. L2 learning Experience or Attitudes (4 items)
1. I like the atmosphere of my Spanish classes with Mango Languages.
2. I find learning Spanish with Mango Languages really interesting.
3. I always look forward to Spanish classes with Mango Languages.
4. I really enjoy learning Spanish with Mango Languages.

F. Learners' Intended Effort or Motivated Behavior in L2 Learning (6 items)
1. If Spanish course were offered in the future, I would like to take it.
2. I expend a lot of efforts in learning Spanish.
3. I do my best to learn Spanish.
4. I spend lots of time studying Spanish.
5. I concentrate on studying Spanish more than any other topic.
6. Compared to other people I know, I think I study Spanish relatively hard.

Mango Languages Efficacy Study 2019

Page 46

Table A3. Language Profile
Developed by Birdsong et al., 2012.
I. Biographical Information
II. Language history
In this section, we would like you to answer some factual questions about your language history.
1. At what age did you start learning English?
Since birth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
2. At what age did you start to feel comfortable using English?
As early as I can remember 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+ not yet

3. How many years of classes (grammar, history, math, etc.) have you had in English(primary school through university)?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
4. How many years have you spent in a country/region where English is spoken?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
5. How many years have you spent in a family where English is spoken?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
6. How many years have you spent in a work environment where English is spoken?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+

III. Language use
In this section, we would like you to answer some questions about your language use.
7. In an average week, what percentage of the time do you use English with friends?
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

8. In an average week, what percentage of the time do you use English with family?
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

9. In an average week, what percentage of the time do you use English at school/work?
0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

10. When you talk to yourself, how often do you talk to yourself in English?
0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

70%

80%

90%

100%

11. When you count, how often do you count in English?
0% 10%

20%

30%

40%

50%

60%

Mango Languages Efficacy Study 2019

Page 47

IV. Language proficiency
In this section, we would like you to rate your language proficiency.
0=not well at all

6=very well

12. How well do you speak English?

0

1

2

3

4

5

6

13. How well do you understand English?

0

1

2

3

4

5

6

14. How well do you read English

0

1

2

3

4

5

6

15. How well do you write English?

0

1

2

3

4

5

6

V. Language attitudes
In this section, we would like you to respond to statements about language attitudes.
0=disagree

6=agree

16. I feel like myself when I speak English.

0

1

2

3

4

5

6

17. I identify with an English-speaking culture.

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0

1

2

3

4

5

6

18. It is important to me to use (or eventually use)
English like a native speaker.
19. I want others to think I am a native speakerof English.

THIS IS THE LAST PAGE

OF THE 2019 MANGO LANGUAGES EFFICACY STUDY**************************** No text bellow this line ****************************