Pimsleur Efficacy Study
FINAL REPORTRESEARCH TEAMROUMEN VESSELINOV1,2, PhD
Queens College, City University of New YorkJOHN GREGO2, PhD
University of South Carolina

April 2019

1
2

Corresponding author: roumen.vesselinov@qc.cuny.edu .
This report represents the individual opinion of the authors and not necessarily of their institutions.

Pimsleur Efficacy Study 2019

Page 1THIS PAGE LEFT INTENTIONALLY BLANK

Pimsleur Efficacy Study 2019

Page 2EXECUTIVE SUMMARY
The Research Team conducted an independent study of Pimsleur efficacy from February to
April 2019. It was based on a random representative sample of 82 novice users of Spanish.
The participants took one Spanish oral proficiency test in the beginning, then studied Spanish
with Pimsleur for two months and took the same test again. We measured the improvement inlanguage abilities as the difference between the final and the initial oral proficiency test results.

PIMSLEUR EFFICACY• The efficacy of Pimsleur program and method is 83%.
Eighty three percent of users who completed Pimsleur Level 1, i.e. completed 30 lessons,increased their oral proficiency level at least by one level and up to three levels.
The 95% confidence interval for the efficacy is between 63% and 94%.
• The efficacy of Pimsleur for users who did not follow the program requirements exactly
but have at least eight hours of study, is 73%.
Seventy three percent of users who had at least eight hours of study increased their oral
proficiency level at least by one level and up to three levels.
The 95% confidence interval for this efficacy is between 61% and 82%.

USER SATISFACTION• The majority of users thought that Pimsleur was easy to use (96%),
helpful (97%), enjoyable (97%), satisfying (93%).
• Pimsleur received a positive Net Promoter Score of +62.8 from the users.
• Pimsleur efficacy was not affected by gender, age, education, native language, etc.
• Participants’ motivation was very high and remained high after two-month study
with average level of 72% of the maximum motivation score.

Pimsleur Efficacy Study 2019

Page 3

THIS PAGE LEFT INTENTIONALLY BLANK

Pimsleur Efficacy Study 2019

Page 4

CONTENTSINTRODUCTION ........................................................................................................................ 6
RESEARCH DESIGN .................................................................................................................. 7
STUDY SAMPLE....................................................................................................................... 10
Sample Description ................................................................................................................. 11
Initial Motivation .................................................................................................................... 13
Language Profile ..................................................................................................................... 17
Final Language Test ................................................................................................................ 19
Final Motivation ...................................................................................................................... 19
Study Time .............................................................................................................................. 21
EFFICACY ................................................................................................................................. 23
Oral Proficiency Test Results ................................................................................................. 23
FACTORS FOR EFFICACY...................................................................................................... 24
Demographic Factors .............................................................................................................. 24
Motivation Effect .................................................................................................................... 25
Language Profile Effect .......................................................................................................... 25
USER SATISFACTION ............................................................................................................. 26
LIMITATIONS OF THE STUDY.............................................................................................. 27
CONCLUSION ........................................................................................................................... 29
CITED LITERATURE ............................................................................................................... 30
APPENDIX ................................................................................................................................. 32
Table A1. Study Participants’ Geographic Distribution: US States ....................................... 32
Table A2. Motivation Scale ................................................................................................... 34
Table A3. Language Profile ................................................................................................... 36

Pimsleur Efficacy Study 2019

Page 5THIS PAGE LEFT INTENTIONALLY BLANK

Pimsleur Efficacy Study 2019

Page 6INTRODUCTION
This is the 11th study of the Research Team testing the efficacy and motivation of
language learning apps (Vesselinov, Grego, et al., 2009-2019). Our previous studies evaluated
Rosetta Stone, Duolingo, busuu, Babbel, Hello English, italki and a new language app. The
statistical design and methodology are practically the same for all 11 studies. The only
deviation is the measurement of study time in the first Rosetta Stone study in 2009 when thetime was self-reported versus recorded by server because participants were given CDs with thesoftware and no objective measure for study time was technologically available.
This is a study designed to evaluate the efficacy of The Pimsleur® Language Program3.
The company describes the Pimsleur program and method as follows:“The Pimsleur Method™ was established nearly 50 years ago by Dr. Paul Pimsleur.
“There exists an urgent national need for self-instructional materials in many of the world's
languages,” wrote Dr. Pimsleur in 1963.
With that goal in mind he developed and recorded Modern Greek, the first of the Pimsleur
Language Courses. It was a test of his new theory of self-instruction which resulted from his
years of teaching and his study in linguistics and the psychology of language, specifically of
how memory could be triggered to best implant a new language.
Dr. Pimsleur's memory-training technique improves the language-learning centers inyour brain. The Pimsleur programs are unique because of something they don't have -- andbecause of something they do have -- exclusively! No Mindless Repetition.
Many language courses try to teach a language by "drumming it in" with endless
repetition of words and phrases, boring conjugations, irrelevant vocabulary, and mindless drills.
Instead, Dr. Pimsleur focused on delivering the information at the right time to train your brain
to respond easily. He called it: Graduated Interval Recall. Dr. Pimsleur's research into thepsychology of linguistics revealed that unstructured mindless repetition (rote-memorization)
has a dulling effect on the brain. It doesn't work. Through his studies at UCLA and Ohio State
University, Dr. Pimsleur discovered that if words and phrases are repeated at specific timeintervals, they are easily stored in long-term memory. This core principle is what makes thePimsleur courses unique and highly effective.
The Pimsleur Language Program is one of the leading audio-based language learning
programs, and is available in over 50 languages, including English as a second language.”This study was funded by Pimsleur, but the data collection and the analysis were carriedout independently by the Research Team. The language test used in the study was designed anddeveloped by an external independent testing company.

3

www.Pimsleur.com

Pimsleur Efficacy Study 2019

Page 7RESEARCH DESIGN
The random sample for this study was drawn from existing or new Pimsleur users
residing in the U.S. There were some additional requirements for the potential participants, who
had to:-Be willing to study Spanish using only Pimsleur for two months;-

Take two sets of oral proficiency language tests;-

Be at least 18 years of age;-

Be novice or beginner learners of Spanish.
The test used in the study was the Oral Proficiency Interview by Computer® (OPIc)4

created by Language Testing International (LTI). LTI is the exclusive licensee of the American
Council on the Teaching of Foreign languages (ACTFL). The online test is recorded andevaluated by independent certified raters.
Table 1. OPIc Ratings
UR
NL
NM
NH
IL
IM
IH

Un-Ratable
Novice Low
Novice Mid
Novice HighIntermediate Low
Intermediate Mid
Intermediate High

AL
AM
AH
S

Advanced Low
Advanced Mid
Advanced HighSuperior

The specific definitions of the levels are presented on the company’s webpage5.
Sample Size and Power Analysis
We based our power and sample calculations on the typical results from our previous
studies using OPIc. We designed the study to test the hypothesis for statistical significance ofproportions of 0.6, 0.7 or 0.8, corresponding to the expected proportion of participants that
improve their oral proficiency. We considered 0.05 statistical significance level (Alpha) and at
least 80% statistical power. Under these assumptions we would need a sample of 60 people ormore. Our initial sample size was set to 120 in order to reflect possible drop-outs from thestudy.

4
5

http://www.languagetesting.com/oral-proficiency-interview-by-computer-opic
https://www.actfl.org/publications/guidelines-and-manuals/actfl-proficiency-guidelines-2012

Pimsleur Efficacy Study 2019

Page 8

Spanish language was selected as one of the most popular languages and because of theexistence of previous research on Spanish for other language learning apps (Vesselinov &Grego, 2009-2019). The study lasted approximately 8 weeks and it was conducted between
February and April 2019. Participants who successfully completed the study were given one
additional level of Pimsleur Premium Spanish or one level of any language supported byPimsleur. No monetary or other incentives were offered to the participants.

Pimsleur Efficacy Study 2019

Page 9Figure 1. Sample Selection Tree

1187 people

Viewed invitation page433 people
Initial Pool754 people

Completed Entry
SurveyDid not complete
Entry Survey

398 people
Eligible Pool

35 people

Eligible

120 people
Initial Random Sample
Completed initial test

82 people
Final Sample
Completed two sets of tests

Ineligible

278 people
Not selected

38 people
Did Not Complete
Used additional learning tools
Did not complete the final test

Pimsleur Efficacy Study 2019

Page 10STUDY SAMPLEIn February 2019 emails were send to current and new Pimsleur potential users with aninvitation to participate in Spanish language study for two months. They were directed to an
online survey designed by the Research Team. This survey collected demographic information,and self-evaluation of their language proficiency level. We received complete responses from433 people, and this constituted the initial pool for the study. From this pool we excluded 35
people who lived outside US, were younger than 18 years of age, or had intermediate or highlevel of knowledge of Spanish. This was the eligible pool (N=398) for this study.
We randomly selected 120 people of the eligible pool of participants and they completed
the initial language test. They constituted our initial random sample (N=120).
Pimsleur study continued for approximately two months (8 weeks), starting in February
2019 and ending in April 2019. During the study the Research Team sent weekly e-mailreminders to the participants with information detailing the amount of time they had usedPimsleur each week. At the end of the study from the initial sample the following people were
excluded:•

People who did not take the final test.

•

People who regularly used additional learning tools during the study.

All participants were instructed at the beginning of the study that they could use only
Pimsleur to study Spanish for the duration of the study. In the exit survey four people stated thatthey had regularly used other language apps and they were excluded from the study. Other
people had occasionally used internet dictionaries, YouTube and translation websites and they
were allowed to stay in the study.
The final study sample consisted of 82 people who had used only Pimsleur during thestudy and had valid initial and final OPIc tests. After the completion of the tests it wasdetermined that two people had technical difficulties (cannot hear well the test questions) andtheir test results were not included in the analysis.
Final Study Sample versus Not Completed
From the initial random sample (N=120) 38 people (31.7%) did not complete the study
for two different reasons: participants who did not take the final test and participants who usedadditional learning tools during the study. The 31.7% dropout rate is about average in this lineof research.

Pimsleur Efficacy Study 2019

Page 11

We compared the two groups, the final sample of 82 people and the 38 people who did
not complete the study by gender, age, education, employment status, initial knowledge of
Spanish (initial OPIc) and reason for studying Spanish. There were no statistically significant
differences (at 5% error level), which means that participants who did not complete the study
were not very different from the ones that did.

Sample Description
In the final study sample 59.5% were female. The age varied from 19 to 80 years of age,with a mean age of 45.3 years. The pool was very well educated with majority participants
having some college, undergraduate or graduate degree.

Figure 2. Age Distribution

Table 2. Age and Gender DistributionAge
18-20 years old
21-30 years old
31-40 years old
Over 40 years old
Total

Female (N)
2
6
9
30
47

Male (N)
1
9
4
18
32

Total (N)
3
15
13
48
79*

Total (%)
3.7
19.5
15.9
61.0
100

* Some people declined to answer specific survey questions, so the number of answers can beless than 82 in some tables in this report. The number of answers is reported in the tables.

Pimsleur Efficacy Study 2019

Page 12

Table 3. Education
Category
2. High School diploma or equivalent
3. Some college but no degree
4. College graduate, BA or equivalent
5. Some graduate school but no degree
6. Master’s degree (MA, MS)
7. PhD/MD/JD
Total

N
3
14
23
12
22
8
82

Percent
3.7
17.1
28.0
14.6
26.8
9.8
100

The majority of the people were employed full time (67.9%).
Table 4. Employment Status
1.
2.
3.
4.
5.
6.
7.
Total

Category
Employed full time
Employed part time
Homemaker
Student
Retired
Unemployed
Other employment

N
53
5
3
4
7
1
5
78

Percent
67.9
6.4
3.8
5.1
9.0
1.3
6.4
100

Ninety-three percent of the participants were English native speakers and the rest
included the native speakers of the following languages: Finnish, French, Japanese, Russian,Thai, and Turkish.
All participants in the final sample described themselves as beginner users or neverstudied Spanish. About 8% of the respondents’ spouse, partner, or close friends spoke Spanish.
About 4% of their parents, grandparents, or great-grandparents spoke Spanish.
About 82% of the final sample had formally studied a foreign language before (mostly at highschool or college).
About 24% have lived outside U.S. in a non-English speaking country for more than 6months. About 17% of the respondents were raised in a multilingual or non-English speaking
household.
The primary reason for studying Spanish was personal interest (53.7%), followed bybusiness or work (25.6%), travel (19.5%), and other reason (1.2%).

Pimsleur Efficacy Study 2019

Page 13

Table 5. Reason for Studying Spanish
Category
1. Personal Interest
2. Business/Work
3. Travel
4. Other
Total

N
44
21
16
1
82

Percent
53.7
25.6
19.5
1.2
100

Initial Language Test
All participants took an initial oral proficiency test and the results are presented below.
Table 6. Initial Oral Proficiency Results (OPIc)
Level
1. Novice Low
2. Novice Mid
3. Novice High
Total

N
78
2
2
82

Percent
95.1
2.4
2.4
100

As expected, almost all participants (95.1%) were placed at the lowest Novice Low oral
proficiency level.

Initial Motivation
All participants completed a motivation scale in the beginning of the study to evaluate
the role of motivation on efficacy.
As in the previous Rosetta Stone study (Vesselinov & Grego, 2019) we adopted a
motivation scale approach largely based on the second language (L2) motivational self-system(Dörnyei, 2005, 2009) which stems largely from the concepts of possible selves and selfdiscrepancy theory. The model proposes that language learners are guided by visions of ‘second
language selves’, one which attracts them toward becoming an idealized L2 user (ideal L2 self)
and one which pushes them from societal obligation or a fear of failure (ought-to L2 self).
We adopted a specific 33 questions, 6 factors version of L2 Motivational Self System(see Appendix, Table A2) created by Kong et al. (2018).
Kong et al. (2018) offer the following descriptions of the motivation scale elements:

Pimsleur Efficacy Study 2019

Page 14

1. Ideal L2 self: “The ideal L2 self refers to a positive future image of the L2 self. For
example, learners who have developed a vivid ideal L2 self are likely to endeavor to learn an L2
by imagining themselves communicating fluently using the L2 in the future.”
2. Ought-to L2 self: “(This element) pushes people from societal obligation or a fear of
failure.”3. International posture: “It captures a tendency to relate oneself to the international
community rather than any specific L2 group. The key characteristics of international posture
are described as an interest in global issues or international affairs, a willingness to travel, stay,or work abroad, and a readiness to interact with foreigners or foreign cultures.”4. Competitiveness: “Competitiveness can be described as the desire to excel incomparison to others and contends that a learner constantly compares oneself with one's
idealized self-image or with other learners, feels pressured to out-do other students.”
5. L2 learning Experience or Attitudes: “L2 learning experience is related to thelearners' environment including teachers, peer groups, curriculum, and their attitudes toward L2
learning.”
6. Learners' Intended Effort or Motivated Behavior in L2 Learning: This motivation
element evaluates how much effort are users determined to make and how hard they are readyto study.
As we can see from the table below and Figures 3 to 9 below, the participants vary a lot
on initial motivation. The scale dimensions were re-coded, so the maximum motivation is equal
to 100%.
Table 7. Initial Motivation Levels (%)
Motivation Dimensions
1. Ideal Self
2. Ought-to-Self
3. International Posture
4. Competitiveness5. Learning Attitude
6. Intended Effort
Total Motivation*
N=74
6First 25% of the sample.
50% middle point.
8
First 75% of the sample.
7

1st Quartile6
70.0
42.9
75.8
72.5
80.0
70.0
69.6

Max=100
Median7
80.0
55.7
80.0
80.0
85.0
80.0
74.4

3rd Quartile8
90.0
65.7
90.0
85.0
95.0
86.7
84.5

Pimsleur Efficacy Study 2019

Page 15

The initial average level of total motivation was very high (Me=74.4%). From themotivation elements the highest level (85%) belongs to “Learning Attitude” which indicatesthat the participants were extremely eager to learn a new language. The element “Ought-toSelf” has the lowest level of all (55.7%) which suggests that the participants were not very
afraid of failure or they were not that susceptible to pressure from societal obligation.
Figure 3. Initial Total Motivation Level (%)

Max=100

As noted above the average initial level of motivation was very high and mostpeople were overall highly motivated. Only a handful of people had motivation level less than60%.

Pimsleur Efficacy Study 2019

Page 16

INITIAL MOTIVATION LEVELS:
Figure 4. L2 Motivation (Max=100)
M1 “Ideal Self”

Figure 5. L2 Motivation (Max=100)
M2. “Ought-to Self”

Figure 6. L2 Motivation (Max=100)
M3. “International Posture”

Figure 7. L2 Motivation (Max=100)
M4. “Competitiveness”

Figure 8. L2 Motivation (Max=100)
M5. “Learning Attitude”

Figure 9. L2 Motivation (Max=100)
M6. “Intended Effort”

Pimsleur Efficacy Study 2019

Page 17

Language Profile
We asked participants to complete an adapted version of Bilingual Language Profile(Birdsong et al., 2012). We used the Global Language Score (GLS) for English and GLS for asecond language for participants with a second language in use. GLS is based on separate
modules for evaluating language history, language use, language proficiency and language
attitudes. GLS can vary from 0 to 218, and we re-coded it, so the maximum is equal to 100.
Figure 10. GLS score for English Language (N=74)

Max=100

The median GLS percent was 97.7 (IQR9=5.7) which corresponds to initial sample ofvery strong English native speakers. Forty-six participants or 56% of the sample, felt
comfortable enough to complete GLS for their second non-English language.
For example, a GLS score of 218 (or rescaled as 100) would be appropriate forparticipants born in English speaking family, in an English-speaking country, who started
studying English immediately, for whom all classes at school were in English, who speak only
English all the time with family, friends, and at work. Their language history and language useare entirely English-based. They feel totally proficient in English, and they identify themselveswith an English-speaking culture.

9Interquartile Range = 3rd Quartile – 1st Quartile

Pimsleur Efficacy Study 2019

Page 18

GLS score will lose some points if the participants started learning English at older age;or some of the school teaching is in another language; or they use another language to speakwith some of their friends and family, etc.
Figure 11. GLS score for Language other than English (N=46) Max=100

The median GLS percent for language other than English was 33.4 (IQR=32.2).

Pimsleur Efficacy Study 2019

Page 19

Final Language Test
The participants in the study completed the same language test at the end of the study.
Table 8. Final Oral Proficiency Results (OPIc)
Level
1. Novice Low
2. Novice Mid
3. Novice High
4. Intermediate Low
5. Intermediate Mid
Total

N
33
38
6
2
1
80*

Percent
41.3
47.5
7.5
2.5
1.3
100

* After the study completion it was determined that two people had technical problems during
the final test and their results were excluded from the analysis.

Final Motivation
Participants completed the same motivation scale (Kong et al., 2018) at the end of the study.
Table 9. Final Motivation Levels (Percent)Motivation Dimensions
1. Ideal Self
2. Ought-to-Self
3. International Posture
4. Competitiveness5. Learning Attitude
6. Intended Effort
Total Motivation**N=78

1st Quartile
65.0
40.0
70.0
70.0
78.8
66.7
68.2

Max=100
Median
77.5
48.6
76.7
76.7
80.0
73.3
72.4

3rd Quartile
85.0
62.9
86.7
83.3
90.0
80.8
78.0

The total motivation level after two-months of studying remains remarkably high at
72.4%, although it is slightly below the initial level of motivation of 74.4%. The champion is“Learning Attitude” (80%) with “Ideal Self” as close second with 77.5% motivation level. Still,at the end of the study the participants were not very afraid of failure or societal obligations
with “Ought-to-Self” 48.6% level of motivation.

Pimsleur Efficacy Study 2019

Figure 12. Final Total Motivation Level (%), N=78

Page 20

Max=100

Figure 13. Initial and Final Motivation

The initial and final motivations are strongly correlated as the graph above shows. It
includes the linear regression line and 95% confidence interval.

Pimsleur Efficacy Study 2019

Page 21

Study TimeWe measured the study time objectively by the actual server time on a weekly basis and
we reported the time to the participants regularly via e-mail in order to encourage them to keep
studying. Several of the participants studied mostly offline by downloading the lessons and theirtime was estimated based on the number of lessons covered. The median study time for the final
study sample (N=82) was about 16 hours, or about 2 hours of study a week. The total study
time varied from about two hours to 140 hours during the two-month study.
We established a cut-off point of 8 hours to be consistent with the requirements for the
OPIc test-retest (2-3 months of study) and comply with the established threshold from ourprevious studies (italki 2018 and busuu 2016).
In addition, the Pimsleur method requires half an hour a day study and Level 1 consistsof 30 lessons. Overall 63 people (76.8%) covered the requirement of 8 hours of study and 24
people covered the Pimsleur requirement for 30 lessons or completed Level 1 of Premium
Spanish.

Figure 14. Study Time Distribution(N=82)Hours

Pimsleur Efficacy Study 2019
Figure 15. Study Time per One Lesson

Page 22(N=82)Hours

The participants spent about 45 min (Me=45 min) per one lesson. The values varied
from about 10 min to 4.5 hours per lesson, which included going several times over one lesson.

Pimsleur Efficacy Study 2019

Page 23

EFFICACY
Oral Proficiency Test Results
All participants took initial OPIc test before the start of the study and then again at theend of the study. We measured the progress or improvement as the difference between the final
test level and the initial level.
Table 10. OPIc Oral Proficiency Level Placement
Level
Novice Low
Novice Mid
Novice HighIntermediate Low
Intermediate Mid
Total

Initial Test%
People (N)
78
95.1
2
2.4
2
2.4

82

Final Test
People (N)
%
33
38
6
2
1
80

100

41.3
47.5
7.5
2.5
1.3
100

Participants at the lowest level decreased from 95.3% to 41.3%.
Table 11. Oral Proficiency Improvement with Study Time of 8 hours or more
Improved#

Level Change

0
+1
+2
+3

Same/No change
One Level Up
Two levels Up
Three Levels Up
Total

People (N)

%

17
40
4
2
63

27.0
63.5
6.3
3.2
100

Study Time
Mean
(Hours)
26.5
29.3
36.6
29.2
29.0

Overall, 73% of the users with 8 hours of study or more improved their oral proficiency
level at least by one level and up to 3 levels. The 95% confidence interval is between 60.8% and82.4%10. People who did not improve their level studied on average less than people who
improved (26.5 vs 30 hours) but this difference was not statistically significant.

10

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Pimsleur Efficacy Study 2019

Page 24

Table 12. Oral Proficiency Improvement with 30 Pimsleur Lessons or more#

Level Change

0
+1
+2
+3

Same/No change
One Level Up
Two levels Up
Three Levels Up
Total

Improved
People (N)
%
4
16.7
15
62.5
3
12.5
2
8.3
24
100

Study Time
Mean (Hours)
37.3
39.2
45.4
29.2
38.8

Overall, 83.3% of the users with 30 Pimsleur lessons completed or more improved theiroral proficiency level at least by one level and up to 3 levels. The 95% confidence interval isbetween 63.4% and 93.8%11. People who did not improve their level studied on average lessthan people who improved (37.3 vs 39 hours) but this difference was not statisticallysignificant.

FACTORS FOR EFFICACY
Demographic Factors
We investigated the impact of main factors on language improvement (Yes/No), namely
age, gender, education, employment, device used, native language, knowing another foreign
language, presence of people around the participant who spoke Spanish (spouse, friend, parents,
grandparents), and reason for studying Spanish.
None of these potential factors had a statistically significant effect on the efficacy(p=.05). In some instances, the number of cases by subgroups was too low to expect enoughstatistical power for the test of hypotheses.
This means that the Pimsleur app works similarly well for people with different gender,
age, native language, education, employment status, etc.

11

95% CI with Agresti-Coull correction (Agresti & Coull, 1998).

Pimsleur Efficacy Study 2019

Page 25

Motivation EffectInitial motivation level.
Table 13. Initial Motivation Levels (%) and Language Improvement
Motivation Dimensions
1. Ideal Self
2. Ought-to-Self
3. International Posture
4. Competitiveness5. Learning Attitude
6. Intended Effort
Total Motivation*
N=57

No Improvement
Mean
74.7
50.7
75.8
76.5
85.9
77.1
73.4

Improved
Mean
80.9
57.5
83.4
80.9
85.4
80.7
78.3

Max=100
ANOVA
p-value
.12
.19
.023
.15
.85
.31
.11

Overall people who successfully increased their oral proficiency by one level or more
were more motivated than those who did not improve. But these differences were notstatistically significant except for International Posture motivation (p=.023).

Language Profile Effect
Table 14. Global Language Score (GLS): English and Non-English (%)
And Language Improvement

Max=100

GLS

Improved
Mean
92.7
33.9

English (n=57)
Non-English (n=38)

No Improvement
Mean
93.8
44.1

ANOVA
p-value.75.20

The language profile (English and Non-English) did not have statistically significant
effect on language improvement.

Pimsleur Efficacy Study 2019

Page 26

USER SATISFACTION
After the study the participants were asked for their opinion about Pimsleur, specifically
how easy it was to use, how helpful, enjoyable, and satisfactory. The 5-point Likert scale wasrecoded into two categories: Strongly Agree/Agree vs Strongly Disagree/ Disagree/Neutral.
Table 15. User Satisfaction (N=78)
Do you agree with the following statement?
“Pimsleur was easy to use”
“Pimsleur was helpful in studying Spanish”“I enjoyed learning Spanish with Pimsleur”“I am satisfied with Pimsleur”

Percent
Strongly Disagree/ Agree/
Disagree/Neutral
Strongly Agree
3.8
96.2
2.6
97.4
2.6
97.4
6.4
93.6

After two months of study, the overwhelming majority of users (93% and above) agreed
with the positive statements that: Pimsleur was easy to use, helpful, they enjoyed learning with
Pimsleur and were satisfied with it.
All respondents in the exit survey declared that they will continue to use Pimsleur after
the study ends.
In the exit survey a special question was included: “How likely are you to recommend
Pimsleur to a colleague or friend?” with 11 possible answers, from 0 “Very unlikely” to 10“Very likely”. The answers to this question were used to compute the so-called Net Promoter
Score (NPS). This is “a management tool that can be used to gauge the loyalty of a firm's
customer relationships” (Wikipedia). It was developed by Reichheld (2003) and it categorizes
users in three categories: “Promoters” (answers 9, 10), “Passives” (answers 7, 8), and“Detractors” (answers 0-6). NPS is equal to the difference between “Promoters” and“Detractors” and in general it can vary from -100 (all detractors) to + 100 (all promoters). As arule, positive NPS is good news for the company and the higher the score the better indicator
for the company.
From our exit survey the “Promoters” were 67.9%, the “Detractors” were 5.1% and“Passives” were 26.9%. The Pimsleur NPS was +62.8.

Pimsleur Efficacy Study 2019

Page 27

LIMITATIONS OF THE STUDY
The population of adult people who are seeking to study foreign language with language
app is highly educated with majority of them having college level education and above. This istrue not only for the U.S.12, but also Europe13 and the rest of the world14. This was confirmed by
all our previous studies15. This population has higher education level than the general
population. Our current sample for the 2019 Pimsleur efficacy study is representative of thispopulation and it should not be compared to the general population.
This study measures the progress of novice/beginner users of Spanish. The study resultscannot be generalized for intermediate or advanced users of Spanish.
The OPIc test used in this study is not tailored to any specific learning tool, including
Pimsleur. On the one hand, some participants in the study complained that the test contained
words or expressions that were not part of their regular course with Pimsleur. On the other
hand, people insisted that they had learned a lot more than the test asked for. The test isvaluable as an independent tool for evaluation which allows us to compare efficacy acrossdifferent apps, however it does not provide a complete measure of the full progress of users.
So, their progress evaluation of language proficiency is generally conservative.
The Research Team sent e-mail messages every week with individualized information
about the study time for the previous week. This seemed to stimulate the study process. In
normal settings when people work individually on their studies, this stimulation is not available.
The results of the study should be valid in a setting where users study regularly for two months.
The study results could be generalized for studying Spanish with Pimsleur. For otherlanguages the results could be markedly different.
The results of this study or the ten previous studies, cannot be compared to a standard
college semester of Spanish for at least two reasons. First, progress or success in college isdetermined usually with one midterm exam and one final exam, plus some form of testing fororal proficiency and homework. The progress is measured very differently in a college setting
compared to our 11 studies so far. Second, the study time at college is difficult to measurescientifically and it is not the same for everybody.
12

Rosetta Stone (2009, 2019), Duolingo (2012), italki (2018)
Babbel (Germany & US), Busuu (UK and US).
14
New Language App, 2015 report, (world sample).
15
Except Hello English (2017) where the participants were of high school age.
13

Pimsleur Efficacy Study 2019

Page 28

If there is a need to compare the progress of language app users and college language
learners, a new study can be designed to satisfy the efficacy definition. To the best of ourknowledge such study has not been done yet and until then the two sets of results cannot becompared scientifically.
There are limited number of studies with measure of efficacy available to compare withthe results of this study. More help is needed from users, investors, and analysts to require thecreators of language learning apps to provide independent efficacy measures. Hopefully in near
future all language apps will be required to present independent efficacy studies to theircustomers.

Pimsleur Efficacy Study 2019

Page 29

CONCLUSION
The Pimsleur efficacy study is based on a random sample of 82 people, 18 years of age
or older, residing in the U.S. All participants were self-reported novice/beginner users of
Spanish.
The main goal of measuring the efficacy of Pimsleur was achieved with this study. The
results show that, on average, 83% of the users who follow the Pimsleur method of half an hour
per lesson and complete 30 lessons (Level 1) improve their oral proficiency level at least by one
and up to 3 levels. The 95% confidence interval for this efficacy measure is between 63% and94%.
In more general sense, comparable to our previous studies, 73% of the Pimsleur users
who did not follow the Pimsleur method exactly but studied for at least 8 hours for a two-month
period, improved their oral proficiency level by at least one level and up to 3 levels.
These results are not generally valid in different context: shorter than two-month period
or for languages other than Spanish, and for intermediate or advanced learners of Spanish.

Pimsleur Efficacy Study 2019

Page 30

CITED LITERATURE
Agresti, A., Coull, B., 1998, Approximation is better than “exact” for interval estimation of
binomial proportions, American Statistician, 52, pp. 119–126.
Birdsong, D., Gertken, L., & Amengual, M. Bilingual Language Profile: An Easy-to-Use
Instrument to Assess Bilingualism. COERLL, University of Texas at Austin.
Web. 20 Jan. 2012. https://sites.la.utexas.edu/bilingual/ .
Dörnyei, Z. 2005. The psychology of the language learners. Mahwah, NJ: Lawrence Erlbaum.
Dörnyei, Z., 2009. The L2 motivational self system. In Z. Dörnyei, & E. Ushioda (Eds.),
Motivation, language identity and the L2 self (pp. 9e42). Bristol, UK: Multilingual
Matters.
Kong, J., Han, J., Kim, S., Park, H., Kim, Y., Park, Hy.
L2 Motivational Self System, international posture and competitiveness of Korean CTL
and LCTL college learners: A structural equation modeling approach,
System, Volume 72, February 2018, Pages 178-189
Reichheld, F., 2003, "One Number You Need to Grow", Harvard Business Review, 2003
December.
Vesselinov, R., Grego, J., Sacco, S., Tasseva-Kurktchieva, M., 2019,
The 2019 Rosetta Stone Efficacy Study .
http://comparelanguageapps.com/documentation/The2019_RS_FinalReport.pdf
Vesselinov, R. and Grego, J., 2018, italki Efficacy Study.
http://blog.italki.com/wp-content/uploads/2017/12/italki2018FinalReport.pdf or
http://comparelanguageapps.com/documentation/italki2018FinalReport.pdf
Vesselinov, R. and Grego, J., 2017, Hello English Efficacy Study.
http://centralsquarefoundation.org/grant/hello-english-efficacy-study/ , or
http://comparelanguageapps.com/documentation/HelloEnglish_2017Study.pdf
Vesselinov, R. and Grego, J., 2016b, The Babbel Efficacy Study.
http://comparelanguageapps.com/documentation/Babbel2016study.pdf , or
http://press.babbel.com/en/releases/2016-09-29-Spanish-Study.html
Vesselinov, R. and Grego, J., 2016, The Busuu Efficacy Study.
http://comparelanguageapps.com/documentation/The_busuu_Study2016.pdf , orhttps://blog.busuu.com/wp-content/uploads/2016/05/The_busuu_Study2016.pdf

Pimsleur Efficacy Study 2019

Page 31

Vesselinov, R. and Grego, J., 2015, Efficacy of New Language App,
http://comparelanguageapps.com/documentation/LA_Final_Report.pdf .
Vesselinov, R. and Grego, J., 2012, Duolingo Effectiveness Study.
http://comparelanguageapps.com/documentation/DuolingoReport_Final.pdf, or
http://static.duolingo.com/s3/DuolingoReport_Final.pdf
Vesselinov, R., Grego, J., Habing, B., Lutz, A., 2009a, Measuring the Attitude and Motivationof Rosetta Stone Users.
http://comparelanguageapps.com/documentation/MeasuringTheAttitudeandMotivationofRS
Users.pdf
Vesselinov, R., Grego, J., Habing, B., Lutz, A., 2009b, Comparative Analysis of Motivation ofDifferent Language Learning Software.
http://comparelanguageapps.com/documentation/ComparativeMotivationAnalysisofDifferentLanguageSoftware.pdf
Vesselinov, R., 2009, Measuring the Effectiveness of Rosetta Stone.
http://comparelanguageapps.com/documentation/MeasuringTheAttitudeandMotivationofRS
Users.pdf, or
http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.

Pimsleur Efficacy Study 2019

Page 32

APPENDIX
Table A1. Study Participants’ Geographic Distribution: US States
Number of people

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

State

ST

Alabama
AlaskaArizona
Arkansas
California
Colorado
Connecticut
Delaware
FloridaGeorgia
Hawaii
Idaho
Illinois
Indiana
Iowa
KansasKentuckyLouisiana
Maine
Maryland
Massachusetts
MichiganMinnesota
Mississippi
MissouriMontana
Nebraska
NevadaNew Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio

AL
AK
AZ
AR
CA
CO
CT
DE
FL
GA
HI
ID
IL
IN
IA
KS
KY
LA
ME
MD
MA
MI
MN
MS
MO
MT
NE
NV
NH
NJ
NM
NY
NC
ND
OH

Initial
Pool
3
1
8
1
57
4
4
2
27
18
6
1
9
6

Eligible
Pool
3
1
7
1
56
4
4
1
26
16
6
1
9
6

InitialSample
1

2
6
2
1
17
15
16
4
2
6

Final
Sample
1

1
17

16

1

1

7
4
3
1
1

6
3
3

2
6
2
1
16
15
16
3
2
6

1
3
1
1
5
6
3
2

1
2

2

1

1
3
1
11
1
26
18

1
3
1
10
1
26
17

1

5
1
11
6

4
1
9
6

5

5

1

1

1
3
2
3
1

Pimsleur Efficacy Study 2019

Page 33

Table A1. Continued

36
37
38
39
40
41
42
43
44
46
45
47
49
48
50

Total

State

ST

Oklahoma
OregonPennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Virginia
Vermont
Washington
Wisconsin
West Virginia
Wyoming
District of Columbia
Unknown state (but US)
Outside US
All

OK
OR
PA
RI
SC
SD
TN
TX
UT
VA
VT
WA
WI
WV
WY
DC

Initial
Pool
3
2
3
2
3

Eligible
Pool
3
2
3
2
1

InitialSample
1
2
1
1

Final
Sample

10
26
4
17
2
20
5
2

10
26
4
17
2
20
5
2

4
3

3
2

7
1
5
2

6

2
25
23
433

2
25

8

2

398

120

82

1
1

1
1

Pimsleur Efficacy Study 2019

Page 34

Table A2. Motivation Scale
Developed by Kong et al., 2018.
A. Ideal L2 self (4 items)
1. I can imagine myself living abroad and having a discussion in Spanish.
2. I can imagine myself speaking Spanish with international friends or colleagues.
3. I can imagine myself speaking Spanish as if I were a native speaker of Spanish.
4. Whenever I think of my future career/life, I imagine myself using Spanish.

B. Ought-to L2 self (7 items)
1. I study Spanish because close friends of mine think it is important.
2. Learning Spanish is necessary because people surrounding me expect me to do so.
3. I consider learning Spanish important because the people I respect think that I should do it.
4. Studying Spanish is important to me in order to gain the approval of mypeers/teachers/family/boss.
5. It will have a negative impact on my life if I don't’ learn Spanish.
6. Studying Spanish is important to me because an educated person is supposed to be able tospeak it.
7. Studying Spanish is important to me because other people will respect me more if I haveknowledge of it.

C. International posture (6 items)
1. I want to make friends with foreigners visiting U.S.
2. I would feel somewhat uncomfortable if a foreigner moved in next door. (reverse-coded)
3. I want to participate in a volunteer activity to help foreigners living in the surrounding
community.
4. I am interested in an international career/living abroad.
5. I often read and watch news about foreign countries.
6. I have thoughts that I want to share with people from other parts of the world.

Pimsleur Efficacy Study 2019

Page 35

D. Competitiveness (6 items)
1. I want to survive in the future.
2. I don't want to be an illiterate person.
3. I want to succeed in life.
4. Other people will consider me an elite if I have a good command of Spanish.
5. I don't want to place behind any of my friends.
6. I want to have a head start on other people.

E. L2 learning Experience or Attitudes (4 items)
1. I like the atmosphere of my Spanish classes with Pimsleur.
2. I find learning Spanish with Pimsleur really interesting.
3. I always look forward to Spanish classes with Pimsleur.
4. I really enjoy learning Spanish with Pimsleur.

F. Learners' Intended Effort or Motivated Behavior in L2 Learning (6 items)
1. If Spanish course were offered in the future, I would like to take it.
2. I expend a lot of efforts in learning Spanish.
3. I do my best to learn Spanish.
4. I spend lots of time studying Spanish.
5. I concentrate on studying Spanish more than any other topic.
6. Compared to other people I know, I think I study Spanish relatively hard.

Pimsleur Efficacy Study 2019

Page 36

Table A3. Language Profile
Developed by Birdsong et al., 2012.
I. Biographical Information
II. Language history
In this section, we would like you to answer some factual questions about your language history.
1. At what age did you start learning English?
Since birth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
2. At what age did you start to feel comfortable using English?
As early as I can remember 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+ not yet

3. How many years of classes (grammar, history, math, etc.) have you had in English(primary school through university)?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
4. How many years have you spent in a country/region where English is spoken?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
5. How many years have you spent in a family where English is spoken?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+
6. How many years have you spent in a work environment where English is spoken?
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20+

III. Language use
In this section, we would like you to answer some questions about your language use.
7. In an average week, what percentage of the time do you use English with friends?
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

8. In an average week, what percentage of the time do you use English with family?
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

9. In an average week, what percentage of the time do you use English at school/work?
0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

10. When you talk to yourself, how often do you talk to yourself in English?
0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

70%

80%

90%

100%

11. When you count, how often do you count in English?
0% 10%

20%

30%

40%

50%

60%

Pimsleur Efficacy Study 2019

Page 37

IV. Language proficiency
In this section, we would like you to rate your language proficiency.
0=not well at all

6=very well

12. How well do you speak English?

0

1

2

3

4

5

6

13. How well do you understand English?

0

1

2

3

4

5

6

14. How well do you read English

0

1

2

3

4

5

6

15. How well do you write English?

0

1

2

3

4

5

6

V. Language attitudes
In this section, we would like you to respond to statements about language attitudes.
0=disagree

6=agree

16. I feel like myself when I speak English.

0

1

2

3

4

5

6

17. I identify with an English-speaking culture.

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0

1

2

3

4

5

6

18. It is important to me to use (or eventually use)
English like a native speaker.
19. I want others to think I am a native speakerof English.

THIS IS THE LAST PAGE

OF THE 2019 PIMSLEUR EFFICACY STUDY**************************** No text bellow this line ****************************