Open Research Online
Citation
Rosell-Aguilar, Fernando (2018). Autonomous language learning through a mobile
application: a user evaluation of the busuu app. Computer Assisted Language Learning,
31(8) pp. 854–881.
URL
https://oro.open.ac.uk/54354/
License(CC-BY-NC-ND 4.0) Creative Commons: Attribution-Noncommercial-No Derivative Works
4.0
https://creativecommons.org/licenses/by-nc-nd/4.0/
PolicyThis document has been downloaded from Open Research Online, The Open University's
repository of research publications. This version is being made available in accordancewith Open Research Online policies available from Open Research Online (ORO) Policies
Versions
If this document is identified as the Author Accepted Manuscript it is the version after peer
review but before type setting, copy editing or publisher branding

Autonomous language learning through a mobile application: a user evaluation of the busuu
app

Fernando Rosell-Aguilar
School of Languages and Applied Linguistics
Faculty of Wellbeing, Education and Language Studies
The Open University
Walton Hall
Milton Keynes
MK7 6 AA
United Kingdom

Fernando.rosell-aguilar@open.ac.uk
ORCID: 0000-0001-9057-0565
Twitter: @frosellaguilar
Linkedin: https://www.linkedin.com/in/fernando-rosell-aguilar-62579668/

1

Abstract

Studies into the use of mobile applications (apps) for language learning have found positive results
on language improvement and learner engagement, but they have mostly focused on small samplesof learners using apps that were selected by a researcher / educator rather than on learners using
apps of their own choice. This article presents the results of a large-scale survey (n=4095) into theuse of one of the most popular language learning apps in the market: the busuu mobile app (over 60
million registered users). Data were collected through an online questionnaire with 30 items. The
survey was distributed both in English and in Spanish. The results provide a profile of busuu app
users, show patterns of use, and what app features learners find most valuable for language
learning. Most users are at beginner level and learn for personal interest. They find the app has
helped them improve their knowledge of the language they are learning, with vocabulary as themain area of improvement. The high expectations from users and the fact that a third of respondents
use busuu as their only language learning source suggest that a large proportion of users consider
apps a reliable tool for language learning.

Introduction: apps for language learning

In the decade or so since smartphones were introduced, these and subsequent devices such as tablets
have become ubiquitous: they are now used by large proportions of the population in mostdeveloped countries. Mobile applications (commonly referred to as apps) can be installed on thesedevices, and offer an enormous market of educational resources, including language learning. Early
claims about the potential of apps for language learning included the possibility to learn, practise,and enhance both language skills and knowledge of the areas where target languages are spoken
2

(Rosell-Aguilar, 2009). Several researchers have identified benefits of apps for language learning(Burston, 2014; Godwin-Jones, 2011; Kim & Kwon, 2012; Lafford, 2011) based on their potential
to engage learners in activities identified in the fields of Second Language Acquisition Theory(SLA) and Computer-Assisted Language Learning (CALL) as conducive to language learning suchas noticing, negotiation of meaning, learning by doing, focus on form and collaborative learning(Skehan, 2003; Doughty & Long, 2003). Apps can provide opportunities to engage in interactive,
meaningful and engaging tasks, promote collaborative, rewarding, and challenging tasks andprovide opportunities to produce target language, as advocated in CALL theory by Oxford (1990),
Meskill (1999), Skehan (2003) and Chapelle (1998). Due to the use of mobile devices to operateapp, the study of language learning apps falls within the field of Mobile-Assisted Language
Learning (MALL).

As with every language learning resource, the quality and potential for language learning of apps
varies enormously. Some researchers have highlighted drawbacks about language learning apps,
questioning the match between pedagogical and technical qualities and claiming that they canprovide very fragmented language practice (Pareja-Lora et al., 2013). Others have claimed thatlearning activities on mobile apps are basic and mostly replicate what has been done before withother technologies (Burston, 2014), and noted that most apps focus on cognitive processes andreceptive language skills and lack the opportunities to engage in socio-cognitive activities oropportunities for collaborative learning (Kim and Kwon, 2012).

Whilst it is arguable whether apps can at this point be considered as a single solution to language
learning, they can effectively support learner autonomy and interest in learning a language(Goodwin-Jones, 2011). Apps can provide a good supplement for language learners who areenrolled in formal instruction as well as a good starting point for beginner independent learners.

3

They can also provide regular practice for language learners who are no longer formally studying alanguage but wish to keep practising it (Rosell-Aguilar, 2017).

Literature review

Previous research on the use of apps for language learning.

Since the appearance of apps that can be used for language learning, many researchers haveinvestigated the student experience of learning with them. A common finding among these studies
has been a very positive attitude towards learning with apps (Brown, Castellano, Hughes & Worth,
2012; Castañeda & Cho, 2016; Khaddage & Lattemann, 2013; Kim, 2013), particularly highlighting
flexibility, convenience (Zou & Li, 2015), portability, and the ability to personalise student learning(Steel, 2012).

Research has also found improvements in several language skills such as vocabulary acquisition(Steel, 2012; Yildiz, 2012), reading and writing (Steel, 2012; Morgana, 2015), translation activities(Steel, 2012); listening practice, writing and, to a lesser extent, speaking (Morgana, 2015);
phonological awareness and listening comprehension skills (Yildiz, 2012; Kim, 2013); and verb
conjugation and grammar (Castañeda & Cho, 2016). Apps were also found valuable by learners ofcharacter-based languages: Rosell-Aguilar & Kan (2015) reported that the majority of participantsin their study of 137 learners using an app to learn Chinese characters felt that they had improved
their recognition of characters and words as well as their ability to write the characters. The
effectiveness of apps has also been found comparable to face-to-face teaching at elementary level:
in a comparative study with 164 children learning Spanish in primary education using Duolingo,
Rachels and Rockinson-Szapkiw (2017) found that there were no significant differences between
the group that used the app and the control group that received tuition without using the app.
4

Autonomous use of language learning apps may be quite different to how those apps are usedwithin a classroom context. Mobile devices foster learner autonomy and ubiquitous learning in
informal settings (Chen & Kessler, 2013), yet while there has been a body of work that looks atmobile device activity in the classroom, there has been a significantly smaller amount of research
examining how learners engage in mobile learning outside of the classroom and there have beencalls for more studies into student use of apps outside formal tuition (Steel, 2012; Stockwell & Liu,
2015) as learners’ own use of apps may be quite different from what the app designers imagined(Stockwell & Hubbard, 2013). In a study looking at mobile app use with 140 learners of Chinese,
Mason and Zhang (2017), found that 94% of learners utilised apps independently and the majorityof them recognised the value of apps to support their language learning, but learners only used a
proportion of the functionalities available.

Two further examples of research into autonomous use of language learning apps are the Duolingo
Effectiveness Study (Vesselinov & Grego, 2012) and the research into the use of apps by distancelearners by Rosell-Aguilar (2016). The first study looked into the learning experience and outcomesof 88 native speakers of English learning Spanish at beginner and intermediate level using theDuolingo app, which found high levels of satisfaction. The learners’ main reason for learning was
personal interest and the results showed statistically-significant improvements in language ability as
well as confidence. Participants who studied because they wanted to travel achieved the largest
improvements, and those whose motivation was personal interest the least. In addition, learners who
were at beginner level made bigger gains than advanced learners. The second example was carriedout with 85 distance learners of Spanish (Rosell-Aguilar, 2016). The main area of language learning
for which respondents used apps was vocabulary (82.26%), followed by translation (66.13%), andgrammar practice (58.06%). The majority of language apps users in his study stated that they used
apps informally, as and when the opportunity arose. The study found high frequency of app use
5among the respondents, with 44.26% using at least once a day, and 36.07% several times a week,
and female respondents used apps less frequently but for longer periods of time. All respondents
stated that they felt that apps had improved their knowledge of Spanish.

The study reported in this article focuses on the use of a well-known, popular app: busuu.

Previous research into the effectiveness of the busuu app

The busuu app is part of the busuu network for learning languages, which also comprises a website.
busuu has over 60 million registered users (Busuu, 2016). It offers 12 different languages organisedin levels according to the Common European Framework of Reference for languages (CEFR). The
app runs on Android and iOS operating systems. Learners have to register to use it and can choose
to learn one or more languages. Learners can use the free materials or upgrade to a premium paidfor membership. The activities within the app practise reading, writing, and listening skills as well
as vocabulary and translation. Activity types include drag and drop, multiple choice, filling inblanks, and free writing, which can be shared with other users to obtain peer feedback andcorrections. The app can be used offline if users download learning units.

Research into the use of busuu has been limited, but some studies have taken place. In a pilot study
using the busuu app, Kétyi (2013) surveyed 59 Hungarian students of German who were given afree 7-day premium membership trial. The service was well received, with 79% of participants
rating it as either good or very good. The most beneficial aspects identified were learning
vocabulary, revising and practising the language, making language learning playful and easy, theuse of audio for pronunciation, and the selection of topics and vocabulary. Some participants foundthe tasks too easy and repetitive, though. After the free trial, 92% of participants indicated that theywould not pay for premium membership. In a follow-up study, Kétyi (2015) compared an
6experimental group of 51 students using the premium version of the busuu app with a control groupof 43 students who did not use the app. The app users utilised it about once a week for an averageof between 10 and 15 minutes. Over 73% of them considered the app helpful for language learning.
Respondents found the app particularly useful for vocabulary and practising writing skills. The
experimental group saw a rise in motivation and a statistically-significant increase in their target
language performance, whereas the control group did not. Despite this, the vast majority ofparticipants indicated that they would not pay for the app after the trial period. In a separate study,
Malerba (2015) found that learners particularly appreciated the flexibility of learning at their own
pace, but the limited number of activity types was their main reason for abandoning such language
learning platforms.

In a study with 21 students of English as a Second Language using busuu, Liu et al (2013) foundthat 52% of participants extended their use of busuu outside the classroom. Their use tended to last
between 30 and 60 minutes (some longer) per session. The students’ favourite features within busuu
were engaging with others, learning new vocabulary and practising their reading and grammar. The
authors of this report did not specify whether students had used the desktop or app version of busuu.
They advocated further research on site-specific features to determine which design features havethe best potential for language learners.

The largest study into the use of busuu so far was carried out by Vesselinov and Grego (2016).
Their study tracked the Spanish language improvements of 144 autonomous learners using busuu
over two months, with pre- and post-tests used as measurements. After this period, 84% ofparticipants had improved their written proficiency and 75% had improved their oral proficiency.
The most successful participants were those that invested the longest time using the service. Factors
such as age, gender, level of education, ethnicity, device used, native language, or reasons forstudying Spanish did not have a statistically-significant effect. The study also measured levels of7

user satisfaction with very positive outcomes. Participants used both the web version (58%) and theapp version (42%) of the service but the report authors did not provide an analysis of the data
results based on the version used.

The study presented in this article has a much larger number of participants and focuses on apponly users, hence providing a more specific picture of this area.

Research questions and methods

Research questions

More research is needed to understand the ways learners engage with apps of their choice forautonomous language learning. The main questions this research sought to address are:

• Who uses the busuu app?
• How do they use it?
• What features do they like best / least?
• Does the app meet users’ expectations?
• What patterns of use of the busuu app can be identified from respondents?

This research aims to create a realistic picture of autonomous users in their own settings, as opposed
to previous research, which had either mostly focused on learners using publicly-available apps
because they had been directed to by their teacher, or focused on both the app and web version of
busuu, and not just the app.

Methods8

An online survey was developed based on a previous study into use of apps for language learning(Rosell-Aguilar, 2016) but tailored to the specific features of the busuu app. The survey consisted of
30 items, most of which were multiple-choice, with four open follow-up questions asking
participants to provide further details depending on their replies, and three standalone open
questions (Appendix 1).

The research was approved by the University Human Research Ethics Committee (Ref:
HREC/2016/2212/Aguilar). The questionnaire was hosted at the university survey site with servers
in the UK, complying with European regulations on data hosting and storage. An agreement wasentered into with the Head of Education at busuu, who offered to distribute the link to the survey.
Nothing was asked for in exchange, other than being able to see a summary of the main(anonymised) results.

The survey was available in two languages: English and Spanish, both with the same questions inthe same order. A link to the relevant survey asking potential participants to take part was sent as an
in-app message by busuu to their registered users, therefore ensuring that respondents were active
app users (as opposed to users of the website) before the start of the research. The users wereselected among those who had chosen to use the English or Spanish interface within the busuu app.
No incentive was offered for taking part in the study. The surveys were open for three weeks in
May 2016 and a total of 4095 unique valid responses were collected: 2265 for the English survey
and 1830 for the Spanish one. As none of the questions in the survey were compulsory, not allquestions received the same number of responses.

The results were first analysed using the online survey's own statistics tools. In addition, SPSSv21
was utilised for comparative statistical analysis. Bias analysis was not possible as no information
9

about app non-users was collected. The responses to open questions were categorised following thethematic analysis process suggested by Braun and Clarke (2006).

Results

The results from both surveys are presented here in relation to the research questions. Where
relevant, distinctions will be made between the data from the English language survey and the
Spanish language survey.

Who uses the busuu app?

A profile of respondents is presented in Figure 1.

Figure 1: participant profile by sex, location and age.

Most respondents (79.1%) were learning one language. A further 14.6% were learning two
languages, 3.5% three, and 2.8% more than three languages. Participants were asked to select which
language they were learning with the busuu app. Those who had indicated they were learning morethan one were asked to select the main one they were prioritising. In Table 1, the languages studied
are presented in three columns: all respondents, respondents to the English-language survey andrespondents to the Spanish-language survey.

LanguagestudiedAll
respondents

EnglishsurveySpanishsurveyEnglish1585 (40.5%)638 (29.8%) 947 (53.4%)

French531 (13.6%)

298 (13.9%) 233 (13.1%)

German433 (11.1%)

286 (13.4%) 147 (8.3%)

Spanish325 (8.3%)

311 (14.5%) 14 (0.8%)

Japanese

281 (7.2%)

197 (9.2%)

84 (4.7%)

10

Italian

271 (6.9%)

114 (5.3%)

157 (8.9%)

Turkish159 (4.1%)

139 (6.5%)

20 (1.1%)

Portuguese 151 (3.9%)

43 (2%)108 (6.1%)

Russian

87 (2.2%)

52 (2.4%)

35 (2%)

Chinese58 (1.5%)

36 (1.7%)

22 (1.2%)

Polish31 (0.8%)

24 (1.1%)

7 (0.4%)

Table 1: Languages studied with the busuu app by respondents.

In addition, 80% of all respondents indicated that they did not live in a geographical area where thelanguage is spoken, whereas 20% did. This varied to a large extent between the English andSpanish versions of the survey. In the English version results, the split between those who lived in atarget-language-speaking area and those who did not is 71/29% whereas in the Spanish version the
split was 90.7/9.3%.

Participants were asked to select the level they consider themselves to be in the language they arelearning with the busuu app. Most learners identified as Beginner / A1 (2067, 52.8%), followed byElementary / A2 (912, 23.3%), Intermediate / B1 (641, 16.4%), Upper Intermediate / B2 (204,5.2%), Advanced / C1 (68, 1.7%), and Nearly Fluent / Fluent / C1 (25, 0.6%).

The respondents' reasons for learning the language they selected were varied. Respondents wereallowed to select more than one reason. The most popular, selected by 29% of respondents, was'personal interest'. This was followed by 'I want to use the language when I travel' (17.1%), 'relevant
to my career aspiration or profession' (16.1%), 'I want to study / live overseas' (15.2%), 'I want to
use the language to communicate with family / friends' (11.2%), and 'relevant to my current studies'(7.1%).

11

The replies to the question about whether participants paid for busuu premium membership(n=3596) were as follows: 593 (16.5%) did, 2614 (72.7%) did not, and 389 (10.8%) did not knowwhether they pay for premium membership or not.

How do learners use the app?

Participants were asked which device they used most often to access the busuu app. Most
respondents (63.4%) used their smartphones and a smaller proportion (35%) used tablet devices.
Only 1.6% expressed a preference for the smartwatch version of the app. With regards to the lengthof time participants had been using the app, 43.2% had been using it for less than a month, 31.6%
between one and six months, 7.2 between seven and 12 months and 18.1% over a year.
Respondents were asked how often they used the app and how long they normally spent using it at atime. Figure 2 shows the replies to these questions correlated with gender to allow comparisonswith previous research (Rosell-Aguilar, 2016).

Figure 2: Frequency and length of use of the busuu app by respondents.

To ascertain whether users perceived using the app as a formal or informal language learning
activity, they were asked whether they use the app mostly in planned learning sessions (e.g. routine
evening sessions) or more informally, as and when the opportunity or need arises. One third of
respondents (33%) selected the former and the remaining 67% the latter. Furthermore, participants
were asked what else they did to learn the language they are learning with the app. Some 23.9%
were using the app to supplement other, more formal, activities such as being registered on a course
at school / college / university, or taking part in private tuition or conversation classes. A further
40.2% were not registered on a course but used additional language learning resources
independently, and 35.8% claimed that the busuu app is the only resource they use to learn thelanguage. Those who used additional resources independently were asked to indicate what other12

resources they use (participants were allowed to select more than one resource). The most popularwere websites (27.9%), followed by books or ebooks (23.9%), other apps (17.4%), DVDs or online
videos (12.3%) and CDs/mp3s/podcasts (22%).

As a follow-up question, respondents were asked to select what types of app they used in addition
to busuu from a number of choices. Respondents were asked to select all that applied and a total of5363 responses were collected, presented in Table 2.

Other apps used
Self-study language apps

Translation apps
Dictionary apps
News apps in target
language
Flashcard apps
Language exchange apps
Other apps

Percentage
Examples
47.0% Duolingo (50.8%)
Babbel (13.9%)
Rosetta Stone (8.9%)
Speakeasy (4.6%)
Others (21.7%)
19.4% Google Translate, iTranslate
16.3% WordReference
6.5% BBC News
5.8% Quizlet, Memrise
3% Tandem, HelloTalk
2% YouTube, Skype, Facebook,
Twitter, Instagram

Table 2: apps other than busuu used by respondents.

What features do users like best / least?

Participants were asked to select up to three features that they liked best and least among the busuu
app activity types, presented in Table 3. Among the 5.8% who selected 'other' in the least-liked
features, the most common comment related the fact that premium membership is required to accessthe full content of the app. Other comments referred to some software glitches or crashes, the chatfacility, and lack of oral practice exercises.

Best-liked features
Vocabulary practice
Listening practiceGrammar practice

20.3%
16.6%
15.1%

Least-liked features
Writing practiceGrammar practiceCorrecting other members' writing

14.8%
14.2%
12.0%

13

Reading practice
Writing practiceTranslation practice
Feedback on own writing
Correcting other members' writing
Other

13.0%
12.1%
9.0%
6.9%
6.5%
0.5%

Listening practiceReading practiceTranslation practice
Vocabulary practice
Feedback on own writing
Other

11.5%
11.3%
10.8%
10.5%
8.8%
5.8%

Table 3: best and least-liked features of the busuu app.

The quality of the automated feedback that users receive from the app (which within the app islimited to whether the answer is right or wrong) was positively regarded. Of the 3751 participants
who replied to the question about feedback quality, 43.8% considered it 'very good' and a further
32.3% rated it 'good'. Some 14.4% rated the feedback 'OK' and 2.5% and 0.7% rated it negatively as'not very good' and 'very bad' respectively. The remaining 6.3% claimed that they did not receive
any feedback.

Respondents were provided with a number of choices about what else they would like the app toprovide. They were allowed to select all the choices they considered relevant. A total of 19,482
responses were collected, and the results appear in Figure 3 in order of popularity. The comments
provided by those who selected 'other' mainly refer to making the premium content free. Other
comments that recurred suggested the use of video materials as stimuli for activities and theprovision of audio and text chat for communication among learners and native speakers.

Figure 3: additional features users would like the busuu app to provide.

Does the app meet users’ expectations?

The participants' most popular expectation when they downloaded it was to improve their speaking
skills (selected by 15.6% of respondents), followed by listening skills (13.6%), reading (11.5%),
and writing (11.1%). The fifth most popular expectation (with 10.8% of responses) was to become14fluent in the language, followed closely by improving vocabulary (10.8%), grammar (9.8%), andtranslation skills (8.6%). The remaining expectation, selected by 6.3% of respondents, was to meet
people with whom to practise the language. Finally, 1.9% of respondents indicated that they had no
expectations when they downloaded the app. The app is well rated in terms of meeting expectations,with 30.5% of respondents stating that the app is better than they expected and 61.9% rating it 'as
expected'. Only 7.7% of respondents rated the app 'worse than expected'.

To produce an overall snapshot of how useful participants found the app, they were asked to what
extent they agreed with the statement 'using the busuu app has helped me improve my knowledge ofthe language I'm learning'. A total of 3,541 participants responded to this question. Some 25.4%
strongly agreed, 57.4% agreed, and 15.3% neither agreed nor disagreed. In contrast, only 1.5% ofparticipants disagreed and 0.5% strongly disagreed. Similarly, participants were asked to rate theoverall quality of the busuu app. Out of the 3565 responses received to this question, 38.7% rated it'very good' and 47.5% rated it as 'good'. Some 12.4% of participants gave the app an 'OK' rating andonly 1.5% gave a negative rating: 1.2% rated it 'not so good' and 0.3% rated it 'very bad'.

When asked to select the area that participants thought they had improved the most, 27.7% selected'vocabulary'. This was followed by 'speaking' (16.2%), 'listening' (12.9%), 'grammar' (11.1%),'reading' (8.6%), 'writing' (7.6%), 'pronunciation' (6.8%) and 'translation' (3.9%). A further 5.1% of
respondents selected 'none'.

Patterns of use

Respondents were able to select one of four categories to describe the length of time they had beenusing the busuu app and how frequently they used the app, ranging from multiple times per day to15

less than once a week (as presented in Figure 2). This data was cross referenced to identify six subgroups (Table 4):
5. How long have you
been using Busuu?

<1 month
1-6 months
7-12 months>1 year

Q5
Total

Severaltimes per
day
2292
811
111
221
343

6. How often do you use Busuu?
Every daySeveral times About once aa week
week
4502
1771
341
901
751

4495
4214
933
1933
1156

2766
2904
663
1633
795

Total
Less thanonce a week
2636
2704
763
2393
848

1667
1239
280
707
3893

Superscript indicates group membership

Table 4. Cross-tabulation of questions 5 and 6
1. Consistent users: those who have been using the app for more than one month, and use it
every day (or more frequently) (n = 415).
2. Enthusiasts: they use the app frequently (at least once every day), but have been using theapp for less than one month (n = 679).
3. Committed users: they use the app several times per week and have used the app for aperiod of more than 7 months (n = 830).
4. Casual users: they have been using the app between once and six months and use it several
times a week or fewer (n = 981).
5. Probing users: those who have been using the app for less than a month but have used it
several times a week (n = 449).
6. Visitors: they use the app infrequently (once a week or less) and have been using the app forless than one month (n = 539).

Group membership (1-6) was used as a new independent variable to determine whether patterns ofusage reflected users’ attitude towards the busuu app in the Likert-type items. One-way analysis ofvariance (ANOVA) and post-hoc Tukey honest significant differences (HSD) tests were then used
to identify statistically significant differences between groups. Due to the number of statistical tests

16

being carried out, Bonferroni correction was used to identify a stricter level of significance toreduce the likelihood of type I error (p < .003). Findings are presented in Appendix 2.

One-way ANOVA suggested that the six groups were appropriate for identifying different response
patterns in the data. Patterns of use were indicative of users’ attitudes towards the app and its utility
for language learning, as there were significant differences (p < .001) for fifteen of the items forwhich significance testing took place.

The largest effect sizes were associated with three items. The first item indicated that those whohave been using the app for the longest time tend to use it for longer periods each time they engage
with a study session. The second item indicated that the longer individuals use the app, the more
likely they are to indicate that it has had a positive effect on their language learning (Table 5).
Finally, the third item indicated that the longer that individuals use the app for, the greater the
likelihood that they will pay for premium content.

‘Using the busuu app has helped me improve my knowledge of the language I’mlearning’
Strongly agree

Agree

Neither agree
nor disagree

Disagree

Total

Strongly
disagreeConsistent users

128

195

26

4

3

356

Enthusiasts

213

303

58

5

1

580

Committed users

186

462

126

9

1

784

Casual users

189

554

155

15

7

920

Probing users

90

251

53

4

1

399

Visitors

75

242

117

13

5

452

Total

881

2007

535

50

18

3491

Table 5. User groups’ attitudes towards the busuu app

17

The most unique group across all items was group 6, the visitors, which was significantly different
from all other groups in four items. This group were the least likely to upload examples of theirwriting and request feedback, used the app for a shorter period than other groups, and were
therefore the most likely to state that the app had not had a positive impact on their learning.

Other results: paying for content, further thoughts, and assessment

Among those participants who pay for premium membership, their main reasons for paying were togain access to additional content (35.6%), because it represents good value for money (27.4%), tomotivate them to keep using it given that they're paying for it (14.4%), and access to support(14.3%). Among participants who do not pay for premium membership, their main reasons for not
subscribing to this service were the belief that language learning apps should be free (43.2%), nothaving a way of paying (29.4%), finding the cost too expensive (9.8%), and concerns about thesafety of online payments (6.1%).

A total of 1,146 participants provided responses to an open question asking them to provide any
further thoughts on the app. The thematic analysis categorised the replies into 5 groups: positive,
negative, comments regarding cost, requests for further content or functionality, and other. Some
responses were coded into more than one category as some participants’ comments covered severalof these, so the total number of comments categorised amounted to 1,279. Of these, 46.8% werepositive. Comments in this category ranged from generally positive such as ‘OK’, to very
enthusiastically positive, such as ‘awesome’, ‘revolutionary’ or ‘perfect’. Respondents praised thequality of teaching (‘the quality of the texts is very good, usually consistently the right level forme’), the flexibility provided by the portability of app (‘I can practise any time’), its interface (‘avery easy app to use’), the ease of being able to practise different language skills within one app(‘esta aplicación me ofrece aprender varios idiomas y no solo uno’ [this app offers me several
18

languages to learn, not just one]), and the community formed among learners and native speakers(‘really like the community aspect’). Some learners also mentioned how motivating using the app
had been for them (‘es muy gratificante’ [It’s very satisfying], ‘[it] has inspired me to further myjourney in learning other languages’).

A total of 6.4% of comments were categorised as negative. The issues most commonly mentioned
in this category included quality of teaching (‘the grammar lessons are not clear’, ‘it’s missing an
incredible amount of creativity’), software bugs / crashes, problems logging in, customer service,finding the level too difficult for beginners, particularly for languages with characters other than the
Latin alphabet such as Russian, Chinese and Japanese learners (‘for beginners [it is] hard torecognize new characters and its [sic] meaning’). Other comments included poor interface (‘Not asinteractive or intuitive as I would have liked’), and lack of variety in activity types (‘ejercicios
repetitivos y aburridos’ [boring and repetitive exercises]). Some respondents compared the app toother apps (mostly Duolingo) unfavourably and several commented on the app giving access tofewer tools than the website version of busuu (‘the PC version is a worthy learning tool… themobile app is very poor’).

Some 14% of responses were categorised under ‘cost’. These comments refer to the pricing for thepremium membership being too high or a preference for a one-off payment rather than a monthly
subscription. The majority of comments on this section reflect the respondents’ beliefs that apps forlanguage learning should be free. Among the 200 comments (17.2%) categorised as requests forfurther content or functionality, the most common were requests for more grammar explanations /practice, oral practice and pronunciation exercises. Many respondents also missed the ability to use
synchronous audio / video chat with native speakers, which used to be available on the app andremains available on the website version. Other requests include the addition of further languages,
the use of more video, certification of attainment, and provision of online tutorials. Finally, 200
19

comments (17.2%) were categorised as ‘other’. Some of these referred to the participants’experience of using the app: there were some reflections on issues relevant to informal and distancelearning such as self-management, discipline, keeping motivated, commitment, and effort (‘I needmore self discipline to be more consistent using the app’, ‘I need to commit more time’, ‘yo soy laque debo esforzarme más’ [it’s me who needs to make a bigger effort]).

The final question asked participants whether they would be interested in taking some form ofassessment leading to a certification in the language they study if busuu offered one. Most of the3,579 respondents to this question showed an interest in assessment: 41.6% chose 'yes' and 51.8%'yes, but only if it was free'. The remaining participants (6.6%) selected 'no'.

Discussion
The results gathered have provided a rich amount of data in response to the research questions. In
this section, the results are discussed in relation to previous research and their contribution to thefield of language learning with apps.

The data collected provided a profile of the busuu app user. The language of tuition marked an
important difference among respondents, highlighting the fact that a number of learners learn alanguage with that same language as the medium of instruction. The results suggest that a large
number of users, Asian learners in particular, choose to learn English through an English interface.
This is supported by the fact that 29.8% of respondents to the English-language survey werelearning English, whereas only 0.8% of those who took the Spanish-language survey were learning
Spanish. With regards to language level, CEFR levels A1 and A2 accounted for 76.1% of
respondents. This supports the claims that apps attract and can best support learners at beginnerlevel (Vesselinov & Grego, 2012, 2016). The most popular reason for learning in this study was 'for
personal interest', which had been identified as a popular response but less effective for language
20

learning with apps by previous research (Vesselinov & Grego, 2012). It was counterintuitive that'relevant to current studies' was the reasons for using the app that received the lowest percentage, as
it might have been anticipated that supporting formal learning would be a popular reason to use
apps, particularly as almost 24% of respondents reported being registered on a formal tuition
course. The split among those users who use the app when the opportunity arises and those who
used it in planned study sessions was consistent with previous findings by Rosell-Aguilar (2016).

With regards to how the app is used, the correlation between gender and frequency and length ofuse of the app shows that female users tended to use the app less frequently but for a longer periodsof time, which is also consistent with previous research by Rosell-Aguilar (2016). The frequency ofreported use was similar to that of Lui et al’s (2013) research but much higher than previous studies
into the use of busuu (Kétyi, 2015). One key difference between the participants in Kétyi's researchand this study is that Kétyi introduced his participants to the app, whereas the participants in thisresearch are already familiar with it. This probably explains the difference in terms of frequency ofuse and also in willingness to pay for the premium service. Other results, such as finding using theapp motivating or the repetitiveness of the activity types as a drawback, were consistent with
Kétyi’s findings as well as Malerba’s (2015).

The overall positive impression of the app (82.8% agree or strongly agree that using busuu has
helped them improve their knowledge of the language and 86.2% rated the app as good or verygood) matches the findings from previous research into the use of apps for language learning(Brown, Castellano, Hugher & Worth, 2012; Castañeda & Cho, 2016; Khaddage & Lattemann,2013) and into the busuu app in particular (Kétyi, 2013, 2015; Vesselinov & Grego, 2016). Also
consistent with Kétyi’s research is the fact that learning vocabulary was the skill that users
considered to have benefitted from most. The results show that users like many of the features, but
not all features are popular with all learners: for example, whereas 12.1% of respondents considered21writing practice as one of the best features, it was also considered one of the least liked features by
14.8% of them. Learners preferences and different aims for learning the language may account forthese differences.

Desire for 'real' interaction ranks high on learners' wish lists. Recent developments by otherproviders, such as Duolingo's chatbot, bring the learner one step closer to interacting, albeit withlimitations, in the target language. Even though this interaction was, at the time the study was
carried out, limited to textual output, learners can make use of the speech-to-text tool on theirdevices to enhance the oral element of the interaction. The recently-released 'friends' feature on
busuu provides the opportunity for learners to interact both in writing and in speech in a closed
environment with chosen peers and gain confidence without exposing themselves to making errors
in public.

The expectations about the potential of apps for language learning suggest that a number of users
now consider apps a credible and reliable source of language learning. This was supported by thefinding that over a third of participants used busuu as their only source of language learning. These
facts show a certain level of expectations about the amount and quality of features that apps canprovide which may be difficult for some providers to achieve at this point, but paves the way forfurther developments and directions in language learning through apps. The issue of mismatch of
expectations and what can be achieved with apps for language learning is likely to continue forsome time. Whilst it is unlikely that a learner would pick up a grammar or exercise book and expect
it to provide a full language learning experience inclusive of interaction with other target language
speakers, it seems that this expectation is held by some language learning app users. Perhaps thisexpectation is not altogether unreasonable: users know that they can use their mobile devices tointeract with other people in real time using text messaging, audio and video calls, as well as social
media, so it makes sense that these should be incorporated into the learning experience. In part, this22

assumption is fuelled by the publicity materials from some app providers who continue to market
them as full language learning solutions.

The findings of the correlation between patterns of use and attitudes towards the app wereconsistent with some of the results found in previous research, such as the effect that the amount oftime spent of the app has on performance improvement (Vesselinov & Grego, 2016), or in this casethe belief that performance has improved.

The responses to the open question inviting participants to provide further thoughts were also very
positive and the number of negative responses was relatively low. These comments highlighted thehigh expectations that users have of what can be achieved from using an app, as the requests formore grammar, more interaction, more variety of activity types demonstrate. The respondents’comments on convenience also echo previous results by Zhou & Li (2015).

The fact that respondents were overwhelmingly interested in some sort of certification of theirinformal studies with the app supports the idea that learners are willing to accredit their informal
learning (Law, 2016). With nearly half of these indicating that they be willing to pay for theaccreditation, this is a direction where app developers might want to go to monetise their business.

Limitations, further research and conclusion

Limitations

Since no previous research into language learning from a commercial app using in-app messages as
the call to participate has been published before, it is not possible to comment on the return rate andits comparability to previous studies.
23An error during the upload of the survey meant that two questions that had been asked on aprevious iteration of the research (Rosell-Aguilar, 2016), were left off the survey. This error was
noticed only after responses had started to be received, and it was decided not to change the survey
at that point. These questions asked learners where and when they used the app. This information
would have been very useful to provide a fuller picture of app use.
The data collected for this study is self-reported and therefore subject to the limitations that suchresearch methods have (Paulhus & Vazire, 2007). Although no incentive was offered forparticipating in the research and it was clearly stated that the study was carried out by a university
researcher, it is a possibility that some respondents may have been inclined to offer positive
responses due to the fact that the call for participants came from within the busuu app. The
questionnaire did not manage to capture the voices of the infrequent visitors who did not take up theapp as part of their language learning and which features dissuaded them, and how these could havebeen improved.

The anonymous nature of the data collection meant that the data collected could not be matched todata available to busuu on factors such as actual frequency of use or performance (as opposed to thereported use).

Further researchSome of the limitations of this study listed above provide possible lines of inquiry for further
research: a smaller study could select a number of users to complete the survey and, with theirpermission, correlate this information with the performance data that busuu collects from its users.
It would also be of interest to survey users who have stopped utilising the app to find out why they
did so. The relatively low participation in providing feedback to peers and the fact that users listed
providing feedback as very low on their list of favourite features may be worthy of further
24

investigation. Finally, this study has focused on the affordances of busuu as an app and not on thesocial aspects that the web version has as a Language Learning Social Networking Site, which
merits further research.

Conclusion

This article advances the field of mobile apps for language learning by providing a large-scale studyof autonomous app users, as advocated by Steel (2012). It is innovative in its methods, by sourcing
the data collection directly from within the app. The information gathered about issues such as
choice of device, gender differences in usage, reasons for learning, best and least-liked features, andself-reported improvement provides the first profile of autonomous language learning app users,
their behaviours and preferences, and their views of language learning through mobile apps. This
research is also the first statistically-supported attempt to categorise learners by their use of the app.
The pattern findings provided new information about use of the app, such as the fact that the longer
users have been using the app, the longer their study sessions tend to be. Another new finding was
the fact that the longer learners have been using the app, the more likely they are to pay forpremium content. These data enhance our knowledge of the field and have the potential to improve
the learning experience of language learners, as well as being of use to language teachers and app
developers.

One final observation arises from the fact that many of the features that language app users wish forare already available to language learners through their mobile devices. The device native tools(making calls, messaging, etc.) as well as the wide range of apps and services for finding partners
for exchanges (e.g. the Tandem app), participating in one-to-one tutorials (iTalki), watching andlistening to authentic content (news apps, YouTube…), and much more, can create an immersive
environment for the language learner and some do use these features. However, many learners seem25

to want an all-in-one solution within the same app rather than engage in app-smashing to achievethis. Apps such as Apple's Health app provide a dashboard that can incorporate activity tracked
from other fitness apps (e.g. swimming or running) into a single interface, and this is a model withgreat potential for language learning. A single app that incorporated, for example, grammar
explanations from one app, exercises from another, vocabulary from a flashcard app, availability ofpartners from an e-tandem app, and culturally-rich authentic content from media apps into a single
dashboard would provide a single-solution immersive environment for language learners. Whether
this is the future of language learning through apps remains to be seen.

Acknowledgements:
I would like to thank Kirsten Campbell-Howes, Head of Education at busuu, for her help andenthusiastic support. I would also like to thank Agnes Kukulska-Hulme, Nathaniel Owen andAlessandro Taffetani for their help.

26

REFERENCES
Brown, M., Castellano, J., Hughes, E., & Worth, A. (2012). Integration of iPads into a Japanese
university English language curriculum. The JALT CALL Journal, 8(3), 193-205.
Braun, V., & Clarke, M.J. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77-101.
Burston, J. (2014). The Reality of MALL: Still on the Fringes. CALICO Journal, 31(1), 103-125.
Busuu (2016) Our 8th Anniversary - Busuu in numbers. Retrieved from https://blog.busuu.com/8years-busuu-numbers/
Castañeda, D. A., & Cho, M. H. (2016). Use of a game-like application on a mobile device toimprove accuracy in conjugating Spanish verbs. Computer Assisted Language Learning, 29(7),1195-1204.
Chapelle, C. (1998) Multimedia CALL: lessons to be learned from research on instructed SLA.
Language Learning and Technology, Vol. 2, No. 1, 22-34.
Chen, X. B., & Kessler, G. (2013). Action research tablets for informal language learning: Student
usage and attitudes. Language Learning & Technology, 17(1), 20-36.
Doughty, C. & Long, M. H. (2003) Optimal Psycholinguistic Environments for Distance FL
Learning. Language Learning and Technology, 7 (3), 50-80.
Godwin-Jones, R. (2011). Emerging technologies: Mobile apps for language learning. Language
Learning & Technology, 15(2), 2-11.
Kétyi, A. (2013, November). Using Smart Phones in Language Learning – A pilot study to turn
CALL into MALL. In L. Bradley & S. Thouësny (Eds.), 20 Years of EUROCALL: Learning fromthe Past, Looking to the Future. Proceedings of the 2013 EUROCALL Conference, Évora, Portugal(pp. 129-134). Dublin, Ireland: Research-publishing.net.
Kétyi, A. (2015, December). Practical evaluation of a mobile language learning tool in higher
education. In F. Helm, L. Bradley, M. Guarda & S. Thouësny (Eds), Critical CALL – Proceedingsof the 2015 EUROCALL Conference, Padova, Italy (pp. 306-311). Dublin, Ireland: Researchpublishing.net.
Khaddage, F., &Lattemann C. (2013). The Future of Mobile Apps for Teaching and Learning. In
Berge, Z. L., & Muilenburg, L. (Eds.), Handbook of Mobile Education. (119-128). New York:
Routledge.
Kim, H. S. (2013). Emerging Mobile Apps to Improve English Listening Skills. MultimediaAssisted Language Learning, 16(2), 11--30.
Kim, H., & Kwon, Y. (2012).Exploring smartphone applications for effective Mobile-Assisted
Language Learning. Multimedia-Assisted Language Learning, 16(1), 31-57.
Lafford, P. (2011). Afterword - iPads in the Language Classroom. In Facer, B. R., & Abdous, M.
(Eds.), Academic podcasting and mobile assisted language learning: Applications and outcomes,(196-197).Hershey, New York: IGI Global.
27

Law, P. (2016). The Identified Informal Learner: Recognizing Assessed Learning in the Open. In:
T.J. Bliss, and P. Blessinger (eds). Open Education: International Perspectives in Higher
Education. Cambridge, United Kingdom: Open Book Publishers, 137-158.
Liu, M., Evans, M., Horwitz, E., Lee, S., McRory, M., J. Park, and Parrish, C. (2013). A study ofthe use of social networking sites for language learning by university ESL students. In M.-N. Lamy
& K. Zorou (Eds.) Social networking for language education (pp.15-41). Basingstoke, UK:
Palgrave Macmillan.
Malerba, M. L. (2015). Learners' behaviours and autonomy in LiveMocha and busuu online
communities. Paper presented at the EDEN 2015 Annual Conference, Barcelona, Spain, 11th June
2015.
Mason, A. & Zhang, W. (2017) An exploration of the use of mobile applications to support thelearning of Chinese characters employed by students of Chinese as a foreign language. In Q. Kan &S. Bax (Eds.) Beyond the language classroom: researching MOOCs and other innovations (99112). Research-publishing.net.
Meskill, C. (1999).Computers as Tools for Sociocollaborative Language Learning. In K. Cameron(Ed), CALL: Media, Design and Applications, (141-162). Lisse: Swets & Zeitlinger.
Morgana, V. (2015) Investigating students' and teachers' perceptions of using the iPad in the EFL
classroom. Paper presented at the EuroCALL 2015 conference. Padova, Italy, 26-29th August.
Oxford, R. (1990). Language learning strategies: What every teacher should know. New York:
Newbury House.
Pareja-Lora, A., Arús, J., Martín Monje, E., Read, T., Pomposo-Yanes, L., Rodríguez-Arancón, P.,
Calle-Martínez, C., & Bárcena, E. (2013, November). Toward Mobile Assisted Language Learningapps for professionals that integrate learning into the daily routine. In L. Bradley & S. Thouësny(eds.), 20 Years of EUROCALL: Learning from the Past, Looking to the Future. Proceedings of the2013 EUROCALL Conference, Évora, Portugal (pp. 206-210). Dublin, Ireland: Researchpublishing.net.
Paulhus, D. L., & Vazire, S. (2007). The self-report method. In R. W. Robins, R. C. Fraley, & R. F.
Krueger (Eds.), Handbook of research methods in personality (pp. 224–239). London, England:
Guilford.
Rachels, J. R. & Rockinson-Szapkiw, A. J. (2017) The effects of a mobile gamification app onelementary students’ Spanish achievement and self-efficacy. Computer Assisted Language
Learning, 31(1-2), 72-89. DOI: 10.1080/09588221.2017.1382536
Rosell-Aguilar, F. (2009). Podcasting for language learning: re-examining the potential. In L.
Lomicka & G. Lord (Eds.), The Next Generation: social networking and online collaboration inforeign language learning, (pp. 13-34). San Marcos, TX: Calico.
Rosell-Aguilar, F. & Kan, Q. (2015).Design and user evaluation of a mobile app to teach Chinesecharacters. JaltCALL Journal, 11(1), 19-40.
Rosell-Aguilar, F. (2016). User evaluation of language learning mobile applications: a case study
with learners of Spanish. In: Palalas, Agnieszka and Ally, Mohamed eds. The International
28

Handbook of Mobile-Assisted Language Learning. Beijing: China Central Radio & TV University
Press, pp. 545–581.
Rosell-Aguilar, F. (2017). State of the App: a taxonomy and framework for evaluating language
learning mobile applications. The CALICO Journal, 34 (2), 243-258.
Skehan, P. (2003). Focus on Form, Tasks, and Technology. Computer Assisted Language Learning,
16(5), 391-411.
Steel, Caroline (2012, November). Fitting learning into life: language students’ perspectives on
benefits of using mobile apps. In: M. Brown, M. Hartnett and T. Stewart, ascilite2012 Conference
proceedings. ASCILITE 2012 - Annual conference of the Australian Society for Computers in
Tertiary Education (pp. 875-880). Wellington: New Zealand.
Stockwell, G. & Liu, Y. C. (2015). Engaging in mobile phone-based activities for learning
vocabulary: An investigation in Japan and Taiwan. CALICO Journal 32(2) 299-322.
Stockwell, G., & Hubbard, P. (2013). Some emerging principles for mobile-assisted language
learning. Monterey, CA: The International Research Foundation for English Language Education,
1-15.
Vesselinov, R., & Grego, J. (2012). Duolingo effectiveness study. Retrieved fromhttps://static.duolingo.com/s3/DuolingoReport_Final.pdf
Vesselinov, R., & Grego, J. (2016). The busuu efficacy study. Retrieved fromhttp://comparelanguageapps.com/documentation/The_busuu_Study2016.pdf
Yildiz, S. (2012, June). Use of iPad applications to introduce English as a foreign language toyoung Turkish learners. Paper presented at the 2012 CALICO Conference, University of Notre
Dame, South Bend, IN.
Zou, B., & Li, J. (2015). Exploring mobile apps for English language teaching and learning. In F.
Helm, L. Bradley, M. Guarda & S. Thouësny (Eds), Critical CALL – Proceedings of the 2015
EUROCALL Conference, Padova, Italy (pp. 564-568). Dublin, Ireland: Research-publishing.net.

29

Appendix 1: Survey
Survey on use of Busuu: This research survey aims to find out more about your use of the Busuu
app. It should take 5-10 minutes to complete. The survey is part of research carried out at The Open
University into the use of language learning apps, i.e. it is not a market research survey on behalf ofBusuu. Your answers will be completely anonymous. You may withdraw from the survey at any
time by exiting this page. You may skip questions if you wish. To find out more about this project,
data protection, and how the data will be stored please follow this link: [Link was provided].
1 Are you?
Male
FemaleI don’t identify as either male or female
2 How old are you?
Under 18
18-25
26-35
36-45
46-55
56-65
Over 65
3 Where do you live?
UK / Ireland
Rest of Europe
USA / Canada
Mexico / Central and South America
AfricaMiddle East
Asia
Australia / New ZealandRest of the world
4 Which device do you mostly use to access the Busuu app? (Please select one)
Smartphone (Galaxy, iPhone or similar)
Tablet (iPad, Kindle Fire, Samsung or similar)
Smartwatch (Apple Watch or similar)
5 How long have you been using the Busuu app?
Less than a month
Between 1 – 6 months
Between 7 – 12 months
Over a year
6 How often do you use the Busuu app?
Several times a day
30

Every day
Several times a week
About once a week
Less frequently than once a week
7 How long do you normally spend using the Busuu app at a time?
Less than 5 minutes
Around 5 minutes
Around 10 minutes
Around 15 minutes
Between 16 and 30 minutes
Over 30 minutes
8 How do you use the Busuu app? (Please select one only)
Mostly in planned learning sessions (e.g. routine evening sessions)
Mostly informally, as when and where the opportunity or need arises
9 How many languages are you learning / practising with the Busuu app?
12
3More than 3
10 What language are you learning with Busuu? If you’re learning more than 1, please select themain one you are focusing on at the moment
ChineseEnglishFrenchGermanItalianJapanese
PolishPortuguese
Russian
SpanishTurkish
11 How would you describe your level in that language?
Beginner (A1)
Elementary (A2)Intermediate (B1)
Upper intermediate (B2)
Advanced (C1)
Nearly fluent / fluent (C2)12 Why are you learning this language? (Select all that apply)31

Personal interest
Relevant to my career aspiration or profession
Relevant to my current studies
I want to study / live overseas
I want to use the language when I travel
I want to use the language to communicate with family/friends
Other
13 Do you currently live in a country / area where the language you’re learning is spoken?
YesNo
14 As well as using the Busuu app, what else do you do to learn the language?
I’m registered on a course to learn this language at school / college / university / private tuition /conversation class
I'm not registered on a course but I use additional resources independently
The Busuu app is the only resource I use to learn the language
If you use additional resources independently, which ones are they? (Select all that apply)
Books / eBooks
CDs / mp3s / podcasts
DVDs / online video
Websites
Other apps
Other
15 What other apps do you use to learn / practise languages? Select all that apply
None
Duolingo
BabbelRosetta Stone
Speakeasy
Other self-study Language Learning Apps
Language partner apps (e,g. Tandem, HelloTalk)
Flashcard apps (e.g. Quizlet, Memrise, Educards)
Dictionary apps
Translation apps
News apps in the language I’m learning
Twitter
Other (please specify)
16 What features do you like BEST in the Busuu app exercises? (Select up to 3 answers)
Vocabulary practiceGrammar practiceReading practice
Writing practice32

Listening practiceTranslation practice
Feedback on your writing from other members of the Busuu community
Correcting other members’ writing in your own language
Other (please specify)
17 What features do you like LEAST in the Busuu app exercises? (Select up to 3 answers)
Vocabulary practiceGrammar practiceReading practice
Writing practice
Listening practiceTranslation practice
Feedback on your writing from other members of the Busuu community
Correcting other members’ writing in your own language
Other (please specify)
18 What do you think of the automated feedback you get about your performance whilst using theactivities in the Busuu app (e.g. when an answer goes red if you make an error)?
Very goodGood
OK
Not very good
Very bad
I don’t get any feedback
19 Do you post your writing exercises for comment by other users?
Yes
No
Why?
If “Yes”, what do you think of the feedback you get from other users?
Very goodGood
OK
Not very good
Very badThe quality varies a lot
I don’t get any feedback
20 Do you provide corrections for other users?
Yes
No
21 Do you use the My Vocabulary tool to store words or phrases you want to go back to?
33

Yes
No
If “Yes”, do you actually use My Vocabulary as a revision tool?
Very often
Often
Occasionally
RarelyNever
22 What more would you like to get from the Busuu app? (select all that apply)
More grammar explanations
More grammar exercises
More reading practiceMore writing practiceMore listening practiceMore translation practiceMore written interaction with native speakers
Speaking practice
Pronunciation explanations
Pronunciation exercises
Placement test
Spoken interaction with native speakers
Small group classes with professional teachers
One to one classes with professional teachers
Other (please specify)
23 What expectations did you have of the Busuu app when you downloaded it? (select all that
apply)
I would improve my writing in the language
I would improve my reading in the language
I would improve my listening skills in the language
I would improve my speaking skills in the language
I would improve my translation skills in the language
I would improve my grammar in the language
I would improve my vocabulary in the language
I would be fluent in the language
I would meet people to practice the language with
I had no expectations
Has the Busuu app met your expectations?
Better than expected
As expected
Worse than expected
24 To what extent do you agree with the following statement: “Using the Busuu app has helped me
improve my knowledge of the language I’m learning”
Strongly agree
34

AgreeNeither agree nor disagree
DisagreeStrongly disagree
25 Which areas do you think you’ve improved thanks to using the Busuu app? (select all that apply)Grammar
Vocabulary
Writing
Reading
Speaking
Listening
Pronunciation
Translation
None
26 Which area do you think you’ve improved MOST thanks to using the Busuu app? (select one
only)
Grammar
Vocabulary
Writing
Reading
Speaking
Listening
Pronunciation
Translation
None
27 Please rate the overall quality of the Busuu app
Very goodGood
OK
Not so good
Very bad
28 Do you pay for premium membership?
YesNo
I don’t know
If Yes, why?
Good value for money
Access to additional content
Access to support
It motivates me to keep using it since I’m paying for it
Other
If No, why?
35

I think language learning apps should be freeToo expensive
I don’t have a way of paying
I worry about the safety of online payments
I was not aware of thisOther
29 Please share any other thoughts you have about learning with the Busuu app.
30 If Busuu offered the possibility of taking some form of assessment leading to a certification in
the language you study, would you be interested? (Please note that your answers will NOT be used
to target any promotional content regardless of what you reply).
Yes
Yes, but only if it was free.
No
Thank you very much for taking part in this survey.

36

Appendix 2. One-way ANOVA with eta-squared effect sizes
Items
Nationality

Between Groups
Within Groups

1.

Are you male/female/identify asneither?

Sum ofSquares
43.304920.128df
5
3887

963.432

3892

6.877
1038.819

5
3873

Total
Between Groups
Within Groups

1045.696

3878

606.575
8679.668

5
3876

Total
Between Groups
Within Groups2.

How old are you?

3881

Which device do you use to accessthe Busuu app?

Total
Between Groups
Within Groups

9286.2434.

14.729953.6895
3623

968.4183628

How long do you normally spend
using the Busuu app?

Total
Between Groups
Within Groups

390.8997473.486

5
3858

Total
Between Groups
Within Groups

7864.385

3863

61.975777.85053797

839.8243802

64.2171679.640

5
3833

7.

8.

How do you use the Busuu app?

9.

How many languages are youlearning with the Busuu app?

Total
Between Groups
Within Groups

1743.857

3838

11. How would you describe your levelin your main target language?

Total
Between Groups
Within Groups

184.4924126.868

5
3829

Total
Between Groups
Within Groups

4311.360

3834

14.998592.356

5
3808

Total
Between Groups
Within Groups

607.354

3813

5.237
2194.381

5
3779

Total
Between Groups
Within Groups

2199.618

3784

61.2796438.950

5
3686

Total
Between Groups
Within Groups

6500.229

3691

19. Do you post writing exercises forcomment by other users?

47.532845.4065
3688

892.9383693

23. a. Has the Busuu app met yourexpectations?

Total
Between Groups
Within Groups

28.7321094.783

5
3399

Total
Between Groups
Within Groups

1123.514

3404

63.6901703.750

5
3485

Total
27. Please rate the overall quality of the Between Groups
Busuu appWithin Groups

1767.440

3490

30.7831827.045

5
3511

Total
Between Groups

1857.828

3516

Within Groups

35.001
921.902

5
3536

Total

956.9033541

13. Do you currently live in an area
where your main target language isspoken?
14. As well as the Busuu app, what else
do you do to learn the language?
18. What do you think of the automatedfeedback you receive about yourperformance while using Busuu?

24. “Using the Busuu app has helped
me improve my knowledge of thetarget language?”

28. Do you pay for premium
membership?

F

Sig.

η2

36.587

.000

.000

1.375
.268

5.128

.000

.008

121.315
2.239

54.175

.000

.006

2.946
.263

11.191.000.003

78.1801.937

40.358.000.030

12.395.205

60.505.000.021

12.843.438

29.309.000.009

36.8981.078

34.235

.000

.014

3.000
.156

19.283

.000

.007

1.047
.581

1.804

.109

.002

12.256
1.747

7.016

.000

.008

9.506
.229

41.471

.000

.016

5.746
.322

17.841.000.017

12.738.489

26.056

.000

.030

6.157
.520

11.831

.000

.016

7.000
.261

26.849

.000

.035

Mean
Square8.661.237

37