# Feature Based Sentiment Analysis for Evaluating the Mobile Pedagogical Affordances of Apps

Muneera Bano, Didar Zowghi, Matthew Kearney

► To cite this version:

Muneera Bano, Didar Zowghi, Matthew Kearney. Feature Based Sentiment Analysis for Evaluating the Mobile Pedagogical Affordances of Apps. 11th IFIP World Conference on Computers in Education (WCCE), Jul 2017, Dublin, Ireland. pp.281-291, 10.1007/978-3-319-74310-3_30. hal-01762857

HAL Id: hal-01762857

https://inria.hal.science/hal-01762857

Submitted on 10 Apr 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Feature Based Sentiment Analysis for Evaluating the Mobile Pedagogical Affordances of Apps

Muneera Bano<fnref n="1" />, Didar Zowghi<fnref n="1" />, Matthew Kearney<fnref n="2" />

**Abstract.** The launch of millions of apps has made it challenging for teachers to select the most suitable educational app to support students' learning. Several evaluation frameworks have been proposed in the research literature to assist teachers in selecting the right apps for their needs. This paper presents preliminary results of an innovative technique for evaluating educational mobile apps by analysing the feedback of past app users through the lens of a mobile pedagogical perspective. We have utilized a sentiment analysis tool to assess the opinions of the app users through the lens of the criteria offered by a rigorous mobile learning pedagogical framework highlighting the learners' experience of Personalization, Authenticity and Collaboration (iPAC). The investigation has provided initial confirmation of the powerful utility of the feature based sentiment analysis technique for evaluating the mobile pedagogical affordances of learning apps.

**Keywords.** Mobile Learning, Sentiment Analysis, m-Learning Pedagogies

## 1. Introduction

Over the last decade, a significant amount of research has been conducted to investigate the effectiveness of mobile learning apps in school education [1]. The number of apps has increased exponentially in the last decade and there are millions of educational apps available for educators and students. However, with overload of choice, and the increasing speed of technological development—much of which is driven by corporate markets rather than education [2], it has become challenging for teachers to efficiently select an app that best supports appropriate learning activity types and assessment strategies, and associated pedagogical preferences [3]. To add to these challenges, the majority of apps in repositories such as the iTunes Store are ‘drill and practice’ or ‘instructive’ in nature [4], underpinned by traditional behaviourist principles—essentially replicating traditional transmissionist approaches to learning [5-7]. As mobile apps develop and proliferate, the challenge for educators is to move beyond the hype and rhetoric [8] to focus on new mobile pedagogical opportunities with apps.

App stores often provide the facility for user feedback (comments and ratings) in order to help teachers select apps, and for app developers to improve their designs.

<footnote n="1">¹ Faculty of Engineering and Information Technology, University of Technology Sydney, Australia</footnote>

<footnote n="2">² Faculty of Arts and Social Sciences, University of Technology Sydney, Australia
Muneera.Bano@uts.edu.au, Didar.Zowghi@uts.edu.au, Matthew.Kearney@uts.edu.au</footnote>
These customer ratings and reviews play a critical role in the mobile app market and directly influence app downloads. User feedback has already been used by practitioners and app developers as a source of information in activities such as selection of apps, customer satisfaction, versioning, and bug reports [9-11]. However, the main challenge is processing and synthesising this feedback into useful information. Considering review volumes, analysing every review manually is laborious and time consuming.

Sentiment analysis is an automated approach that aims to determine the polarity of sentiments and emotions within large textual datasets [12]. This approach is used to develop tools for calculating and monitoring the attitude and behaviour of app users from their feedback, comments and reviews in online social media and app review sites [10]. Sentiment analysis tools are a powerful utility in app ranking and selection; however, it has so far been underutilized in the education domain.

In this paper, we present the results of our preliminary investigation exploring the utility of a new technique for evaluating the pedagogical affordances of educational apps. The feedback and comments of app users are assessed for their alignment against evaluation criteria from a well-accepted, rigorous mobile pedagogical framework [13]. This framework focuses on three distinctive mobile pedagogies: personalization, authenticity, and collaboration. The objective of our research is to explore the utility of our novel technique incorporating sentiment analysis and informed by the m-learning pedagogical framework [13].

The main contributions of this research are: (1) Feature based sentiment analysis using the three mobile pedagogical constructs i.e. personalization, authenticity, and collaboration; and (2) initial confirmation of the usefulness of sentiment analysis for evaluating apps in education.

# 2. Background

## 2.1 Mobile Learning

Mobile learning (or m-learning) is described in numerous ways, but these descriptions all consider the nexus between working with mobile devices and the occurrence of learning: the process of learning mediated by a mobile device. Numerous characteristics of m-learning have been identified in the literature [14].

Over the last decade, a significant number of initiatives have been launched that aim to fully utilize and exploit mobile technologies and apps for educational purposes [15]. There is evidence that m-learning environments enhance students’ performance [16, 17]. However, increase in the use of mobile devices does not imply their effective incorporation in educational policies and in practice, mobile devices are not effectively utilised in formal education [18, 19] for a variety of reasons [17, 20].

Various lists of recommended educational apps are available online [21, 22]. These lists are limited in what they provide because they don't guide the teachers and students about pedagogical understanding of how an app could be used to support teaching and learning. Therefore, they are not sufficiently practical to facilitate strong instructional planning and implementation [21]. This has resulted in a pressing need
for an evaluation framework/rubric that facilitates the analysis of pedagogical affordances of educational apps.

## 2.2 A pedagogical framework for m-learning: iPAC

Numerous frameworks have been proposed in the literature, ranging from complex multi-level models (e.g. [23]) to smaller frameworks that often omit important socio-cultural characteristics of learning or of pedagogy. Common themes include portability of m-learning devices and mobility of learners; interactivity; control and communication. The theoretical underpinning for the work described in this paper is a robust and validated mobile pedagogical framework [13]. Informed by sociocultural theory [24], it highlights three central and distinctive pedagogical features of m-learning: personalisation, authenticity and collaboration (or ‘PAC’). The critical influence of context is signalled by the central location of ‘time-space’ at the core of the ‘iPAC’ framework, as depicted in Figure 1.

The personalisation construct consists of the sub-constructs of ‘agency’ and ‘customisation’. High levels of personalisation would mean the learner is able to enjoy an enhanced degree of agency [25] and the flexibility to tailor both tools and activities, interacting with a strong sense of ownership of both the device and the learning process. The authenticity construct privileges opportunities for in-situ, participatory learning [26]. The sub-constructs of ‘task’, ‘tool’ and ‘setting’ focus on learners’ involvement in rich, contextualised tasks, making use of tools in a realistic way, and driven by relevant real-life practices and processes [27]. The collaboration construct captures the conversational, networked features of m-learning. It consists of ‘conversation’ and ‘data sharing’ sub-constructs, as learners engage in negotiated meaning-making, forging connections and interactions with peers, experts and the environment [28]. This iPAC framework provides a useful lens to analyse mobile apps and how use of their features might leverage mobile pedagogies in a range of learning environments.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** The Mobile Pedagogical Framework (iPAC) comprising three distinctive features of mobile learning experiences. Adapted from ([13], p.8)</figcaption></figure>

The iPAC framework has recently been used to inform research on m-learning in school education [29], teacher education [30][35], indigenous education [31] and other areas of higher education [32]. For example, Viberg and Grönlund [33] used the framework to develop a survey instrument for eliciting students’ attitudes toward mobile technology use in and for second and foreign language learning in higher education.
## 2.3 Feature Based Sentiment Analysis

Sentiment analysis is used to analyse human opinions, sentiments, judgements, reviews and behaviours about many aspects of life such as products, business, people, problems, subjects and their features [12]. Sentiment analysis aims to calculate the polarity of emotions in textual data by identifying the positivity or negativity of a statement. Sentiment analysis is one of the widely used evaluation techniques around the world, helping companies to improve their products based on customers' feedback. App stores provide users with facilities to submit their feedback and rank the apps with star ratings [10]. This data is used by the companies to monitor the app users' behaviours and sentiments.

Feature-based sentiment analysis is a specific type of sentiment analysis which aims to capture nuances about objects of interest. Different features of a product can generate different sentiments, for example a mobile phone can have a user-friendly interface but the battery life is very low. This scenario requires identifying relevant entities of interest, extracting these features from the data, and determining whether an opinion expressed on each feature is positive, negative or neutral.

# 3. Study Design

The investigation was carried out in the context of school-based mathematics and science education for two main reasons. Firstly, we are currently conducting a larger ongoing research project<fnref n="1" /> about the effectiveness of mobile apps for science and mathematics in school education, driven by the strong 'political will' in many countries to improve maths and science learning and to build the capability of the workforce for future job markets. Secondly, there is currently a burgeoning interest in STEM education— see for example, the major recent reviews of m-learning research in both science [17] and mathematics [16] education. The investigation was quantitative in nature and addressed the key question: *What is the utility of feature based sentiment analysis for evaluating the mobile pedagogical affordances of educational apps?*

The steps in this investigation were as follows:

1) We selected ten popular discipline-specific education apps (5 science and 5 mathematics) suitable for school students, as described in Table 1. The apps were chosen based on popularity in various forums and blogs.

2) We used a commercial sentiment analysis tool, Appbot (https://appbot.co/), that extracts user reviews and ratings from the app stores and provides full utility of qualitative and quantitative analysis. Other similar sentiment analysis tools could provide the same functionality but we chose Appbot because it provides functions to search within reviews for specific words (i.e. feature extraction) and also allows filtering out relevant reviews that match particular concepts or words.

3) We developed a word bank based on words in the literature associated with the three main constructs of the iPAC framework. Figure 2 shows sample words from these word banks.

<footnote n="1">¹ https://www.uts.edu.au/future-students/education/about-education/news/optimising-mobile-intensive-pedagogies-arc-discovery</footnote>
4) After using the word bank for feature extraction on all of the ten selected apps, we used Appbot to analyse the extracted reviews for the polarity of the sentiments. Data was collected covering reviews from the period of one year i.e. from January 2016 to January 2017 to limit the scope of our investigation.

<table><caption>**Table 1:** Selected Apps for investigation (based on popularity in various forums and blogs)</caption><thead><tr><th></th><th>App</th><th>Store</th><th>Web link</th></tr></thead><tbody><tr><td rowspan="6">Mathematics</td><td>Mathletics Student</td><td>iOS</td><td>http://au.mathletics.com/</td></tr><tr><td>Myscript Calculator</td><td>iOS</td><td>http://www.myscript.com/calculator/</td></tr><tr><td>GeoGebra</td><td>iOS</td><td>https://www.geogebr.org/</td></tr><tr><td>MalMath</td><td>Google Play</td><td>http://www.malmath.com/</td></tr><tr><td>Math: Mental Math Games</td><td>Google Play</td><td>https://play.google.com/store/apps/details?id=com.astepanov.m obile.mindmathtricks&hl=en</td></tr><tr><td>Anatomy 4D</td><td>iOS</td><td>http://anatomy4d.daqri.com/</td></tr><tr><td rowspan="4">Science</td><td>Little Alchemy</td><td>Google Play</td><td>https://littlealchemy.com/</td></tr><tr><td>NASA</td><td>Google Play</td><td>https://play.google.com/store/apps/details?id=gov.nasa&hl=en</td></tr><tr><td>Skeptical Science</td><td>Google Play</td><td>https://www.skepticalscience.com/</td></tr><tr><td>Star Walk</td><td>iOS</td><td>http://vitotechnology.com/star-walk.html</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Word clouds relating to the three constructs of the iPAC Framework [13]</figcaption></figure>

## 4. Results

Table 2 shows the total number of reviews, the percentage of positive sentiments, average of star ratings and the scores. These scores ranged from D- to A+ and are calculated based on the trends in the review sentiments, review volume, and star ratings as expressed by the app users.
<table><caption>**Table 2:** Results from Sentiment Analysis of selected Maths and Science apps</caption><thead><tr><th></th><th>App</th><th>Review Count</th><th>Positive Sentiment</th><th>Avg Review Stars</th><th>Score</th></tr></thead><tbody><tr><td rowspan="5">Maths</td><td>1</td><td>MalMath</td><td>10572</td><td>91.2%</td><td>4.5</td><td>A-</td></tr><tr><td>2</td><td>Myscript Calculator</td><td>1488</td><td>89.3%</td><td>4.3</td><td>B</td></tr><tr><td>3</td><td>Math: Mental Math Games</td><td>47</td><td>88.6%</td><td>4.4</td><td>C-</td></tr><tr><td>4</td><td>GeoGebra</td><td>97</td><td>55.6%</td><td>3.1</td><td>D-</td></tr><tr><td>5</td><td>Mathletics Student</td><td>122</td><td>21.2%</td><td>2.0</td><td>D-</td></tr><tr><td rowspan="5">Science</td><td>1</td><td>Little Alchemy</td><td>7929</td><td>92.3%</td><td>4.5</td><td>A</td></tr><tr><td>2</td><td>NASA</td><td>4161</td><td>88.0%</td><td>4.4</td><td>A-</td></tr><tr><td>3</td><td>Star Walk™</td><td>584</td><td>94.1%</td><td>4.7</td><td>B-</td></tr><tr><td>4</td><td>Skeptical Science</td><td>5</td><td>80.0%</td><td>4.0</td><td>D+</td></tr><tr><td>5</td><td>Anatomy 4D</td><td>30</td><td>58.6%</td><td>3.3</td><td>D-</td></tr></tbody></table>

In the domain of mathematics, MalMath and Myscript Calculator are on top of the list, whereas for science, Little Alchemy and NASA has received highest number of reviews. MalMath and Little Alchemy has received a significant number of reviews from the users and above 90% of these reviews contained positive sentiments. Next, we extracted the data based on the aforementioned word bank. Table 3 shows results for the number of reviews that matched the iPAC word bank. Table 4 shows the breakdown of the sentiments for the extracted reviews for the apps.

<table><caption>**Table 3:** Feature based Sentiment Analysis</caption><thead><tr><th rowspan="2">Apps</th><th rowspan="2">Total Reviews</th><th colspan="2">Authenticity</th><th colspan="2">Collaboration</th><th colspan="2">Personlisation</th></tr><tr><th>reviews</th><th>%</th><th>reviews</th><th>%</th><th>reviews</th><th>%</th></tr></thead><tbody><tr><td>MalMath</td><td>10572</td><td>22</td><td>0.21</td><td>4</td><td>0.04</td><td>40</td><td>0.38</td></tr><tr><td>Mathletics</td><td>122</td><td>8</td><td>6.56</td><td>1</td><td>0.82</td><td>4</td><td>3.28</td></tr><tr><td>GeoGebra</td><td>97</td><td>2</td><td>2.06</td><td>0</td><td>0.00</td><td>3</td><td>3.09</td></tr><tr><td>Math: Mental Math Games</td><td>664</td><td>4</td><td>0.60</td><td>1</td><td>0.15</td><td>6</td><td>0.90</td></tr><tr><td>Myscript Calculator</td><td>1488</td><td>19</td><td>1.28</td><td>3</td><td>0.20</td><td>31</td><td>2.08</td></tr><tr><td>Little Alchemy</td><td>7929</td><td>13</td><td>0.16</td><td>25</td><td>0.32</td><td>88</td><td>1.11</td></tr><tr><td>Skeptical Science</td><td>5</td><td>0</td><td>0.00</td><td>0</td><td>0.00</td><td>0</td><td>0.00</td></tr><tr><td>Anatomy 4D</td><td>30</td><td>1</td><td>3.33</td><td>0</td><td>0.00</td><td>1</td><td>3.33</td></tr><tr><td>NASA</td><td>4161</td><td>8</td><td>0.19</td><td>6</td><td>0.14</td><td>32</td><td>0.77</td></tr><tr><td>Star Walk</td><td>584</td><td>12</td><td>2.05</td><td>3</td><td>0.51</td><td>22</td><td>3.77</td></tr></tbody></table>

Overall, a nuanced picture of pedagogical affordances emerges for these ten sample apps. MalMath produced considerably more positive sentiments in *Personalisation* and *Authenticity*, however there weren't many reviews about the app relating to *Collaboration*. Myscript Calculator has generated positive sentiments for the *Personalisation* and *Authenticity* constructs, but have received only negative reviews for *Collaboration*. In the case of Little Alchemy, it generated significantly positive reviews for app features relating to *Personalisation* and some positive reviews in *Collaboration*, however received low volume of reviews with mixed sentiments in *Authenticity*. NASA received positive sentiments for *Personalisation* but not so favourable sentiments in *Collaboration* and *Authenticity*.
<table><caption>**Table 4:** Breakdown of Feature based Sentiment Analysis (P=positive, Nt=Neutral, Ng=Negative)</caption><thead><tr><th rowspan="2">Apps</th><th colspan="3">Authenticity</th><th colspan="3">Collaboration</th><th colspan="3">Personlisation</th></tr><tr><th>P</th><th>Nt</th><th>Ng</th><th>P</th><th>Nt</th><th>Ng</th><th>P</th><th>Nt</th><th>Ng</th></tr></thead><tbody><tr><td>MalMath</td><td>22</td><td></td><td></td><td>4</td><td></td><td></td><td>37</td><td>1</td><td>2</td></tr><tr><td>Mathletics</td><td>1</td><td></td><td>7</td><td>1</td><td></td><td></td><td>1</td><td></td><td>3</td></tr><tr><td>Geogebra</td><td>1</td><td></td><td>1</td><td></td><td></td><td></td><td>2</td><td></td><td>1</td></tr><tr><td>Math: Mental Math Games</td><td>4</td><td></td><td></td><td></td><td></td><td>1</td><td>5</td><td>1</td><td></td></tr><tr><td>Myscript Calculator</td><td>17</td><td>1</td><td>1</td><td></td><td>2</td><td>1</td><td>23</td><td>3</td><td>5</td></tr><tr><td>Little Alchemy</td><td>6</td><td>3</td><td>4</td><td>25</td><td></td><td></td><td>78</td><td>4</td><td>6</td></tr><tr><td>Skeptical Science</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Anatomy 4D</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td></td></tr><tr><td>NASA</td><td>5</td><td></td><td>3</td><td>4</td><td>1</td><td>1</td><td>29</td><td>1</td><td>2</td></tr><tr><td>Star walk</td><td>9</td><td></td><td>3</td><td>1</td><td></td><td></td><td>11</td><td></td><td>3</td></tr></tbody></table>

Overall, a nuanced picture of pedagogical affordances emerges for these ten sample apps. MalMath produced considerably more positive sentiments in *Personalisation* and *Authenticity*, however there weren't many reviews about the app relating to *Collaboration*. Myscript Calculator has generated positive sentiments for the *Personalisation* and *Authenticity* constructs, but have received only negative reviews for *Collaboration*. In the case of Little Alchemy, it generated significantly positive reviews for app features relating to *Personalisation* and some positive reviews in *Collaboration*, however received low volume of reviews with mixed sentiments in *Authenticity*. NASA received positive sentiments for *Personalisation* but not so favourable sentiments in *Collaboration* and *Authenticity*.

<figure><img src="image_4.png" /><figcaption>**Figure 4:** Feature based Sentiment Analysis results. This provides the graphical representation of the feature based sentiment analysis results.</figcaption></figure>
## 5. Discussion

The results provide evidence of what mobile pedagogical features of apps that users are choosing to comment on in their reviews, without any prompting from rubric or other more formal evaluation instruments. There was a trend in our results showing higher frequencies of positive sentiments relating to the *Personalisation* aspect of the selected apps. Past work has revealed that of the three iPAC constructs, *Personalisation* is the least exploited by teachers in their mobile learning task designs [29], with teachers evidently struggling to give opportunities for learners to control their learning (e.g. the pace of lessons and how m-learning tasks are undertaken). So, in some ways the result from the present study is surprising. However, assuming that our results comprised comments mostly from students and teachers, and given that our study only used recent app reviews (1 year), perhaps the 'struggle' discussed in [29] is the very reason that users 'noticed' these pedagogical affordances (relating to personalisation) i.e. app reviewers were mindful of their previous mobile learning experiences that quite likely lacked a sense of learner control. This claim is entirely speculative and future research will need to triangulate these findings. This triangulation can be performed in two ways: (1) with interviews and surveys of app users, (2) qualitative analysis of the text of the reviews and feedback from app stores and social media (e.g. Facebook or Twitter).

It is too early to draw conclusions about ‘low volumes’ of comments, other than users were not choosing to comment on such features. For example, just because users were evidently not frequently commenting on (or not ‘noticing’) app features relating to *Collaboration*, doesn’t mean that these features are absent. Further research is needed to clarify the exact implications of low and high frequencies of sentiments. Informed by these results, we posit that sentiment analysis technique is a novel and effective augmentation of other more traditional app evaluation procedures. This type of innovative, two-tiered evaluation procedure will ultimately help educators (and app designers) to more accurately evaluate the pedagogical potential and value of education apps.

We also recognise the risk of deterministic views of emerging technologies [34] such as mobile apps, and we are certainly not advocating a ‘one-size-fits-all’ approach to selecting and using educational apps. There are many other factors (beyond pedagogical approaches) that contribute to the effective use of apps for learning, such as the teacher expertise, student characteristics and provision of technical support. However, there is value in teachers using procedures such as the one outlined in this paper to critically examine app features and their potential for leveraging transformational pedagogies [4] in and beyond the classroom.

## 6. Conclusion and Future Directions

In this paper, we have presented a preliminary investigation of exploring the utility of a new technique for evaluating the pedagogical affordances of educational apps. This technique uses feature based sentiment analysis approach to extract the feedback and comments of app users. We have used the results of sentiment analysis to assess their alignment against evaluation criteria from a rigorous mobile pedagogical framework
(iPAC) [13]. The main objective of our research was to explore the utility our novel technique incorporating sentiment analysis and informed by the iPAC pedagogical framework. The preliminary results of our investigation have provided empirical evidence that using sentiment analysis is an effective way of incorporating the opinions of past users of educational mobile apps with use of the iPAC framework. These forms of feedback are very useful for the authors who developed the iPAC framework [13] in their ongoing studies of the usefulness and utility of this framework.

In relation to threats to validity, we concede that the precision of our results is impacted by the accuracy of the word bank and the limitations of the sentiment analysis tool (appbot), so the words may not have matched well against those words used in the reviews even though they may have been synonyms. We plan to extend our study to include deeper semantic analysis of the textual content of the reviews by using cutting edge Natural Language Processing technologies as well as newly emerged algorithms for opinion mining. Another future direction is the integration of the sentiment analysis technique with the recently developed rubric instrument<fnref n="2" /> for app evaluation emerging from iPAC framework. It is recommended that teachers should ideally use this instrument after thoroughly exploring an app, and if possible, after using the app in their teaching. We plan to design a software tool that would seamlessly extract the sentiments of past users of an app to provide additional information about the app within this type of rubric instrument.

## References

1. Sung, Y.-T., K.-E. Chang, and T.-C. Liu, The effects of integrating mobile devices with teaching and learning on students' learning performance: A meta-analysis and research synthesis. Computers & Education, 2016. 94: p. 252-275.

2. Traxler, J., Current state of mobile learning. Mobile learning: Transforming the delivery of education and training, 2009. 1: p. 9-24.

3. Stevenson, M., et al., Visualizing Solutions: Apps as Cognitive Stepping-Stones in the Learning Process. Electronic Journal of e-Learning, 2015. 13(5): p. 366-379.

4. Highfield, K. and K. Goodwin, Apps for mathematics learning: A review of 'educational' apps from the iTunes App Store. Mathematics education: Yesterday, today, and tomorrow, 2013: p. 378-385.

5. El-Hussein, M.O.M. and J.C. Cronje, Defining mobile learning in the higher education landscape. Educational Technology & Society, 2010. 13(3): p. 12-21.

6. Goodwin, K., Use of tablet technology in the classroom. NSW Department of Education and Communities, 2012.

7. Murray, O.T. and N.R. Olcese, Teaching and learning with iPads, ready or not? TechTrends, 2011. 55(6): p. 42-48.

8. Peluso, D.C., The fast-paced iPad revolution: Can educators stay up to date and relevant about these ubiquitous devices? British Journal of Educational Technology, 2012. 43(4).

9. Bano, M. and D. Zowghi. Users' voice and service selection: An empirical study. in Empirical Requirements Engineering (EmpiRE), 2014 IEEE Fourth International Workshop on. 2014. IEEE.

<footnote n="2">² http://www.mobilelearningtoolkit.com/app-rubric1.html</footnote>
10. Guzman, E. and W. Maalej. How do users like this feature? a fine grained sentiment analysis of app reviews. in *Requirements Engineering Conference (RE), 2014 IEEE 22nd International*. 2014. IEEE.

11. Maalej, W. and H. Nabil. Bug report, feature request, or simply praise? on automatically classifying app reviews. in *Requirements Engineering Conference (RE), 2015 IEEE 23rd International*. 2015. IEEE.

12. Pang, B. and L. Lee, Opinion mining and sentiment analysis. *Foundations and Trends®* in *Information Retrieval*, 2008. 2(1-2): p. 1-135.

13. Kearney, M., et al., Viewing mobile learning from a pedagogical perspective. *Research in learning technology*, 2012, 20.

14. Koole, M.L., A model for framing mobile learning. *Mobile learning: Transforming the delivery of education and training*, 2009. 1(2): p. 25-47.

15. Wu, W.-H., et al., Review of trends from mobile learning studies: A meta-analysis. *Computers & Education*, 2012, 59(2): p. 817-827.

16. Crompton, H. and D. Burke, *Research Trends in the Use of Mobile Learning in Mathematics*, in *Blended Learning: Concepts, Methodologies, Tools, and Applications*. 2015, IGI Global. p. 2090-2104.

17. Crompton, H., et al., The use of mobile learning in science: a systematic review. *Journal of Science Education and Technology*, 2016, 25(2): p. 149-160.

18. Levin, T. and R. Wadmany, Teachers' views on factors affecting effective integration of information technology in the classroom: Developmental scenery. *Journal of Technology and Teacher Education*, 2008. 16(2): p. 233.

19. Groff, J. and C. Mouza, A framework for addressing challenges to classroom technology use. *AACe Journal*, 2008. 16(1): p. 21-46.

20. Goktas, Y., S. Yildirim, and Z. Yildirim, Main barriers and possible enablers of ICTs integration into pre-service teacher education programs. *Educational Technology & Society*, 2009. 12(1): p. 193-204.

21. Cherner, T., J. Dix, and C. Lee, Cleaning up that mess: A framework for classifying educational apps. *Contemporary Issues in Technology and Teacher Education*, 2014. 14(2): p. 158-193.

22. Lee, C.-Y. and T.S. Cherner, A comprehensive evaluation rubric for assessing instructional apps. *Journal of Information Technology Education: Research*, 2015. 14: p. 21-53.

23. Parsons, D., H. Ryu, and M. Cranshaw, A design requirements framework for mobile learning environments. *JCP*, 2007. 2(4): p. 1-8.

24. Wertsch, J.V., Mind as action. 1998.

25. Pachler, N., B. Bachmair, and J. Cook, *Mobile learning: structures, agency, practices*. 2009: Springer Science & Business Media.

26. Radinsky, J., et al., Mutual benefit partnership: A curricular design for authenticity. *Journal of curriculum studies*, 2001. 33(4): p. 405-430.

27. Burden, K. and M. Kearney, Conceptualising authentic mobile learning, in *Mobile Learning Design*. 2016, Springer. p. 27-42.

28. Wang, M. and R. Shen, Message design for mobile learning: Learning theories, human cognition and design principles. *British Journal of Educational Technology*, 2012. 43(4): p. 561-575.
29. Kearney, M., K. Burden, and T. Rai, Investigating teachers' adoption of signature mobile pedagogies. *Computers & Education*, 2015. 80: p. 48-57.

30. Kearney, M. and D. Maher, Mobile learning in maths teacher education: Using iPads to support pre-service teachers' professional development. *Australian Educational Computing*, 2013. 27(3): p. 76-84.

31. Townsend, P., Mobile Devices for Tertiary Study–Philosophy Meets Pragmatics for Remote Aboriginal and Torres Strait Islander Women. *The Australian Journal of Indigenous Education*, 2015. 44(02): p. 139-149.

32. Kinash, S., J. Brand, and T. Mathew, Challenging mobile learning discourse through research: Student perceptions of Blackboard Mobile Learn and iPads. *Australasian journal of educational technology*, 2012. 28(4).

33. Viberg, O. and Å. Grönlund, Cross-cultural analysis of users' attitudes toward the use of mobile devices in second and foreign language learning in higher education: A case from Sweden and China. *Computers & Education*, 2013. 69: p. 169-180.

34. Selwyn, N., Looking beyond learning: Notes towards the critical study of educational technology. *Journal of Computer Assisted Learning*, 2010. 26(1): p. 65-73.

35. Burden, K. & Kearney, M. Investigating and critiquing teacher educators’ mobile - learning practices. *Interactive Technology and Smart Education*, 2017. 14(2), 110-125