# Adopting Learning Analytics to Inform Postgraduate Curriculum Design

Denis Dennehy, Kieran Conboy, Jaganath Babu, Johannes Schneider, Joshua

Handali, Jan Vom Brocke, Benedikt Hoffmeister, Armin Stein

► To cite this version:

Denis Dennehy, Kieran Conboy, Jaganath Babu, Johannes Schneider, Joshua Handali, et al. Adopting Learning Analytics to Inform Postgraduate Curriculum Design. International Working Conference on Transfer and Diffusion of IT (TDIT), Dec 2020, Tiruchirappalli, India. pp.218-230, 10.1007/978-3-030-64849-7_20. hal-03701770

HAL Id: hal-03701770

https://inria.hal.science/hal-03701770

Submitted on 22 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Adopting Learning Analytics to Inform Postgraduate Curriculum Design

Denis Dennehy<fnref n="1" />, Kieran Conboy<fnref n="1" />, Jaganath Babu<fnref n="1" />, Johannes Schneider<fnref n="2" />, Joshua Handali<fnref n="2" />, Jan vom Brocke<fnref n="2" />, Benedikt Hoffmeister<fnref n="3" />, Armin Stein<fnref n="3" />

**Abstract.** Understanding students’ sentiment is valuable to understanding the changes that could or should be made in curriculum design at third level. Learning analytics has shown potential for improving student learning experiences and supporting teacher inquiry. Yet, there is limited research that reports on the adoption and actual use of learning analytics to support teacher inquiry. This study captures sentiment of postgraduate students by integrating learning analytics with the steps of teacher inquiry. This study makes two important contributions to teaching and learning literature. First, it reports on the use of learning analytics to support teacher inquiry over three iterations of a business analytics programme between 2016 and 2019. Second, evidence-based recommendations on how to optimise learning analytics to support teacher inquiry are provided.

**Keywords:** Learning Analytics, Sentiment Analysis, Curriculum Design.

## 1 Introduction

Students’ sentiment have increasingly received attention by researchers in higher education because understanding sentiment is valuable to understanding the changes that could or should be made in curriculum design [1, 2]. Students’ sentiment is recognised as an integral part of student learning and a critical element in the learning process [3, 4]. Sentiment is closely intertwined with students’ motivations and strategies for learning, and self-regulation [5-7]. Sentiment has a considerable impact on students’ performance and it influences students’ academic achievements [6]. There are concerns, however, that research on student sentiment in higher education is disjointed [6]. A focus on the perspectives of students is essential to the development of analytics related to their needs, rather than to the needs of institutions [8].

Yet, third level education, ‘a field that gathers an astonishing array of data about its “customers,” has traditionally been inefficient in its data use, often operating with substantial delays in analyzing data and feedback’ [9]. Learning analytics has emerged as an area with high potential for improving student learning experiences

<footnote n="1">¹NUI Galway, Galway, Ireland</footnote>

<footnote n="2">²University of Liechtenstein, Vaduz, Liechtenstein</footnote>

<footnote n="3">³University of Münster, Münster, Germany</footnote>
and curriculum design [4, 8]. Learning analytics involves the use of "analytic techniques integrated with learning outcomes assessment to better understand student learning and more efficiently and meaningfully target instruction, curricula and support" [10 p. 2]. Integrating learning analytics with teacher inquiry has been identified as critically important [11, 12]. There is however, a scarcity of research that examines the adoption and use of learning analytics to support teacher inquiry [13-15].

The overarching aim of this study is to 'explore how adopting learning analytics can be used to understand students' sentiment and to use this understanding to inform curriculum design.'

Understanding how students' sentiment can inform teacher inquiry is important for two key reasons. *First*, a systematic literature review on the use of learning analytics by [15] concluded that "little research attention has been placed on 'providing recommendations to educators for translating the analysed data to actionable actions on their educational design and delivery' (p.20)." We provide such recommendations, which is important because the process of obtaining actionable insights for curriculum design is generally considered to be a time-consuming activity for educators [14-16]. *Second*, although learning analytics has been traditionally applied to understand and optimise the learning process at module level, it can also be used to understand and optimise learning at the program level [17]. This study adopts analytical techniques to understand the learning process at programme level.

The remainder of this paper is structured as follows. First, theoretical background to learning analytics and teacher inquiry is presented. Next, the research method and background to the case studied is provided. Then, the findings and analysis are presented, followed by discussion and recommendations. The paper ends with a conclusion, limitations, and future research.

## 2 Theoretical Background

Academic analytics refers to the use of analytics within academic settings and may be applied at the level of the institution, the department, or the learner, depending on the goals and objectives of the analysis [1, 18]. The term 'analytics' holds different meaning for people across the various academic departments and business units, as well as their use of analytics within the university. Nevertheless, the use of analytics in education must be used to transform curriculum design [9].

Academic analytics consist of two types of applied analytics called 'institutional analytics' and 'learning analytics' [1]. *Institutional analytics* is generally used to understand factors that relate to running the business of the university i.e. predicting student success and retention rates [19]. *Learning analytics* focuses specifically on students and their learning behaviors [9, 18]. Learning analytics defined as "*the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs*" [8]. This definition includes techniques such as predictive modeling, social network analysis, concept analysis, and sentiment analysis
[20]. [9] outline the differences between academic analytics and learning analytics (see Table 1).

<table><caption>**Table 1.** Differences between academic and learning analytics</caption><thead><tr><th>Type</th><th>Level or object of analysis</th><th>Who benefits?</th></tr></thead><tbody><tr><td rowspan="3">Academic analytics</td><td>Institutional: learner profiles, performance of academics, knowledge flow</td><td>Administrators, funders, marketing</td></tr><tr><td>Regional (state/provincial): comparisons between systems</td><td>Funders, administrators</td></tr><tr><td>National and international</td><td>Governments, education authorities</td></tr><tr><td rowspan="2">Learning analytics</td><td>Course-level: social networks, sentiments, discourse analysis</td><td>Learners, faculty</td></tr><tr><td>Departmental: predictive modeling, patterns of success/failure</td><td>Learners, faculty</td></tr></tbody></table>

The link between learning analytics and curriculum design is based on the premise that data collection and analysis is conducted at various levels (i.e. module, programme) to inform the; (i) learning experience, (ii) design process, and (iii) community of curriculum designers [21].

## 2.1 Mapping teacher inquiry with learning analytics

Teacher inquiring is defined as a cyclical process in which “teachers identify questions for investigation in their practice and then design a process for collecting evidence about student learning that informs their subsequent educational designs” [22pp. 249-250]. Teacher inquiry is a process that can guide reflection and enhancements in a systematic and evidence-based approach [23]. Despite the critical importance of integrating learning analytics with teacher enquiry [12, 24], there is limited research that reports on the actual use of learning analytics to support teacher enquiry [13, 15, 24]. Most concerning is that learning analytics are increasingly being implemented in different educational settings, often without the guidance of a research base [20]. There is also a scarcity of research on how to analyze the learning process at the program level in order to guide the design or redesign of a program [25].

Research does suggest, however, that educators may lack the skills to formulate questions and identify solutions [26, 27] or they be unable to make sense of the data in order to inform curriculum redesign [26, 28, 29]. While many analytics studies identify patterns in students' learning behaviour, understanding of the pedagogical context that influences student activities is lacking [12, 30]. A related issue is the need to use the insights generated from the use of learning analytics to make interventions to improve learning and generate ‘actionable insights’ [31, 32].

Evidence-based insights generated by learning analytics provides support for educators to reflect on and improve curriculum design and delivery [15, 24, 33].
Therefore, it supports the concept of teacher inquiry [14] and can be linked to the teacher inquiry cycle [15]. Table 1 maps learning analytics with the steps of teacher inquiry. To the best of our knowledge, this is the first study to integrate sentiment analytics, a form of learning analytics, with teacher inquiry.

<table><caption>**Table 2.** Mapping learning analytics with the steps of teacher inquiry</caption><thead><tr><td>Step</td><td>Teacher inquiry cycle</td><td>Description [15]</td><td>Mapping learning analytics with teacher inquiry</td><td>Literature sources</td></tr></thead><tbody><tr><td>1</td><td>Problem identification</td><td>Identification of a specific aspect of educational design (i.e. module, programme) and/or delivery to be evaluated in order to improve it</td><td>Can be used to measure, collect, analyse and report on students' learning experience and the context of their learning</td><td>[8, 34]</td></tr><tr><td>2</td><td>Develop inquiry questions</td><td>Specific questions, data to be collected, and the method for data collection is established</td><td>Can be used to identify specific problems related to the module/programme</td><td>[14, 15]</td></tr><tr><td>3</td><td>Educational design</td><td>Formulation of educational design in which the teacher will deliver in order to implement their inquiry.</td><td>Can be used to improve the learning experience for individual learners or groups of learners</td><td>[8, 24, 34]</td></tr><tr><td>4</td><td>Deliver educational design and collect data</td><td>Delivering the educational design to the learners and collects the educational data using the collection method</td><td>Can be used to collect the educational data that have been defined to answer their inquiry question</td><td>[15, 34]</td></tr><tr><td>5</td><td>Analyse educational data</td><td>Data is analysed in order to elicit insights to answer the inquiry questions</td><td>Can be used to analyse and report on the collected data and facilitate sense-making</td><td>[15, 23, 34]</td></tr><tr><td>6</td><td>Reflect on data</td><td>The analysed data are used in order to answer the defined inquiry question and revise the practice in which the educational design and/or delivery is practiced</td><td>Can be used as an evidence-based approach to guide reflection</td><td>[23, 24, 33]</td></tr></tbody></table>

## 3 Research Method

The research described in this paper follows the principles of a case study method [35], focused on postgraduate students completing a one-year fulltime masters programme in business analytics at a university in Ireland. The specific case was
purposefully chosen because, (i) reputation of the course was important as it is the largest masters programme in business analytics in Ireland, (ii) monitoring student sentiment was critical as the programme underwent significant growth with annual student enrolment, increasing from 12 to 102 students within 3-years, (iii) ensuring the course was designed for inclusive teaching was important as a diverse student population (i.e. 8 different nationalities, varying academic backgrounds), and (iv) the researchers had continuous direct access to the students and alumni.

*Programme Learning Outcomes:* The following learning outcomes are intended to equip students with the required industry-standard skills and knowledge: (A) Understand and be able to use specific IT which is used in developing business analytics. (B) Analyse and solve business problems using applied data analytics tools and techniques. (C) Understand and apply techniques for managing IT in organisations. (D) Identify, analyse and solve applied problems in individual and team-based settings. (E) Apply effective data-driven decision-making to global business and social problems.

*Programme Growth:* The programme commenced in the 2015-16 academic year with 12 students and has increased student enrolment for the subsequent academic years, namely, 2016-17: 36 students, 2017-18: 57 students, and 2018-19: 102 students. Students from Ireland, India, UK, France, Pakistan, Nigeria, China, USA, Germany, and Malaysia are represented on the programme.

## 3.1 Data collection and analysis

Cross Industry Standard Process for Data Mining (CRISP-DM) is an industry standard methodology that prescribes a set of guidelines to guide the efficient extraction of information from data (Chapman et al., 2000; Shearer, 2000). The CRISP-DM methodology consists of six cyclical steps, namely (i) Business Understanding, (ii) Data Understanding, (iii) Data Preparation, (iv) Modeling, (v) Evaluation, and (vi) Deployment. We adapt this process methodology to suit the context of our research, which includes four cyclical phases (see Fig. 1). The adapted model (Fig. 1) does not exclude any of the six phases of CRISP-DM, instead, it merges them into four inter-related activities, namely, (i) Business and Data Understanding, (ii) Modeling, (iii) Evaluation, and (iv) Actionable Intelligence.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Process of data preparation and sentiment analysis</figcaption></figure>
**Business and data understanding:** In this phase, business understanding focused on the context, aim and business problem in order to align with the project objectives, and data understanding provided an understanding of the data that needed to be analysed, identify potential issues (i.e. quality) and prepare for modeling. Input data comprised of (i) three years of module data (e.g. 12 modules per year), (ii) two end of year programme reviews, and (iii) interview data.

**Data preparation:** This phase included deciding what needed to be included in the dataset, cleaning the data and all other activities that needed to be done to process data which served as an input to the modeling tool in the next step. Data extraction and integration using Python scripts whereby messages were converted from RAR file format into .CSV file format. Text was then converted into Pandas DataFrame format for compatibility purposes with the sentiment analysis algorithm.

**Modeling:** Essentially, this phase performed sentiment analysis across three consecutive academic years, namely, 2016-17, 2017-18, and 2018-19. To ensure high response rates, all responses were anonymised. The response rate for each end of year programme review was 72% (2016-17), 96% (2017-18), and 70% (2018-19). As the response rate varied across the academic years, a number of analytical techniques were applied to calculate an overall rating scale of 0 to 5. Zero being the lowest overall score the programme could receive and five been the highest rating.

**Evaluation:** In this iterative phase, the model, data, and emerging findings were analysed in relation to the business and data understanding (e.g. learning outcomes). This involved regular meetings with staff and the research team. This iterative process ensured that the emerging findings and ‘actionable intelligence’ supported the business understanding (e.g. to inform curriculum design of the programme). These findings are presented in the next section.

# 4 Findings and Analysis

The findings presented in this section are intended to provide insight of how student sentiment influenced curriculum redesign rather than compare teaching staff. The end of year programme review for 2016-2017 was the starting point of our empirical analysis as (i) this was the first programme review conducted since the programme commenced, (ii) the programme review was conducted by the newly appointed programme director, and (iii) this dataset provided a baseline from which to establish student sentiment and to identify potential curriculum design issues that can be monitored in subsequent reviews. First, we were keen to understand if students were aware of the learning outcomes of the programme (Q1), if the programme delivered the expected learning outcomes (Q2), if the assessment and examination requirements were clearly communicated (Q3), and if the modules on the programme were linked effectively (Q4). The sentiment for each of these questions is presented in Fig 2. There was concern about disconnect between stated (see purple lines in Fig. 2) and realised learning outcomes (see yellow lines in Fig. 2).
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Sentiment of learning outcomes and assessment</figcaption></figure>

To gain deeper insight of the ratings identified from the baseline data, engagement with students was necessary in order to distinguish if there was a recurring pattern relating to curriculum design issues or if it was unique to the 2016-17 class. Engagement with students revealed that the majority of students did not distinguish between programme and module learning outcomes (Q1, Q2) and many students admitted that they did not know the programme learning outcomes or where to find them.

A number of initiatives were implemented that has positively increased student sentiment for the 2018-19 academic year (see Fig. 4). These included (i) a standard template was designed for module descriptions with no more than five learning outcomes linked to a module, (ii) learning outcomes were based on Bloom's taxonomy, (iii) learning outcomes of the programme and module descriptions with the associated learning outcomes were made available on the college website, and (iv) the programme learning outcomes were incorporated into the programme orientation. The impact of these curriculum changes had a positive impact on student sentiment (see Fig. 3). Using the 2016-17 programme as a baseline rate of 3.89 out of 5, sentiment increased to 4.02 in 2018-19 academic year. While sentiment for the overall programme rating moved in a positive direction (see amber square in Fig. 3.), there was a concern that students (2016-17, 2017-18) did not find the programme intellectually stimulating (Q7). The response to this question is presented in Fig. 3 (see purple circles and lines).

Students also reported that they did not receive helpful and timely feedback during the programme (Q8). This was surprising considering there were only 36 students in the 2016-17 cohort. Yet, this sentiment remained the same (25 out of 50 points) for the subsequent academic year. Interviews with students indicated that students were unable to identify when educators were providing 'formative' assessment. Staff now explicitly inform students when they are providing formative assessment, and this had a considerable impact on student sentiment in the 2018-19 academic year, despite growth in student enrolment (see yellow circles in Fig. 3).
<figure><img src="image_4.png" /><figcaption>Fig. 3. Annual sentiment analysis</figcaption></figure>
Fig. 4 presents the sentiment trend over three academic years. To help students 'connect the dots' between modules and to get a new perspective on threshold concepts, the programme director initiated and supported a number of curriculum design changes. These included, inviting industry experts, setting up a business analytics society, and appointing an Honorary Professorship to Bill Schmarzo, Chief Innovation Officer, Hitachi Vantara. Schmarzo visits the university to deliver a number of workshops and lectures in order to demonstrate how his data science teams apply 'Design Thinking' in the context of business analytics.

The impact of inviting industry guest speakers (Q13) has positively improved student sentiment (see purple circles in Fig. 4). New module changes were informed by student feedback and used to influence staff responsible for designing and approving new modules.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Sentiment trend over three academic year</figcaption></figure>

Although curriculum design changes will continue to be implemented, these changes provide students with the right content and supports that will enable them to shift from 'surface learning' to 'deep learning' of the fundamentals of business analytics.

# 5 Discussion and Recommendations

As there is a lack of research that examines the value of learning analytics to support curriculum design [13-15], there is a risk that learning analytics is not used appropriately and thereby, its real value not realized. Our study showed that adopting and integrating learning analytics with teacher inquiry, advantageously informed curriculum redesign. The following inter-related recommendations are intended to support educators to realise the value of learning analytics in the context of curriculum design, as well as to provide a more positive student learning experience.
*Creating a learning analytics culture:* Support educators to adapt, apply, and integrate learning analytics into their teacher inquiry. This implies that educators will require training in the use of analytical tools and analysis of data. Tailored training is important as [36] highlights that learning analytics solutions that do not incorporate diverse “alternates for action” might not achieve the desired results for students and educators.

*Using learning analytics in context:* Understanding the contextual factors of teaching and learning is critical when determining curriculum changes, rather than sole relying on learning analytics. While learning analytics can be used to support inclusive teaching and learning, it should be used as part of a suite of tools and methods rather than be used in isolation. Learning analytics should also be used to support students to develop their critical thinking and problem solving through the process of reflecting and acting on data, rather than simply a tool to generate evidence for quality assurance [37].

*Establishing of baseline learning analytics:* As student feedback can be emotive, it is critical that educators establish baseline metrics from which to their build analytic capabilities, and over time, identify patterns and trends, rather than prematurely acting on negative and positive feedback. Establishing a baseline has been identified as a useful indicator for progress in other studies [cf. 38].

*Inclusive learning analytics:* Educators need to design curriculum that will facilitate the learning of a more diverse group of learners. This implies we need to value what individual students bring to the curriculum design process. Specifically, while inclusion in information systems has received significant attention in recent years [c.f. 39-40], research on inclusion within IS curriculum design and delivery has not received sufficient attention.

*Differentiating features of sentiment data:* This study showed that sentiment analysis adds data points and information that adds different value to other types of information from and on students and their learning. Sentiment analysis can sense issues the students themselves may not even be aware of or know how to articulate themselves through the traditional survey. Traditional surveys are limited in that they only elicit what the survey designer asks, and so may miss crucial issues or issues that emerge after the survey was designed. Sentiment analysis can track emerging behaviours and use of keywords in an organic and grounded manner. However, we recommend that lecturers and educators consider these differences, use these instruments accordingly, and ensure they consider these differences when acting on the emerging sentiment feedback.

We believe that a real contribution of these recommendations is the ease at which they can be tailored and applied to other educational contexts.

# 6 Conclusions, Limitations, and Future Research

Learning analytics is an emerging research field that aims to support educators during the process of inquiry. This study reported the value of using sentiment analytics as a form of learning analytics to improve student-learning experiences and inform
curriculum design. Sentiment analytics offers a dynamic and evidence-based approach to guide teacher inquiry and inform curriculum design. However, it assumes that educators have the ability to use these types of analytical tools and techniques and align these with their teacher inquiry. This is most likely not the case in many universities, due to a range of factors including, (i) the capacity of the discipline, (ii) availability of funding, (iii) tailored training in the use of learning analytics, and, (iv) continuous support in the use of learning analytics and curriculum design.

We acknowledge that the recommendations provided are not exhaustive but they do however contribute to the wider discourse on the need for more academic research that provides recommendations (c.f. Sergis and Sampson [15] to educators in order to maximise the use of sentiment analytics, and other learning analytics tools and techniques.

Indeed, all research has limitations and we acknowledge three limitations of this research. First, conventional textual sentiment analysis was not conducted due to limited data points, making it difficult for the reliable predictions for open-ended questions. Second, the findings are based on a single case which by nature, limits generalisability. However, the data gathered was based on three iterations (e.g. within case analyses) of a one-year master's programme and in-depth background to the case studied and rich contextual data was provided, which can help readers to relate the findings to their own educational context. Third, while learning analytics has become increasingly popular, it is only one approach to inform curriculum design. It should, therefore, not be used in isolation but rather to complement other data sources (i.e. academic analytics) and the knowledge possessed by educators and curriculum designers.

In terms of next steps, future researchers could (i) study the sentiment of students across multiple postgraduate programmes within the same university or across universities for the purpose of generalisability, and (ii) combine sentiment analytics with academic analytics for the purpose of policy changes relating to teacher inquiry and curriculum design.

## References

1. Dunbar, R.L., M.J. Dingel, and X. Prat-Resina, *Connecting Analytics and Curriculum Design: Process and Outcomes of Building a Tool to Browse Data Relevant to Course Designers*. Journal of Learning Analytics, 1(3), pp.223-243. 2014.

2. Baxter Magolda, M.B., *Identity and learning: Student affairs' role in transforming higher education*. Journal of College Student Development, 44(2), pp.231-247. 2003.

3. Linnenbrink-Garcia, L. and R. Pekrun, *Students' emotions and academic engagement: Introduction to the special issue*. Contemporary Educational Psychology, 36(1), pp.1-3. 2011.

4. Henritius, E., E. Löfström, and M.S. Hannula, *University students' emotions in virtual learning: a review of empirical research in the 21st century*. British Journal of Educational Technology, 50(1), pp.80-100. 2019.
5. Mainhard, T., et al., *Student emotions in class: The relative importance of teachers and their interpersonal relations with students*. Learning and Instruction, 53, pp.109-119. 2018.

6. Mega, C., L. Ronconi, and R. De Beni, *What makes a good student? How emotions, self-regulated learning, and motivation contribute to academic achievement*. Journal of educational psychology, 106(1), p.121. 2014.

7. Pekrun, R., et al., *Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research*. Educational psychologist, 37(2), pp.91-105. 2002.

8. Ferguson, R., *Learning analytics: drivers, developments and challenges*. International Journal of Technology Enhanced Learning, 4(5/6), pp.304-317. 2012.

9. Siemens, G. and P. Long, *Penetrating the fog: Analytics in learning and education*. EDUCAUSE review, 46(5), p.30. 2011.

10. Bach, C., *Learning analytics: Targeting instruction, curricula and student support*. Office of the Provost, Drexel University. 2010.

11. Bos, N. and S. Brand-Gruwel, *Student differences in regulation strategies and their use of learning resources: Implications for educational design*. In *Proceedings of the Sixth International Conference on Learning Analytics & Knowledge* (pp. 344–353). New York: ACM. 2016.

12. Lockyer, L., E. Heathcote, and S. Dawson, *Informing pedagogical action: Aligning learning analytics with learning design*. American Behavioral Scientist, 57, 1439–1459. 2013.

13. Dyckhoff, A.L., et al., *Supporting action research with learning analytics*. In *Proceedings of the Third International Conference on Learning Analytics and Knowledge* (pp. 220–229). New York: ACM. 2013.

14. Mor, Y., R. Ferguson, and B. Wasson, *Learning design, teacher inquiry into student learning and learning analytics: A call for action*. British Journal of Educational Technology, 46, 221–229. 2015.

15. Sergis, S. and D.G. Sampson, *Teaching and learning analytics to support teacher inquiry: A systematic literature review*. In A. Pe-na-Ayala (Ed.), *Learning analytics: Fundaments, applications, and trends* (pp. 25–63). Berlin, Germany: Springer. 2017.

16. Marsh, J.-A. and C.-C. Farrell, *How leaders can support teachers with data-driven decision making: A framework for understanding capacity building*. Educational Management Administration & Leadership:1-21. 2014.

17. Ochoa, X., *Simple metrics for curricular analytics*. In *Proceedings of the 1st Learning Analytics for Curriculum and Program Quality Improvement Workshop*. 2016.

18. Van Barneveld, A., K.E. Arnold, and J.P. Campbell, *Analytics in higher education: Establishing a common language*. EDUCAUSE learning initiative, 1(1), pp.1-ll. 2012.

19. Oblinger, D.G., *Let's Talk... Analytics*. EducausE Review, 2012. 47(4): p. 10-13.

20. Siemens, G., *Learning analytics: envisioning a research discipline and a domain of practice*. In *Proceedings of the 2nd international conference on learning analytics and knowledge* (pp. 4-8). 2012.
21. Hernández-Leo, D., et al., Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), pp.139-152. 2019.

22. Avramides, K., et al., A method for teacher inquiry in cross-curricular projects: Lessons from a case study. British Journal of Educational Technology, 46(2):249-264. 2015.

23. Dana, N. and D. Yendol-Hoppey, The reflective educator's guide to classroom research: Learning to teach and teaching to learn through practitioner inquiry. Corwin Press, London. 2014.

24. Bakharia, A., et al., A conceptual framework linking learning design with learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 329-338). New York: ACM. 2016.

25. Méndez, G., X. Ochoa, and K. Chiluiza, Techniques for data-driven curriculum analysis. In Proceedings of the fourth international conference on learning analytics and knowledge (pp. 148-157). 2014.

26. Olah, L., N. Lawrence, and M. Riggan, Learning to learn from benchmark assessment data: How teachers analyze results. Peabody Journal of Education 85(1): 226-245. 2010.

27. Means, B., et al., Teachers' Ability to Use Data to Inform Instruction: Challenges and Supports. Office of Planning, Evaluation and Policy Development, US Department of Education. 2011.

28. Heritage, M., et al., From evidence to action: A seamless process in formative assessment?. Educational Measurement: Issues and Practice, 28(3), pp.24-31. 2009.

29. Young, V. and D. Kim, Using assessments for instructional improvement: A literature review. Education Policy Analysis Archives, 18(19). http://epaa.asu.edu/ojs/article/view/809 (accessed Feb 2020). 2010.

30. Gašević, D., et al., Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, pp.68-84. 2016.

31. Campbell, F., et al., Hearing the student voice: Final report. HEA, Subject Centre for Education, Napier University. Retrieved from http://dera.ioe.ac.uk/13053/2/3911.pdf. 2007.

32. Clow, D., The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 134-138). 2012.

33. Greller, W., M. Ebner, and M. Schön, Learning Analytics: From Theory to Practice-Data Support for Learning and Teaching. In M. Kalz, R. Marco (Eds.) Computer Assisted Assessment. Research into E-Assessment, Springer International Publishing, pp. 79-87. 2014.

34. Papamitsiou, Z. and A. Economides, Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4):49-64. 2014.

35. Yin, R.K., Case study research: Design and methods. Vol. 5. 2009: Sage publications, INC.
36. Vatrapu, R. (2011, February). Cultural considerations in learning analytics. In *Proceedings of the 1st International Conference on Learning Analytics and Knowledge* (pp. 127-133).

37. Tsai, Y. S., Poquet, O., Gašević, D., Dawson, S., & Pardo, A. (2019). Complexity leadership in learning analytics: Drivers, challenges and opportunities. *British Journal of Educational Technology, 50*(6), 2839-2854.

38. Rehrey, G., Shepard, L., Hostetter, C., Reynolds, A. M., & Groth, D. (2019). Engaging faculty in learning analytics: Agents of institutional culture change. *Journal of Learning Analytics, 6*(2), 86-94.

39. Coleman, E., Carter, M., Davison, R. M., Chigona, W., & Urquhart, C. (2017). Social Inclusion in the AIS Community: What, Why and How?

40. Trauth, E. (2017). A research agenda for social inclusion in information systems. *ACM SIGMIS Database: The Database for Advances in Information Systems, 48*(2), 9-20.