# Exploring Associations Between Participant Online Content Engagement and Outcomes in an Online Professional Development Programme

Ketan S. Deshmukh, Vijaya Sherry Chand, Kathan D. Shukla, Arnab K. Laha

► To cite this version:

Ketan S. Deshmukh, Vijaya Sherry Chand, Kathan D. Shukla, Arnab K. Laha. Exploring Associations Between Participant Online Content Engagement and Outcomes in an Online Professional Development Programme. International Working Conference on Transfer and Diffusion of IT (TDIT), Dec 2020, Tiruchirappalli, India. pp.126-136, 10.1007/978-3-030-64849-7_12. hal-03701779

HAL Id: hal-03701779

https://inria.hal.science/hal-03701779

Submitted on 22 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Exploring associations between participant online content engagement and outcomes in an Online Professional Development Programme

Ketan S. Deshmukh¹[0000-0003-0184-3870], Vijaya Sherry Chand¹, Kathan D. Shukla¹ and Arnab K. Laha¹

¹ Indian Institute of Management Ahmedabad, Ahmedabad Gujarat 380015, India
ketand@iima.ac.in

**Abstract.** Online Professional Development (PD) programmes for government school teachers provide benefits of low costs to the administration and flexible schedules for the participants. However, research on the use of technology in PD programmes has reported mixed results, thus warranting further investigation. Exploring the associations between the variation in engagement of and outcomes among the participants may provide insights for future research. The paper presents analysis of pageview logs and survey responses of 6933 participants of an online PD programme. First, four latent online engagement profiles were extracted using mixture modelling. Then, associations between participants’ latent profiles and reported change in self-efficacy beliefs were analyzed. Finally, limitations and implications of the work are presented.

**Keywords:** Online Professional Development, Mixture Modelling, Latent Profile Analysis, Self-Efficacy Beliefs.

## 1 Introduction

The benefits of low cost and flexible schedules made it attractive and viable to adopt information technology to facilitate professional development (PD) programmes [1], [2]. The advances in data gathering and analysis, in recent times, provided precise tools to answer the “what works?” question for in-service teacher’s online PD. We present the results of an exploratory study of online learner’s engagement patterns and their association with outcomes within the context of an online professional development programme for teachers.

The next section (Section 2) provides the background of study viz. teacher professional development programme. Next, we present the method in section 3 and present the findings in section 4. Subsequently, we proceed to discuss these findings in section 5 and highlight the limitations of the work in section 6. Finally, we mention the implications of the study in section 7, followed by the acknowledgment section (Section 8).
## 2 Background

Teacher's PD programmes improve implementation of education policy and student outcomes [3], [4]. Many studies have indicated that improving teacher's self-efficacy beliefs is a desirable objective for PD programmes [5]–[8]. Teaching self-efficacy belief is defined as "the judgement of one's capabilities to influence student engagement and learning" [9]. Improving teacher's self-efficacy beliefs is associated with higher student achievement [9]–[11] and effective classroom practice [12]–[16].

Facilitating a PD programme for teachers involves huge costs, which can be reduced by adoption of technology. Evaluation studies of technology-based PD have reported mixed results [17]–[19], warranting additional research on outcomes of technology-based PD. Studies have not only found that participants in the same PD vary in their preference for and access to the different components of the programme [20], [21], but also found variation in outcomes for participants attending the same programme [22]. One of the affordances of technology-based PD is the ability to track and analyze participants engagement with the PD content. Tracking learners enables identification of specific content or activities which improve participant outcomes.

The prominent field that deals with studying a learner's engagement/participation in a programme is Educational Data Science (EDS). EDS grew due to Massive Open Online Courses (MOOCs) which generated huge amounts of data on participant's interaction with educational content [23], [24]. EDS is constituted of various sub-fields like Learning Analytics, Educational Data Mining, Institutional/ Academic Analytics, etc. [23]. Researchers who studied the patterns of participant engagement in MOOCs have classified them into either four [25] or seven [26] groups. Analyzing patterns of participant engagement one could identify areas of improvement in courses [27], design courses that accommodate different learning styles [28] and understand how to retain learners till the end of the course [29]. Kalakoski, Ratilainen, and Drupsteen [30] proposed the use of EDS methods to improve job training for professionals. Our study intends to identify groups of learners with similar engagement patterns and determine if the change in participant's self-efficacy is associated with their engagement pattern.

## 3 Method

### 3.1 Context

An online PD programme was offered to all teachers (19,267) in the state teaching science and mathematics at upper primary level in state-run primary schools. The training was offered in two batches and teachers were randomly assigned to one of the two batches. This article deals with the outcomes and engagement profiles of the participants in the first batch. In the first batch, a total of 10535 had registered and filled the pre-training survey of which 7935 participants completed the full programme and then completed the post-training survey. The course content consisted of Subject specific content (Science & Mathematics) and Managerial content (Classroom Management, School Comprehensive Evaluation (SCE) and Use of ICT). Content for the PD was
from two sources, subject matter expert and authentic case studies. The teachers were required to execute a real-world project in classroom, which was peer evaluated by other participants on the platform.

## 3.2 Self-Efficacy Belief

Participant's self-efficacy beliefs were measured via online surveys when they registered for the programme in May - June 2018 and later once they completed it. As self-efficacy beliefs are subject specific, teachers' self-efficacy beliefs about teaching Science and Mathematics were captured using instruments Science Teaching Efficacy Belief Instrument (STEBI) [31] and Mathematics Teaching Efficacy Belief Instrument (MTEBI) [32]. The subject specific self-efficacy scales each consist of two sub-constructs the personal subject teaching efficacy belief and the subject teaching outcome expectancy beliefs. In this article, we will be analyzing participant responses to only the personal science teaching efficacy belief. The personal science teaching efficacy belief subscales measure the teacher's own efficacy belief about teaching science using items like "I generally teach science effectively" and "I find it easy to explain to students why science experiments work". The scales were translated to local language by experts and validated by back translation. The responses to the survey were recorded on a 5-point Likert scale.

## 3.3 Online Content Engagement

Online activities of the participants on the platform were captured by logging pageviews on an external server. This provided pageview logs of the participants as they engaged with content on the PD website. Activity logs of participants were downloaded from the server in the form of json files. Data from the downloaded files was extracted using R [33] with R-Studio [34]. The difference between timestamps was used to calculate the time spent on the content. Additionally, time spent on pages at the end of the session was estimated using the user level average time spent on the page when the pageview was not at end of session.

## 3.4 Analysis

We analyzed the survey responses using Mplus 8.4 [35]. We conducted a confirmatory factor analysis. The responses to survey items were treated as categorical and the weighted least square (WLSMV) estimator [36] was used in the analysis. Measurement model consisting of both Pre-test and Post-Test responses was evaluated. Model Fit was evaluated based on criteria provided by Hu and Bentler [37] & Marsh, Hau and Wen [38] i.e. CFI >= .95, TLI >= .95, RMSEA <= .06, & SRMR <= .08. Items with standardized loadings less than 0.5 and with correlation with other items were dropped. The different patterns of engagement with online content were determined using mixture modelling [35], [39]. Percentage of total time calculated from pageview logs was used to determine latent profiles among the participants. Nylund, Asparouhov and Muthén [40] recommended the use of Bayesian Information Criteria (BIC) or Bootstrapped Likelihood Ratio tests (BLRT) to determine the appropriate number of
homogenous classes/profiles. They also note that at a higher number of observations (n > 1000) the performance of Likelihood Ratio tests (LRT), viz. VLMR LRT & ALMR LRT, is reliable. Also, they mention that the LRTs are robust and valid even if the distribution of data within classes/profile is non-normal. The steps outlined in the software user guide were followed to determine the final number of latent groups among the learners.

# 4 Findings

Complete pageview log and survey responses to both Pre and Post training were available for 6933 participants. Majority of the participants, 81%, had qualified the State's Teacher Eligibility Test (TET) and 44.9% of the total participants were females. Among the participants, 51.8% had a graduation degree and 44.5% of the teachers had earned a post-graduate degree while the rest had either a diploma or professional teachers' certification. The average work experience was 75.20 months and the average age was 32.14 years.

## 4.1 Self-Efficacy Beliefs

The analysis of survey responses resulted in a good model fit for measuring the latent constructs. The item-wise descriptive of survey responses is provided in the following table. The self-efficacy constructs at both time points demonstrated composite reliability of above 0.7 and had average variance explained more than 0.5.

<table><caption>**Table 1.** Responses to Personal Science Teaching Efficacy Scale</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Range</th><th>Pre</th><th>Post</th><th>Pre</th><th>Post</th></tr><tr><th>M (SD)</th><th>M (SD)</th><th>Std. Loadings</th><th>Std. Loadings</th></tr></thead><tbody><tr><td>I am continually finding better ways to teach science.</td><td>1 - 5</td><td>4.469 (.58)</td><td>4.578 (.70)</td><td>CR = .93 AVE= .57</td><td>CR = .94 AVE= .58</td></tr><tr><td>I know how to teach science concepts effectively.</td><td>1 - 5</td><td>4.071 (.64)</td><td>4.478 (.60)</td><td>0.776</td><td>0.794</td></tr><tr><td>I am very effective in monitoring science experiments.</td><td>1 - 5</td><td>4.241 (.62)</td><td>4.492 (.61)</td><td>0.819</td><td>0.816</td></tr><tr><td>I generally teach science effectively.</td><td>1 - 5</td><td>4.332 (.58)</td><td>4.576 (.57)</td><td>0.823</td><td>0.822</td></tr><tr><td>I understand science concepts well enough to be effective in teaching elementary science.</td><td>1 - 5</td><td>4.318 (.59)</td><td>4.552 (.56)</td><td>0.839</td><td>0.846</td></tr><tr><td>I find it easy to explain to students why science experiments work.</td><td>1 - 5</td><td>4.284 (.61)</td><td>4.543 (.57)</td><td>0.841</td><td>0.850</td></tr><tr><td>I am always able to answer students' science questions.</td><td>1 - 5</td><td>4.086 (.73)</td><td>4.346 (.68)</td><td>0.754</td><td>0.753</td></tr></tbody></table>
<table><thead><tr><th rowspan="2"></th><th rowspan="2">Range</th><th>Pre</th><th>Post</th><th>Pre</th><th>Post</th></tr><tr><th>M (SD)</th><th>M (SD)</th><th>Std. Loadings</th><th>Std. Loadings</th></tr></thead><tbody><tr><td>Given a choice, I would invite the principal to evaluate my science teaching.</td><td>1 - 5</td><td>4.128 (.73)</td><td>4.404 (.74)</td><td>0.577</td><td>0.611</td></tr><tr><td>When a student has difficulty understanding a science concept, I am usually not at a loss about how to help the student understand it better.</td><td>1 - 5</td><td>4.028 (.77)</td><td>4.347 (.79)</td><td>0.621</td><td>0.652</td></tr><tr><td>When teaching science, I usually welcome student questions.</td><td>1 - 5</td><td>4.555 (.53)</td><td>4.715 (.49)</td><td>0.712</td><td>0.741</td></tr><tr><td>I know how to motivate students to study science.</td><td>1 - 5</td><td>4.323 (.58)</td><td>4.603 (.54)</td><td>0.802</td><td>0.821</td></tr><tr><td>Correlation Pre vs Post</td><td></td><td></td><td></td><td></td><td>0.569</td></tr></tbody></table>

## 4.2 Latent profiles of online engagement

Pageview logs of 8,131 participants were downloaded from the external server. The timestamped entry of page URL was used to determine the time spent on the page content. As the logging of pageviews was being maintained on an external server, entries for some participants were incomplete. We analyzed pageview logs of 7037 participants to identify the different online engagement profiles. The percentage of time spent on content was used for analyses. This enabled comparisons between time spent in one content with another while accounting for variations due to internet speeds, reading proficiency and module content. Also, the values were median centered to enable easy identification of the latent profiles in the output. The summary of the time spent on the content by the participants is tabulated as the percentage of total time logged online in Table 2. The range, mean and median values indicate that the time spent on each content is a long-tailed distribution.

<table><caption>**Table 2.** Percentage of time spent by participants in the online programme</caption><thead><tr><th rowspan="2">Content</th><th colspan="4">Time (as a percentage of total time)</th></tr><tr><th>Range</th><th>Mean</th><th>Std. Dev.</th><th>Median</th></tr></thead><tbody><tr><td>Science Expert Content</td><td>0.77 - 38.95</td><td>11.795</td><td>4.44</td><td>11.526</td></tr><tr><td>Science Case Study</td><td>0.43 - 23.41</td><td>7.163</td><td>2.84</td><td>6.86</td></tr><tr><td>Math Expert Content</td><td>0.37 - 24.11</td><td>7.39</td><td>3.34</td><td>6.932</td></tr><tr><td>Math Case Study</td><td>0 - 21.71</td><td>6.115</td><td>2.31</td><td>5.888</td></tr><tr><td>Classroom Management Expert Content</td><td>0 - 18.49</td><td>4.459</td><td>2.19</td><td>4.073</td></tr><tr><td>Classroom Management Case Study</td><td>2.14 - 43.49</td><td>17.011</td><td>4.98</td><td>16.957</td></tr><tr><td>SCE Expert Content</td><td>0 - 10.08</td><td>1.239</td><td>1.07</td><td>0.914</td></tr><tr><td>SCE Case Study</td><td>0 - 15.39</td><td>2.418</td><td>1.22</td><td>2.18</td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Content</th><th colspan="4">Time (as a percentage of total time)</th></tr><tr><th>Range</th><th>Mean</th><th>Std. Dev.</th><th>Median</th></tr></thead><tbody><tr><td>ICT Use Expert Content</td><td>0 - 13.01</td><td>1.618</td><td>1.25</td><td>1.253</td></tr><tr><td>ICT Use Case Study</td><td>0.15 - 17.83</td><td>4.337</td><td>1.9</td><td>4.104</td></tr><tr><td>Grade Peer Projects</td><td>0 - 17.80</td><td>1.787</td><td>1.35</td><td>1.413</td></tr><tr><td>View Peer Feedback on Project</td><td>0 - 13.52</td><td>0.951</td><td>1.15</td><td>0.589</td></tr></tbody></table>

As the distribution of time spent is long-tailed, LRT tests are more valid than BLRT to determine the correct number of latent profiles. The latent profile analysis shows LRTs favor a four-profile solution. Thus, we accept the four-profile solution of the analysis. The extracted profiles of the participants can be interpreted based on the content they spent most of their time Project Feedback (5.0%), Subject Modules (57.9%), Case-Study Content (30.0%) and Expert Content (7.1%).

<figure><img src="image_2.png" /><figcaption>Fig. 1. Latent profiles of participants based on the percentage of time spent on content</figcaption></figure>

## 4.3 Association of Online Content Engagement and Change in Self-Efficacy Belief

Alessandri, Zuffiano, and Perinelli's[41] method of setting up of the second-order latent variables which measure baseline and change in the latent constructs was implemented to analyze the association of the engagement patterns with the change in self-efficacy. The article's example was adapted for categorical responses using the Mplus user guide and reference materials available on their website. The model regressed the change in self-efficacy upon the profile membership indicators (here, the Most likely profile membership) and participant covariates. The relevant results of the analysis have been tabulated in the following table.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Structural Model to evaluate association of latent online profiles to change in self-efficacy</figcaption></figure>

<table><caption>Table 3. Association of most likely Online Latent profile with change in self-efficacy beliefs</caption><thead><tr><th>Change in Self-Efficacy</th><th>Standardized Estimates [β (S.E.)]</th></tr></thead><tbody><tr><td>Intercept</td><td>.668*** (.14)</td></tr><tr><td>Baseline</td><td>-.573*** (.02)</td></tr><tr><td>Project Feedback</td><td>.059 (.06)</td></tr><tr><td>Case Study Content</td><td>.028 (.03)</td></tr><tr><td>Expert Content</td><td>-.098 (.05)</td></tr><tr><td>Gender: Female</td><td>-.021 (.02)</td></tr><tr><td>Age</td><td>-.010 (.02)</td></tr><tr><td>Work Experience</td><td>-.022 (.02)</td></tr><tr><td>TET Qualified</td><td>-.043 (.04)</td></tr><tr><td>Education: (PTC/Diploma)</td><td></td></tr><tr><td>Graduate</td><td>-.087 (.07)</td></tr><tr><td>Post Grad and above</td><td>-.135 (.07)</td></tr><tr><td><u>Model Fit Indices</u></td><td></td></tr><tr><td>χ<sup>2</sup></td><td>1486.185</td></tr><tr><td>df</td><td>413</td></tr><tr><td>CFI</td><td>0.993</td></tr><tr><td>TLI</td><td>0.993</td></tr><tr><td>SRMR</td><td>0.031</td></tr><tr><td>RMSEA</td><td>0.019</td></tr><tr><td>RMSEA 90% CI</td><td>[.018 .020]</td></tr><tr><td colspan="2">Note: "." p&lt;0.1 "*" p&lt;.05, "**" p&lt;.01, "***" p&lt;.001</td></tr></tbody></table>
The findings show that the participants reported a large and significant positive change in their personal science teaching efficacy beliefs. Additionally, participants who reported low teaching efficacy beliefs benefitted significantly more from the programme. Participant's age, gender and work experience did not have a significant effect on the change in self-efficacy. We do notice a slight negative association with change in self-efficacy, at the 90% significance level, among participants who engaged more with expert content. Similarly, participants with post graduate degrees seemed to have an adverse association with change in self-efficacy (at 90% significance level).

## 5 Discussions

The different online engagement profile of the participants clearly shows 1. Variations in needs of teachers from a professional development course and 2. Need for an online PD programme which can offer the flexibility of indulging in desired content. The large, significant and positive change among the participants validate the design and content of the programme. Further, the effect size of the negative association of participants who indulged in expert content is very small and failed to be rejected at 95% significance. We could ignore the negative effects for practical reasons, but it satisfies the objective of the article by demonstrating the benefit of extracting latent engagement profiles of participants to evaluate the content of an online course.

## 6 Limitations

The external server used for tracking pageview activity provided log entries with time precision in minutes. Thus, page views which occurred within a minute were not captured, thus time on content may have been underestimated. Further, for this article, we have used "most likely profile membership" to analyze the relationship between engagement profile and change in self-efficacy. Latent profile memberships in probabilities, with suitable transformations, could be used in the analysis to provide a more accurate estimate of the association.

## 7 Implications

The first implication of our findings is that the contents of this particular online PD programme for teachers were sufficiently varied to cater to all participants. Irrespective of whether the teachers engaged more with the subject specific content, case-studies, expert content or project feedback there was no variation (at 95% significance) in the reported positive change in science teaching self-efficacy. Thus, the online programme offered a sufficient balance of content from both experts and peers that the participants needed. Secondly, our study shows one way of evaluating the contents of online programmes. Extracting latent profiles based on navigation logs enables further investigation on the effects of the programme on subgroups of participants. This would facilitate
continuous improvement of content and design of the online course or programme. Future research could attempt at exploring these latent profiles with qualitative methods. Researchers could also investigate activities participants undertake outside the learning platform and the effect of these off-platform activities on outcomes.

## 8 Acknowledgements

The online professional development programme was developed as a collaboration between the Government of Gujarat and Ravi J Matthai Center for Educational Innovation, IIM Ahmedabad. We are thankful for the time and support provided by various professionals working at both these institutions. We also like to thank Mr. Avinash Bhandari, Ms. Megha Gajjar, Mr. Sanket Savaliya, Mr. Lekh Nakrani & Ms. Nishanshi Shukla in coordinating the development and implementation of the programme.

## 9 References

[1] C. Terwiesch and K. Ulrich, "Will Video Kill the Classroom Star? the Threat and Opportunity of Massively Open Online Courses for Full-Time Mba Programs," 2014.

[2] C. Dede, D. Jass Ketelhut, P. Whitehouse, L. Breit, and E. M. McCloskey, "A Research Agenda for Online Teacher Professional Development," *Journal of Teacher Education*, vol. 60, no. 1, pp. 8–19, 2009.

[3] H. Borko, "Professional Development and Teacher Learning: Mapping the Terrain," *Educational Researcher*, vol. 33, no. 8, pp. 3–15, 2004.

[4] L. M. Desimone, "Improving Impact Studies of Teachers' Professional Development: Toward Better Conceptualizations and Measures," *Educational Researcher*, vol. 38, no. 3, pp. 181–199, 2009.

[5] J. M. S. Desouza, W. J. Boone, and O. Yilmaz, "A study of science teaching self-efficacy and outcome expectancy beliefs of teachers in India," *Science Education*, vol. 88, no. 6, pp. 837–854, 2004.

[6] C. Dyer et al., "Knowledge for teacher development in India: The importance of 'local knowledge' for in-service education," *International Journal of Educational Development*, vol. 24, no. 1, pp. 39–52, 2004.

[7] R. Singh and S. Sarkar, "Does teaching quality matter? Students learning outcome related to teaching quality in public and private primary schools in India," *International Journal of Educational Development*, vol. 41, pp. 153–163, 2015.

[8] P. Sehgal, R. Nambudiri, and S. K. Mishra, "Teacher effectiveness through self-efficacy, collaboration and principal leadership," *International Journal of Educational Management*, vol. 31, no. 4, pp. 505–517, 2017.

[9] M. Tschannen-Moran and A. W. Hoy, "Teacher efficacy: Capturing an elusive construct," *Teaching and Teacher Education*, vol. 17, no. 7, pp. 783–805, 2001.

[10] R. D. Goddard, W. K. Hoy, and A. W. Hoy, "Collective Teacher Efficacy: Its Meaning, Measure, and Impact on Student Achievement," vol. 37, no. 2, pp.
479–507, 2010.

[11] A. Lumpe, C. Czerniak, J. Haney, and S. Beltyukova, “Beliefs about Teaching Science: The relationship between elementary teachers’ participation in professional development and student achievement,” *International Journal of Science Education*, vol. 34, no. 2, pp. 153–166, 2012.

[12] A. J. Gabriele and E. Joram, “Teachers’ reflections on their reform-based teaching in Mathematics: Implications for the development of teacher self-efficacy,” *Action in Teacher Education*, vol. 29, no. July 2015, pp. 60–74, 2007.

[13] S. Gibson and M. H. Dembo, “Teacher efficacy: A construct validation,” *Journal of Educational Psychology*, vol. 76, no. 4, pp. 569–582, 1984.

[14] M. Gregoire, “Is It a Challenge or a Threat? A Dual-Process Model of Teachers’ Cognition and Appraisal Processes during Conceptual Change,” *Educational Psychology Review*, vol. 15, no. 2, pp. 147–179, 2003.

[15] E. Levenson and H. Gal, “Insights from a teacher professional development course: Rona’s changing perspectives regarding mathematically-talented students,” *International Journal of Science and Mathematics Education*, *Dordrecht*, vol. 11, no. 5, pp. 1087–1114, 2013.

[16] J. J. Summers, H. A. Davis, and A. W. Hoy, “The effects of teachers’ efficacy beliefs on students’ perceptions of teacher relationship quality,” *Learning and Individual Differences*, vol. 53, pp. 17–25, 2017.

[17] H. C. Hill, M. Beisiegel, and R. Jacob, “Professional Development Research: Consensus, Crossroads, and Challenges,” *Educational Researcher*, vol. 42, no. 9, pp. 476–487, 2013.

[18] A. K. Pehmer, A. Gröschner, and T. Seidel, “How teacher professional development regarding classroom dialogue affects students’ higher-order learning,” *Teaching and Teacher Education*, vol. 47, pp. 108–119, 2015.

[19] M. J. Kennedy, W. J. Rodgers, J. E. Romig, J. W. Lloyd, and M. T. Brownell, “Effects of a Multimedia Professional Development Package on Inclusive Science Teachers’ Vocabulary Instruction,” *Journal of Teacher Education*, vol. 68, no. 2, pp. 213–230, 2017.

[20] Y. Qian, S. Hambrusch, A. Yadav, and S. Gretter, “Who Needs What: Recommendations for Designing Effective Online Professional Development for Computer Science Teachers,” *Journal of Research on Technology in Education*, vol. 50, no. 2, pp. 164–181, 2018.

[21] C. L. Rosaen, J. F. Carlisle, E. Mihocko, A. Melnick, and J. Johnson, “Teachers learning from analysis of other teachers’ reading lessons,” *Teaching and Teacher Education*, vol. 35, pp. 170–184, 2013.

[22] E. C. Minor, L. M. Desimone, J. C. Lee, and E. D. Hochberg, “Insights on How to Shape Teacher Learning Policy: The Role of Teacher Content Knowledge in Explaining Differential Effects of Professional Development,” *Education Policy Analysis Archives*, vol. 24, no. 61, pp. 1–34, 2016.

[23] C. Romero and S. Ventura, “Educational data science in massive open online courses,” *Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery*, vol. 7, no. 1, pp. 1–12, 2017.

[24] S.-F. Tseng, Y.-W. Tsao, L.-C. Yu, C.-L. Chan, and K. R. Lai, “Who will pass?”
Analyzing learner behaviors in MOOCs,” *Research and Practice in Technology Enhanced Learning*, vol. 11, no. 1, p. 8, 2016.

[25] R. F. Kizilcec, C. Piech, and E. Schneider, “Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses,” in *Proceedings of the Third International Conference on Learning Analytics and Knowledge - LAK '13*, 2013, p. 170.

[26] R. Ferguson and D. Clow, “Examining engagement: analysing learner subpopulations in massive open online courses (MOOCs),” in *Proceedings of the Fifth International Conference on Learning Analytics And Knowledge - LAK '15*, 2015, pp. 51–58.

[27] C. Coffrin, L. Corrin, P. de Barba, and G. Kennedy, “Visualizing patterns of student engagement and performance in MOOCs,” in *Proceedings of the Fourth International Conference on Learning Analytics And Knowledge - LAK '14*, 2014, no. March, pp. 83–92.

[28] C. Milligan, A. Littlejohn, and A. Margaryan, “Patterns of engagement in connectivist MOOCs,” *MERLOT Journal of Online Learning and Teaching*, vol. 9, no. 2, pp. 149–159, 2013.

[29] A. Ramesh, D. Goldwasser, B. Huang, H. Daum, and L. Getoor, “Modeling Learner Engagement in MOOCs using Probabilistic Soft Logic,” *NIPS Workshop on Data Driven Education*. pp. 1–7, 2013.

[30] V. Kalakoski, H. Ratilainen, and L. Drupsteen, “Enhancing learning at work. How to combine theoretical and data-driven approaches, and multiple levels of data?,” in *Esann*, 2015, no. April, pp. 22–24.

[31] I. M. Riggs and L. G. Enochs, “Toward the development of an elementary teacher’s science teaching efficacy belief instrument,” *Science Education*, vol. 74, no. 6, pp. 625–637, 1990.

[32] L. G. Enochs, P. L. Smith, and D. Huinker, “Establishing Factorial Validity of the Mathematics Teaching Efficacy Beliefs Instrument,” *School Science and Mathematics*, vol. 100, no. 4, pp. 194–202, 2000.

[33] R-Core Team, “R: A language and environment for statistical computing.” R Foundation for Statistical Computing, Vienna, Austria, 2016.

[34] RStudio-Team, “RStudio: Integrated Development for R.” RStudio, Inc., Boston, MA, 2016.

[35] L. K. Muthén and B. O. Muthén, *MPlus user's guide*, Eighth Ed. Los Angeles, CA: Muthén & Muthén, 2017.

[36] B. O. Muthén, S. H. C. Du Toit, and D. Spisic, “Robust Inference using Weighted Least Squares and Quadratic Estimating Equations in Latent Variable Modeling with Categorical and Continuous Outcomes,” *Psychometrika*. p. 49, 1997.

[37] L. T. Hu and P. M. Bentler, “Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives,” *Structural Equation Modeling*, vol. 6, no. 1, pp. 1–55, 1999.

[38] H. W. Marsh, K. T. Hau, and Z. Wen, “In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings,” *Structural
*Equation Modeling*, vol. 11, no. 3, pp. 320–341, 2004.

[39] J. Magidson and J. K. Vermunt, “Latent class models for clustering: A comparison with K-means,” *Canadian Journal of Marketing Research*, vol. 20, no. 1, pp. 36–43, 2002.

[40] K. L. Nylund, T. Asparouhov, and B. O. Muthén, “Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study,” *Structural Equation Modeling*, vol. 14, no. 4, pp. 535–569, 2007.

[41] G. Alessandri, A. Zuffianò, and E. Perinelli, “Evaluating intervention programs with a pretest-posttest design: A structural equation modeling approach,” *Frontiers in Psychology*, vol. 8, no. MAR, 2017.