THE EFFECT OF LEARNING MANAGEMENT SYSTEM TRAINING ON TEACHERS ’ ONLINE TEACHING

Universities invest considerable resources in learning management system (LMS) training for their staff. One measure of the effectiveness of this training is participants’ post-training behaviour, which can be obtained from LMS usage logs. In this paper we report preliminary analysis of these logs, showing that both teachers who have received LMS training and their students are more active in their online courses compared to those who have not. This preliminary analysis of usage data in conjunction with training information suggests a positive effect of training and can potentially help to provide information to ensure training is targeted and effective. The Effect of Training on Teachers’ Learning Management System Use Considerable resources are invested by universities in providing eLearning training to staff, particularly in relation to use of their institutional learning management system (LMS). However, training effectiveness is usually measured by post-training surveys where participants provide their views about the training, such as whether they feel it was effective and whether they were satisfied with the training. A more objective method for evaluating training effectiveness is participants’ behaviour post-training, one measure of which can be obtained from LMS tracking logs. In this paper we report on preliminary analysis of these logs, which shows that both teachers who have received LMS training and their students are more active in their online courses compared to those who have not. Utilisation studies of LMS data are not new. A number of studies have been published reporting on students’ use of LMS tools derived from data logs within the system (e.g., Jurado, Pettersson, Gomez, & Scheja, 2014; Lam, Lo, & Lee, 2010; Lam, Keing, McNaught, & Cheung, 2006; Morris, Finnegan, & Wu, 2005; Phillips, 2006; Romero, Ventura, & García, 2008). However, with advances in learning analytics, greater attention is being paid to analysing large data sets to understand learner behavior and optimising learning outcomes for students (Reyes, 2015). In addition to improving learning outcomes for students, learning analytics can also assist institutions in gaining valuable insights to inform strategic decision making, particularly in regard to resource allocation (Lam et al., 2006; Macfadyen & Dawson, 2012). In this paper we report on the application of learning analytics to Blackboard usage logs to understand the effects of LMS training on teachers’ and students’ activity in online courses. LMS Usage Logs LMS usage data have been analysed in a number of studies and for various purposes. In one of the earlier reports on analysis of LMS logs, Phillips (2006) reported that the institutional LMS at several universities was being used mainly for providing students with content and information. This type of use was described as teacher-centred and not ICICTE 2016 Proceedings 64 consistent with an online learning environment designed according to constructivist principles. Classification systems based on tool usage have also been developed for analysing LMS usage data. For example, Montenegro-Marin, Cueva-Lovell, Sanjuan and Nuñez-Valdez (2011) developed an ontology of modules common in learning management system platforms, which included tools, consisting of administration, communications, course, curricula design, and productivity, and users. Another classification system for LMS features was developed by Jurado and colleagues (2014) where tools are categorised according to purpose: for distribution (e.g., contents page, URL, documents, etc.), communication (e.g., mail, calendar, announcements), interaction (e.g., discussion areas, assignments, surveys, quizzes) or course management (e.g., gradebook, student tracking). Their work has shown that tools for distribution are used far more than tools for communication or interaction, which is consistent with Phillips’ finding from eight years earlier. Analysis of usage data at this level provides useful information. For example, counts of tool use have been shown to be significantly correlated with students’ final grades (Macfayden & Dawson, 2010; Morris et al., 2005). In their study of student behavior, persistence and achievement in online courses, Morris and colleagues (2005) report a regression analysis showing that the statistically significant predictors of final grades included number of discussion posts viewed and number of content pages viewed. They also found that students who successfully completed the course engaged with online learning activities with greater frequency and for longer durations than did unsuccessful students who eventually withdrew. Given these results, we compared LMS usage data for courses taught by teachers who have undertaken LMS-related training with those of teachers who have not to provide insight into the effect of training on LMS use. In doing so, we hoped to obtain important evidence to inform support for the effectiveness of training for promoting LMS usage by both students and staff, as well as to inform future training practice at our institution. Understanding how eLearning training, particularly in relation to an institutional LMS, impacts teaching practice and use of the LMS is important for assessing the effectiveness of training and staff development. To help address the question of impact, we have begun to explore use of LMS data to investigate differences in online behavior of students and teachers between courses taught by staff who have attended LMSrelated training and those who have not. The aim was to provide objective data that addresses the question of what changes occur following training and how this impacts students’ and teachers’ online behaviour. As Picciano (2014) notes, data-driven decision making relies on an appropriate model and valid data. This proof of concept demonstrates that our method for extracting and analysing data results in valid, reliable and useful information that is valuable in decision making relating to both the LMS and staff training related to its use. The focus on actual behaviour is an important aspect of this approach research by Saks and Burke (2012) showed that self-report transfer of training is significantly predicted by training evaluation, but only if the evaluation includes analysis of behaviour and outcomes. In particular, they found that organisations report higher rates of transfer of training where more frequent evaluation of training in terms of behaviour and results is conducted. In terms of evaluating LMS training effectiveness, usage data can be used as measures of behaviour and results and represents a new approach to assessing training outcomes. This is important, because, as Weaver (2006) notes, training of staff to ICICTE 2016 Proceedings 65 support them in using the LMS needs to continually evolve to promote discussion and adoption of best practice, to cater to different staff requirements and to keep up with changes in the LMS itself as well as changes to other elearning tools.