The comprehensive evaluation of the use of technology in education
Creators: name: Lai, Jennifer
Publisher: name: Macquarie University
Year: 2024
Subjects: classificationCode: 399999
subject: Other education not elsewhere classified

description: Over recent decades, digital technologies have offered new ways to support learning and teaching. The use of technology to enhance learning is a complex process which in turn makes effective evaluation of the technology application in education a challenging endeavour. Reliable comparison of the use of technologies in education could potentially help stakeholders determine which approaches are effective across implementations, which in turn can inform the design of technology-enhanced curriculum and guidelines for the development and use of technology in education. According to research conducted in this thesis, there is a lack of field-specific instrument or approach that facilitates the holistic evaluation of the use of technology in education. The main purpose in this thesis is to address the question “<em>How can we comprehensively evaluate the use of technology in education?</em>”
This thesis by publication is comprised of six interrelated studies. Firstly, a systematic literature review of 365 empirical papers was conducted to investigate what researchers evaluated when using technology in education. Eight evaluation themes emerged from the systematic literature review: learning outcomes, affective elements, behaviours, design, technology elements, pedagogy, presence/community, and institutional environment (Paper 1). To make sure that the results of the first systematic review were robust and comprehensive, a tertiary systematic literature review was carried out to investigate the trends in learning technology evaluation (Paper 2). The results indicated that the outcomes of past literature reviews could be entirely encapsulated by the eight identified dimensions, providing a validation that the dimensions were broadly applicable. Based on criticisms that the evaluation of technology use in education is often not theoretically aligned, a systematic analysis of theory underpinning evaluation was conducted (Paper 3). This analysis revealed that the diverse theories used tended to encapsulate only some aspects of educational technology use, but once again, the eight dimensions were able to comprehensively represent the different aspects of theories that were used to evaluate of use of technology in education. 
Next, items for a survey instrument were identified and validated by exploring what field experts think is important to focus upon when evaluating the use of technology in education (Paper 4). There was an alignment between what the specialists felt were important and the eight constructs that had typically been the focus of empirical studies. These eight dimensions were also validated in an analysis of informal student evaluations of a Coursera course - Excel Skills for Business Specialisation (Paper 5). The results indicated that, once again, all aspects of student informal evaluations were able to be encapsulated within the eight dimensions, and at the same time the paper helped to characterise the nature of the course that was used for the final validation study. In the final study, the eight-factor model was tested by surveying 1,352 participants undertaking a Coursera open online course (Paper 6). The ultimate 28-item “<em>Comprehensive Evaluation of Use of Technology in Education</em>” (CEUTIE) instrument was validated using Confirmatory Factor Analysis (CFA) and qualitative feedback from participants. The CEUTIE model provides researchers and other educational stakeholders with the means to perform consistent, holistic and robust evaluation and comparison of the use of technology for learning and teaching across diverse educational contexts.