Cultivating, visualising and assessing critical thinking in online learning environments at the higher education level
Creators: name: HART, Claire
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 390303
subject: Higher education
classificationCode: 390408
subject: Learning analytics
classificationCode: 490501
subject: Applied statistics

description: Critical thinking is considered to be one of the most important outcomes of higher education courses (Dunne, 2015; Facione, 1990), with critical thinking either explicit or implicit as a learning objective within these courses. It is therefore important that critical thinking is incorporated into online learning instruction at the higher education level. Given that online learning environments are an instructional mode distinct from traditional teaching, a lack of research within this area may contribute towards instructors making poor choices regarding online critical thinking instruction. This thesis presents three major chapters that address three distinct but related domains of critical thinking instruction within an online learning environment: the cultivation, visualisation and assessment of critical thinking, respectively. The literature review undertaken in this dissertation sought to develop theory- and evidence-based recommendations for online critical thinking instruction for application across the three major chapters. 

In the first major chapter, a mixed methods study is presented that explored how student engagement with online critical thinking instruction can be improved. Student disengagement not only limits the capacity of students to consistently cultivate their critical thinking skills but also presents as a challenge to instructors and researchers seeking to evaluate the effectiveness of online critical thinking interventions. The quantitative component of this study involved evaluating students' level of engagement with online critical thinking modules based on a total of 31 undergraduate students enrolled in a second-year statistics course. This included the analysis of student data relating to word count, the number of questions attempted, and the number of students who completed all questions. The qualitative component involved conducting a focus group discussion to explore students' perceptions of their experiences of the online critical thinking modules and how instructors can improve student engagement with online critical thinking instruction. A total of seven students enrolled in the same course participated in the focus group. It emerged that student engagement generally declined throughout the online critical thinking modules. However, the qualitative findings suggested that participants generally felt positive about their experiences of the modules and the decline in student engagement may have been predominantly due to students perceiving the modules to be too time-consuming as well as insufficient course emphasis on the modules. There was no evidence to suggest that the critical thinking learning objectives helped improve student participation. 

The second major chapter explored the use of data visualisation techniques on student critical thinking data. The critical thinking data were based on learning analytics stemming from the above mentioned online critical thinking modules, such as the mean word count and completion rate for each module, as well as students' open-ended responses to the critical thinking questions. Data visualisations are graphic representations of data in a format that enables the easy identification of trends, patterns and outliers. Data visualisations based on student critical thinking data might have the potential to provide instructors with valuable insights regarding student engagement with online critical thinking instruction and student performance in respect to certain skills, dispositions or dimensions. Such visualisations should be based on critical thinking data that have a strong theoretical basis as well as be designed in such a way that instructors are able to identify trends as accurately and efficiently as possible. The data visualisations presented within this chapter were designed to provide insights into student engagement, the content of students' open-ended critical thinking responses, student performance based on critical thinking dimension scores, and changes in students' critical thinking performance over time. The participants for this study were the 31 undergraduate students enrolled in the statistics course in which the modules were implemented. This research has important implications for instructors seeking to monitor student engagement and performance in respect to online critical thinking modules. It is suggested that future studies expand on this research by exploring instructors' perceptions regarding the interpretability and usefulness of critical thinking data visualisations. 

In the third and final major chapter, a study is presented that evaluated the reliability and validity of a pilot critical thinking assessment tool that was designed to assess higher education students' critical thinking skills. This assessment tool consisted of both constructed- and selected-response items. Constructed-response items require test-takers to construct or articulate their answers, such as short-answer or essay type questions. These constrast with selected-response items which require test-takers to select the 'correct' response from a list of options, such as multiple choice questions or true-false statements. Given the psychometric limitations associated with many of the existing critical thinking assessments as well as the lack of assessments utilising constructed-response items, there is a need for critical thinking assessments that are not only psychometrically sound but also permit test-takers to articulate their reasoning. This research study also performed an exploratory content analysis on test-takers' critical thinking responses, including a comparison between test-takers' selected and constructed responses. A total of 95 undergraduate students were included in the study, with participants recruited from psychology, chiropractic and computer science programs. This study found preliminary evidence of inter-rater reliability for the constructed-response portion of the assessment tool, yet mixed evidence regarding the validity of the tool. It also found several discrepancies between participants' selected and constructed responses, thereby raising concerns regarding the use of selected-response items in critical thinking assessments. These findings have important implications for the assessment of higher education students' critical thinking skills. 

The findings of the three major studies presented herein provide valuable insights on improving the quality of online critical thinking instruction at the higher education level.  Ultimately, these findings have important implications for the development of higher education students' critical thinking skills and dispositions that promote success in academic and professional contexts.