Measuring Learner Engagement in Computer-Equipped College Classrooms

Although engagement and learning seem linked, quantitatively measuring this relationship is challenging. New technologies offer a window into studying the interactions among classroom activity, student engagement, and positive learning outcomes in computer-equipped classrooms. A Classroom Behavioral Analysis System (CBAS) was developed to measure student engagement in a college writing class, and to test the hypothesis that an interactive lesson would increase student engagement levels in a computer-equipped classroom. Student computer-based behaviors (off-task and on-task Internet visits) were compared during a traditional, lecture-based lesson (no-simulation condition) and an interactive simulation-based lesson (simulation condition). The dependent variable was student engagement as measured by the number of off-task and on-task Internet activities during the lesson. Off-task Internet activities were operationalized as website visits that were not part of the classroom activity; on-task Internet activities included websites that related to the assigned class activity. CBAS recorded all student computer actions during the observed instructional periods. Students attending a simulation-based lesson performed more on-task Internet actions, and significantly fewer off-task Internet actions than did students attending a lecture-based lesson. These findings support the hypothesis that interactive lessons increase student engagement levels in computer-equipped classrooms, and demonstrate that CBAS is a promising tool for studying student engagement. Measuring Learner Engagement 3 Measuring Learner Engagement in Computer-Equipped College Classrooms In a typical classroom situation, at any given time, some students are paying attention to varying degrees and others are not. It is difficult for instructors and researchers to determine the extent to which students are actually engaged with the classroom activities. Behavioral cues, such as students looking at the teacher, may provide some indication of engagement levels; however, students who appear to not be paying attention may be completely engaged and vice-versa. While engagement and learning seem innately linked, there is little research to quantify relations among classroom activity, student engagement, and positive learning outcomes (Fredricks, Blumenfeld, & Paris, 2004). Part of the difficulty in determining this relation lies in the challenge of measuring student engagement. A potential solution to this problem rests in new technologies, which offer a window into student engagement that has previously not been available. Previously problematic areas of learning and cognition, such as student engagement levels within a classroom, can now be measured using emerging technologies to record and analyze student computer actions (Dickey, 2005; Zhang, Almeroth, & Bulger, 2005). In this study, we developed a Classroom Behavioral Analysis System (CBAS) to measure student engagement levels as reflected by their on-task and off-task Internet actions during an instructional episode. CBAS consists of monitoring software that records all student computer actions performed during a class session, including keystroke activities, active applications, and website visits. At the end of each class session, CBAS reports this information as a log file that can then be studied to determine Measuring Learner Engagement 4 whether patterns exist in student engagement levels. This record of Internet activity can then be evaluated in light of the classroom instructional environment. In order to validate CBAS as a tool for measuring engagement, we compared student computer behaviors in a writing class taught by lecture (no-simulation condition) and a writing class taught by using an interactive simulation exercise (simulation condition). We used CBAS to record student Internet activities during a class session and then counted student off-task and on-task behaviors. We then compared these behaviors to determine whether student engagement levels, as measured by off-task and on-task Internet actions, were affected by the teacher’s instructional style. If CBAS is a valid tool for measuring engagement, the level of on-task behaviors should be higher and that of off-task behaviors lower in the simulation class than in the lecture class. What Mediates the Notion that Engagement Improves Learning? Engagement is a central component in many theories of academic learning. In fact, a general assumption of learning studies is that students learn more if they pay attention (Fredricks, Blumenfeld, & Paris, 2004; McMahon & Portelli, 2004). Early studies defined engagement in terms of interest (Dewey, 1913), effort (Meece & Blumenfeld, 1988), motivation (Pintrich & DeGroot, 1990; Skinner & Belmont, 1993), and time on task (Berliner, 1979, 1990; Lentz, 1998). In these studies, a conceptual correlation between engagement and positive learning outcomes was established by linking interest, for example, to active learning (Dewey, 1913; Schraw & Lehman, 2001), or effort to goal achievement (Brophy, Rashid, Rohrkemper, & Goldberger, 1983; Meece & Blumenfeld, 1988). Research on active engagement consistently shows that when Measuring Learner Engagement 5 students are focused on a task, they are more likely to apply effort during their learning experience (Ames, 1992; Brophy, Rashid, Rohrkemper, & Goldberger, 1983). A trend in recent research is to study the cognitive strategies that result from varying levels of motivation (Pintrich, 2002; Pressley, 1986; Winne, 1992). Metacognitive control, which is evident in students’ ability to not only know what to do in a learning situation (cognitive strategies), but when to do it, is measured by selfefficacy cues, self-regulation, and goal setting. Pressley (1986) and Winne (1992) argue that metacognitive control is teachable. Current studies of classroom engagement consistently find that classroom environment, including the teacher’s lesson plan and lecture delivery style, can affect students’ practice of metacognitive control (Dickey, 2005; Winne, 2006). Importantly, students demonstrating cognitive strategies such as task-mastery goals report higher levels of engagement and perform better on assigned tasks (Ames & Archer, 1988; Meece, 1988). Students who believe they are capable of performing assigned tasks (i.e., students high in self-efficacy beliefs) also demonstrate high levels of academic performance (Ames & Archer, 1988; Schunk, & Hanson, 1985,: Schunk, 1991; Zimmerman & Martinez-Pons, 1990). Research on self-efficacy shows a correlation between self-efficacy beliefs and active learning strategies, as well as between self-efficacy beliefs and improved performance on achievement tests (Ames, 1992; Pintrich & DeGroot, 1990; Schunk, 1985, 1989; Zimmerman & Martinez-Pons, 1990). Students’ self-efficacy beliefs reflect how well they believe they will perform on a task. Although factors such as experience and aptitude play a role in the self-efficacy beliefs students bring to the classroom, Ames (1992) has shown that self-efficacy beliefs are also formed during the instructional Deleted: 1983 Deleted: 9, Deleted: According to Mayer (2003), these beliefs come from four sources within the student’s classroom experience: “interpreting one’s own performance, interpreting the performance of others, interpreting others’ expressions of your capabilities, and interpreting one’s physiological state” (p. 470). Measuring Learner Engagement 6 episode. Since students revise their self-efficacy beliefs based on interpretations of peer and teacher interactions, the presentation of information during the lesson is essential to encouraging high self-efficacy and therefore active engagement in the learning process (Zimmerman & Martinez-Pons, 1990). A basic tenet of these cognitive theories of learning and instruction is that students learn more deeply when they are engaged in active learning than when they are passive recipients of information (Grabinger, 1996; Mayer, 2003; Pearce, Ainley, & Howard, 2005). Active learning occurs when a learner engages in active cognitive processing during learning, such as attending to relevant information, organizing the selected information into a coherent cognitive structure, and integrating the information with existing knowledge (Mayer, 2001, 2003). Active learning takes place in an environment where a student is not a passive listener but is instead an active participant in his/her learning experience (Gee, 2003; Jonassen, 1996,; Jonassen, Peck, & Wilson, 1999). How Can New Technologies be Used to Study Student Engagement? Computer lab settings provide fresh opportunities for measuring classroom behaviors because students use the computer for both course-relevant and recreational activities. In this setting, it is possible to collect student behavior data such as applications used, time spent using each application, Internet activities, frequency of attention shifts within program use, and keystrokes. These computer actions offer a window into the cognitive interplay between student and computer. Computer actions show where students focus their attention during instruction, the duration of this focus, and when their attention focus shifts. Deleted: 1997