Assessing the Learning and Transfer of Data Collection Inquiry Skills Using Educational Data Mining on Students ’ Log Files

In this paper we explored whether engaging in two i nqu ry skills associated with data collection, designing controlled experime nts and testing stated hypotheses, within microworlds for one physical science domain (density) impacted the acquisition of inquiry skills in another domain (ph ase change). To do so, we leveraged educational data mining techniques to bot h assess and estimate students’ inquiry skills across domains. Analyses revealed th at honing these skills in density activities provided benefits in terms of transfer a nd skill acquisition. More specifically, students who practiced in density act ivities first were more likely to show mastery of the designing controlled experiment s skill than those who had no prior practice. These same students were also more likely to test their stated hypotheses during their first data collection in ph ase change. Thus, practice in one domain can positively impact acquisition and transf er of skill in a second domain, suggesting that inquiry skills also have a degree o f domain generality. Introduction Science educators agree that cultivating inquiry sk ills is critical for students to become scientifically literate (National Research Council, 1996, 2000, 2011; Kuhn, 2005a). However, typical standardized science tests do not adequatel y reflect or assess complex inquiry process skills (Quellmalz, Timms & Schneider, 2009). Perfor mance assessments of inquiry, instead, have been argued to be better-suited for this purpo se (cf. Black, 1999; Pellegrino, 2001). Devising scalable and reliable performance assessme nt , though, is difficult for two reasons. First, it is difficult to separate inquiry skills f rom content understanding (Mislevy, Steinberg & Almond, 2002; Mislevy, et al., 2003). Second, inq uiry processes are mutli-faceted, and there is no one single “right or wrong” way to enga ge in science inquiry (Shute, Glaser & Raghavan, 1989; Glaser, Schauble, Raghavan, & Zeitz , 1992). Given inquiry’s importance, proper techniques for measuring inquiry are needed. It is also important to better understand inquiry l earning so that we can foster transfer of such skills to novel tasks (Kirschner, Sweller & Clark, 2006; Hmelo-Silver, Duncan & Chinn, 2007). Regarding transfer, it has been sugge sted that inquiry skills are tightly tied to the domain in which they are learned (van Joolingen , d Jong & Dimitrakopoulout, 2007), but some evidence exists that long-term, repeated p ractice of inquiry (Kuhn, Schauble & Garcia-Mila, 1992; Dean Jr. & Kuhn, 2006; Kuhn & Pe as , 2008), and scaffolding or teaching these skills explicitly (Klahr & Nigam, 20 04) can lead to successful acquisition and transfer to novel tasks. In the present paper, we address two goals. First, we describe our approach for developing reliable, scalable performance measures of inquiry. Second we leverage those assessment techniques to examine how inquiry skills transfer between two physical science domains. We focus on on two inquiry skills, designi ng controlled experiments and testing stated hypotheses. Designing controlled experiments entails selecting experiments to yield data that supports determining the effects of manip ulable variables on outcomes. Testing stated hypotheses refers to generating data with th e in ention to support or refute a specific hypothesis. These skills are measured as students c o du t inquiry within microworlds for two domains, phase change and density, developed within the Science Assistments system (Gobert et al., 2007; Gobert et al., 2009). In our approach, we leverage techniques from Educat ion l Data Mining (cf. Baker & Yacef, 2009; Romero & Ventura, 2010) to assess and tr ck inquiry skills across several activities within each domain. To assess these skil ls, we use validated detectors (models) of students’ inquiry behaviors that were constructed b ased on student log files (Sao Pedro et al., 2010, in press). We then produce estimates of stude n proficiency for each skill by aggregating all assessments into a Bayseian Knowled ge Tracing model (Corbett & Anderson, 1995). This approach is rigorous because an EDM aff ords the ability to estimate how well the models assess and track skill. Furthermore, these c an be done in real time. Thus, we argue, the approach is scalable, and could provide a possi ble model for inquiry assessment. These techniques were also leveraged to measure whe ther the two skills of interest, designing controlled experiments and testing stated hypotheses, transfer across two physical science domains. These two domains are Density and Phase Change. More specifically, we analyzed whether students who practiced in density activities first had a greater likelihood of demonstrating the skills or reaching mastery than s tudents with no prior practice. With our assessment and skill tracking models, we can assess this transfer in a finer-grained way than other prior studies of inquiry (e.g. Kuhn & Pease, 2008). The remainder of this paper is organized as follows . Fir t, we describe the two skills of interest in more detail, and present related wor k on assessing data collection skills. We then present a high-level view of our approach for assessing and estimating proficiency at these data collection skills using our educational data mining techniques. Next, we present our results on the transfer of these skills between domains which leveraged our assessment and estimation techniques. Finally, we present a di scussion and conclusions of our paper.