Creating Capacity to Explore what Students Learn from Reﬂection Activities: Validating the Knowledge-gain Survey

interested in all aspects of engineering education, including how to support engineering students in reﬂecting on experience, how to help engineering educators make effective teaching decisions, and the application of ideas from complexity science to the challenges of engineering education. Abstract This paper reports on the methodological process of validating a survey instrument to measure student learning from reflection activities. Reflection is thought to be a helpful teaching and learning tool. In engineering education, reflection is gaining traction as a tool to help students think about their study habits, exam performance, command of the course content, and team interactions. Yet few validated instruments exist to systematically document what students are learning from reflection experiences. The purpose of this research project is to provide preliminary evidence of validation for an instrument to capture the knowledge gains of students from doing reflection activities in a course context. Having a validated survey will allow researchers and educators to compare knowledge gains across activities, between classes, and even across institutions. In order to create the instrument, the research team followed the survey validation process. The 72 items, or questions for the survey, were developed using various learning models, such as Bloom’s Taxonomy cognitive and affect domain, and Dee Fink’s Taxonomy of Creating Significant Learning Experiences, to ensure we captured multiple learning opportunities. Students give answers by choosing their level of agreement on a likert scale for each of the items. Our items ask about expected learning outcomes such as “I better understood what had been confusing about a topic” and “I understood how the topics in this course can be applied to the real world,” which relate to course content knowledge and knowledge relevant to their careers. Additionally, our items ask about novel potential learning outcomes such as “I realized the skills I gained [in this context] will help me in my career” and “my confidence as a student increased,” to help students connect their learning in the classroom to the real world and make their own progress in learning more visible. Pilot tests were conducted for comprehensibility and the data collection process is in progress. Exploratory factor analysis will be used to group and reduce the 72 current items. In the exploratory factor analysis, we will focus on undergraduate engineering students who have completed a reflection activity in the past academic school year. We are using a stratified random sampling approach to ensure we document student learning from a diversity of reflection activities. Here, we report the results of an exploratory factor analysis, highlighting the reduced number of questions contributing to the validated survey. We have reduced the number of questions from 72 to 16, as described by four factors: Engineering Self, Course Understandings, Areas for Growth, and Social Impact. We report data on inter-item reliability and concurrent validity. The contribution of this work is evidence of preliminary validation for a survey that will allow the engineering community to learn more about what students learn from doing reflection activities and what settings or types of activities lead to specific characteristics of learning. students these wrappers. exam wrappers gains their students exam wrappers was planned. These trends show that the RKGI strongly measures learning designed factors, but also gives insights to other potential knowledge gains the including their reflection