Balancing Formative and Summative Science Assessment Practices: Year One of the GenScope Assessment Project.

This paper describes the GenScope Assessment Project, a project that is exploring ways of using multimedia computers to teach complex science content, refining sociocultural views of assessment and motivation, and considering different ways of reconciling the differences between these newer views and prior behavioral and cognitive views. The project is refining a computer-supported introductory genetics curriculum and the associated formative and summative assessment practices. Researchers then manipulate motivational conditions in an effort to maximize participation in formative feedback. In the first of three planned annual implementations, four secondary school life-science teachers implemented a month-long curriculum. A within-teacher/between class design contrasted three different assessment conditions: grade-oriented, standards-oriented, and accountability-oriented. Self-report and observational assessments of learning, engagement, and motivation were collected in 16 implementation classrooms and 2 comparison classrooms. The implementation classrooms show significantly larger gains on the new transfer measures and comparable gains on the far transfer measures. In the 26 students whose participation in formative feedback was videotaped and analyzed, engagement was strongly related to learning gains. However, overall engagement in formative feedback was disappointingly modest, and there were no noteworthy effects of the motivational manipulation on learning or motivation. Further modifications to the curriculum and formative assessment practices are being undertaken in response to these findings. Four appendixes contain sample assessments and scoring rubrics. (Contains 5 figures and 50 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document. Balancing Formative and Summative Science Assessment Practices: Year One of the Gen Scope Assessment Project U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) his document has been reproduced as received from the person or organization originating it. Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. 1 PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) Daniel T. Hickey° Learning & Performance Support Laboratory University of Georgia, Athens, GA USA Ann Cale Kruger Laura D. Fredrick Nancy Jo Schafer Georgia State University, Atlanta, GA USA Ann C. H. Kindfield Educational Designs Unlimited, Neshanic Station, NJ USA Abstract This project is (1) exploring ways of using multimedia computers to teach complex science content, (2) refining sociocultural views of assessment and motivation, and (3) considering different ways of reconciling the differences between these newer sociocultural views and prior behavioral and cognitive views. This paper summarizes these lines of inquiry and describes a project that explores the issues that emerge in their convergence. We are refining computer-supported introductory genetics curriculum and associated formative & summative assessment practices; we are then manipulating motivational conditions in an effort to maximize participation in formative feedback. In the first of three annual implementations, four secondary school life-science teachers implemented a month-long curriculum. A withinteacher/between class design contrasted three different assessment conditions (grade-oriented, standards-oriented and accountability-oriented). Self-report and observational assessments of learning, engagement, and motivation were collected in sixteen implementation classrooms and in two comparison classrooms. The implementation classrooms showed significantly larger gains on the near transfer measures and comparable gains on the far transfer measures. In the 26 students whose participation in formative feedback was videotaped and analyzed, engagement was strongly related to learning gains. However, overall engagement in formative feedback was disappointingly modest, and there were no noteworthy effects of the motivational manipulation on learning or motivation. Further modifications to the curriculum and formative assessment practices are being undertaken in response to these findings.This project is (1) exploring ways of using multimedia computers to teach complex science content, (2) refining sociocultural views of assessment and motivation, and (3) considering different ways of reconciling the differences between these newer sociocultural views and prior behavioral and cognitive views. This paper summarizes these lines of inquiry and describes a project that explores the issues that emerge in their convergence. We are refining computer-supported introductory genetics curriculum and associated formative & summative assessment practices; we are then manipulating motivational conditions in an effort to maximize participation in formative feedback. In the first of three annual implementations, four secondary school life-science teachers implemented a month-long curriculum. A withinteacher/between class design contrasted three different assessment conditions (grade-oriented, standards-oriented and accountability-oriented). Self-report and observational assessments of learning, engagement, and motivation were collected in sixteen implementation classrooms and in two comparison classrooms. The implementation classrooms showed significantly larger gains on the near transfer measures and comparable gains on the far transfer measures. In the 26 students whose participation in formative feedback was videotaped and analyzed, engagement was strongly related to learning gains. However, overall engagement in formative feedback was disappointingly modest, and there were no noteworthy effects of the motivational manipulation on learning or motivation. Further modifications to the curriculum and formative assessment practices are being undertaken in response to these findings. Balancing Formative and Summative Science Assessment Practices: Year One of the Gen Scope Assessment Project Presented at the annual meeting of the American Educational Research Association, New Orleans, April, 2002. This research is supported by the USA National Science Foundation Grant REC-0196225 to the University of Georgia. The opinions presented here belong to the authors and do not necessarily represent the positions of the University of Georgia or the National Science Foundation. We gratefully acknowledge the key contributions of many graduate assistants and educators on this project, including Novella Abbott, Bryon Davis, Jessica DeCuir, Bryon Hand, Brandon Kyser, Simona Laprocina, Monkieta Lucas, Morgan Nolan, Marcus Norman, Annette Parrott, John Price, Art Russell, Christine Yang, and Steve Zuiker. We also gratefully acknowledge the district administrators who helped coordinate our efforts and the student who participated in it. Daniel Hickey, Room 611 Aderhold Hall, Athens, GA, 30602, dhickey@coe.uga. edu. 1 0 EST COPY AVAILABLE This multifaceted project represents the convergence of three lines of inquiry. One line of inquiry concerns the use of multimedia computer technology to teach introductory genetics. The second line concerns the pragmatic implications of contemporary sociocultural views of assessment and motivation, particularly as they relate to motivating students to participate in formative assessment. The third involves the complex theoretical issues that emerge when attempting to reconcile the relationship between these newer sociocultural views and prior behavioral and cognitive views. Following is a summary of each of these lines and the research goals associated with each. Curricular Inquiry. The first line of inquiry concerns the use of multimedia computers to teach introductory genetics. Key genetics phenomena are not directly observable, and secondary genetics is often students' first formal exposure to probabilistic reasoning. These and other factors that make genetics difficult to teach and learn also made it a promising candidate to profit from classroom multimedia technology. Starting in 1991, with the support of the National Science Foundation, a team at BBN Labs (headed by Paul Horwitz, now at the Concord Consortium) began developing and refining software for teaching introductory genetics in middle and secondary science classrooms, (Horwitz, Neumann, & Schwartz, 1996; Horwitz & Christie, 2000). The resulting Gen Scope software has been widely acknowledged as a noteworthy example of the synergy between advances in educational technology and contemporary constructivist pedagogical principles (e.g., Bransford, Brown, & Cocking, 1999, Chapter 9). The learning environment afforded by Gen Scope is generally consistent with the software recommendations for K-12 educational technology issued by the President's Committee of Advisors on Science and Technology (PCAST, 1997). During a four-year collaboration funded by NSF's AAT Program (Grant RED-95-5348) Gen Scope's developers and a team including Dan Hickey and Ann Kindfield (initially at Educational Testing Service) implemented and evaluated Gen Scope in over 40 classrooms. This research found Gen Scope to be an effective tool for enhancing or supplanting conventional introductory genetics instruction. (Hickey, Kindfield, Horwitz, & Christie, 1999; 2000). The effort also yielded formative and summative assessment tools that are central to the present project. The former were shown to be very effective at improving learning in the Gen Scope environment; the latter was shown to be an effective tool for measuring learning in Gen Scope and conventional introductory genetics environments. Our primary curricular goal is refining the formative assessment tools developed in previous research. Our