Toward a shared meaning of the "impact" of engineering education research: Initial findings of a mixed methods study

In an environment of increased scrutiny and accountability, engineering education researchers are being called upon to describe and defend the tangible impacts of their work on a regular basis. The lack of scholarship within and beyond the field of engineering education on the impact of research contributes to the lack of shared language around what research impact looks like in this context—and a lack of productive conversations on how research can impact practice. Using a convergent parallel mixed methods research design, the aim of this study is to develop a valid framework that characterizes the impact of engineering education research, and describe engineering education researchers’ and practitioners’ perspectives on how research does and should influence practice in our context. Together, the two data forms enable greater insights on the problem than would be obtained by either type of data separately. This project started in Summer 2016 (one year ago). Data collection for the quantitative strand is well underway while data collection for the qualitative strand is complete. One of the key findings thus far is that the field of engineering education is closer to the development of a research impact framework than previously realized. Details on the research design and progress on the framework development will be discussed in this paper. Next steps will also be presented. Overview In an environment of increased scrutiny and accountability, engineering education researchers are being called upon to describe and defend the tangible impacts of their work on a regular basis. Having a shared language around the impact of engineering research will enable members of this community to effectively communicate it to diverse audiences and advocate for its support using consistent messages. Furthermore, a shared understanding among engineering education researchers and practitioners of what impact looks like in our field lends itself to more productive conversations on how research can influence practice. Unfortunately, there is very little scholarship within and beyond the engineering education community on how to characterize the impact of research [1]. Over the last decade, researchers in other fields have begun to take a scholarly approach to distill what impact looks like for them [e.g., 2-5]. This study serves as a comparable exercise for the field of engineering education while adding an emphasis on how research does and should impact practice in our context. Using a convergent parallel mixed methods research design [6], the two-fold aim of this study is to develop a valid framework that characterizes the impact of engineering education research, and describe engineering education researchers’ and practitioners’ perspectives on how research does and should influence practice in our context. In short, the quantitative strand of this project relies on card sorting and cluster analysis techniques to broadly and comprehensively define “impact” in the context of engineering education research. The resulting framework will define impact in ways that extend beyond the esoteric scientific impacts to include the social and societal impacts of our work. Additionally, this study includes interviews with various engineering education stakeholders (e.g., tenure-track faculty, lecturers and professors of practice, administrators, student advisors, co-curricular support personnel) to garner their insights on the current and ideal relationship between research and practice in engineering education. The data collection and analysis for both the quantitative and qualitative strands of this study are well underway. The quantitative and qualitative strands of this study will ultimately converge to reveal the extent to which the interview findings agree with and expand the data used to develop the research impact framework. The findings will also include practical recommendations for improving the relationship between research and practice in engineering education. Together, the two data forms enable greater insights on the problem than would be obtained by either type of data separately. The remaining sections of this document provides additional insights on the research design and the quantitative strand. Research Design & Rationale This mixed methods study explores and characterizes perceptions surrounding the impact of engineering education research. A convergent parallel mixed methods research design [6] is used, since it is the kind of design in which qualitative and quantitative data are collected in parallel, analyzed separately, then merged. Both forms of data collection and analysis are equally prioritized, collected simultaneously, and will be mixed at the end of the study to address the overarching research question. In the study, quantitative card sorting and survey data is being collected during workshops at engineering conferences and used as part of adopting an instrument development approach to framework development that characterizes the impact of engineering education research. Researchers will be accessible at targeted conferences, and these quantitative methods lend themselves to formal framework validation. The qualitative data, which includes interviewing various engineering education practitioners, explores their perceptions of the impact of the research, how research informs practice, and the kinds of research impacts stakeholders would like to see. There are three reasons for collecting and analyzing both qualitative and quantitative data in this study. 1) The two data sets allow us to answer different research questions that support an overarching research objective [6]. 2) Engineering education researchers and practitioners may have different views on the impact of research; collecting the two data sets creates the opportunity to explore the diversity of views and uncover relationships between the two perspectives [6-8]. 3) Including both qualitative and quantitative data enables us to triangulate the findings that stem from the respective sets of data [6-8]. Together, the two data forms enable greater insights on the problem than would be obtained by either type of data separately. The research questions guiding this study are: 1. What is a meaningful and shared description of the impact of engineering education research, according to engineering education researchers? (QUANT) 2. How do different engineering education practitioners –including non/tenure-track faculty, department chairs, co-curricular support personnel, engineering deans and engineering staff advisors— perceive the impact of engineering education research? (QUAL) 3. To what extent does the interview findings with practitioners agree with and expand the data used to develop a framework characterizing the impact of engineering education research? (MM) Quantitative Methods & Research Insights (To Date) The objective of the quantitative strand of this mixed methods study is to develop a valid framework for characterizing the impact of engineering education impact that is inclusive of international perspectives. The methodological lens for the quantitative strand of this mixed method study is an adaption of Messick’s unified theory for instrument development and validation [9]. Messick defined validity as “an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores or other modes of assessment” [9, p. 245]. Based on Messick’s theory, validity includes six different facets: content, substantive, structural, generalizability, external, and consequential. Messick’s model is widely used by psychometricians and has been translated by engineering education researchers Purzer and Cardella [6] into an instrument blueprint researchers can use to guide the development of items and the collection of validity evidence. Messick’s theory is typically used for standardized tests, but Purzer and Cardella’s blueprint reveals that the framework is much broader than a series of statistical analyses. It brings in activities like a literature review, review of existing instruments, stakeholder analysis, and content experts perspectives. The research impact framework that will result from this study employs this blueprint and will rely on multiple sources: 1) previous research data (i.e., content analysis of how principal investigators on federally-funded STEM education projects talk about the impact of their work) [1]; 2) review of existing research impact frameworks developed for other fields (i.e., [2-5]) perspectives of engineering education researchers and content experts via workshops and virtual card sorting activities. The workshops associated with the quantitative phase of this study are designed to get input from members of the global engineering education community on what should be included in the research impact framework, to validate to it, and to share guidelines on how to use it. To date, we have collected quantitative data from 50 participants via workshops at two engineering conferences and online using card sorting exercises (see Table 1). Card sorting [12] is a knowledge acquisition technique that is underutilized in engineering education, and is the method used in this study to garner input from the engineering education research community. Conference Name Date & Location Participants Australasian Association for Engineering Education Conference (AAEE) December 7-9, 2015 (Victoria, Australia) 5 American Society for Engineering Education Conference (ASEE) June 26-29, 2016 (New Orleans, LA) 11 Virtual Card sorting Round 1 Round 2 Online June 14 – July 14, 2016 January 20 – (Feb 28, 2017) 34 TBD (In progress) Scheduled: Research in Engineering Education Symposium (REES) July 6-8, 2017 (Bogotá, Columbia) TBD Scheduled: Research Impact Workshop Engaging Interdisciplinary Experts Fall 2017 (Arizona State Univ.) TBD T