The Challenges of Producing Evidence-Based Claims: An Exploratory Study of NSF's Math and Science Partnership Community. WCER Working Paper No. 2009-4.

s accepted for presentation (N = 47) that were provided before the meeting and (b) data collected during the meeting, consisting of interviews (N = 68), observations of all breakout sessions (N =26), and think-piece exercises (N = 98) written by participants. The data-gathering instruments and the LNC call for abstracts are included in Appendices A and C, respectively. A group of 10 researchers from Westat (see acknowledgements in footnote 2) provided invaluable assistance in gathering data during the meeting. The analytic procedures for our research included the use of a holistic scoring rubric for the LNC abstracts and inductive analyses of the interview, observation, and think-piece data using a structured approach to grounded theory (for details, see Appendix B). This report presents findings from this analysis, which we hope will be of value to NSF, MSP grantees, and the broader STEM education community. We emphasize that our findings must be interpreted in light of four limitations. First, interviewees’ self-reports, on which our findings largely depend, may be subject to social desirability bias; that is, interviewees’ responses may be biased toward giving socially acceptable answers. Second, self-reported experiences recalled from memory may be filtered or modified by the very act of narration. Third, our sample represents a stratified sample of the entire MSP population, limited to those individuals who attended the 2008 LNC meeting. Thus, this selfselected population is not representative of the larger MSP community and includes disproportionate numbers of certain position types (e.g., project directors, junior staff, PIs, etc.). Finally, we used a semistructured interview protocol, which has both strengths and limitations. Its strengths lie in the validity of the data—interviewees freely choose to present what is most salient, important, or memorable to them, rather than responding, as in forced-choice questionnaires, to specific questions that a researcher believes are important. Weaknesses lie in the fact that respondents’ failure to mention a topic does not imply that the topic was not relevant or of concern to them. Because our thematic findings are based on comments volunteered by respondents, we cannot infer the perspectives of respondents who did not think to mention each specific theme. Thus, it is not appropriate to provide quantitative data at the level of detail that is justified when reporting on forced-choice responses. However, to provide readers an understanding of the relative numbers of respondents who volunteered specific points, we use the following verbal quantifiers: 2–3 respondents = few 4–9 respondents = several 10–30 respondents = many All but a few respondents = generally 4 A 48 accepted abstract, submitted by one of the authors of this report, was omitted from our analysis due to conflict of interest. 5 The think-piece activity was designed to elicit descriptions of factors that affected participants’ efforts to make evidence-based claims. Some 320 individuals (including presenters and observers) attended the 2008 LNC. Although all were asked, only 150 completed think pieces. Of these individuals, only 98 provided the kind of information needed for inclusion in this study. The 52 think pieces omitted from the data set provided only descriptive accounts of projects with no discussion of factors that affected project evaluation efforts.