Using Focus Groups in Preliminary Instrument Development: Expected and Unexpected Lessons Learned

In the context of conducting survey research, item generation is a critical step in instrument development. Often, the psychometric literature focuses on reliability and validity of items, rather than how the items emerged or evolved (Rowan & Wulff, 2007). Likewise, qualitative interviewing may be conducted in preparation for subsequent quantitative studies (Rowan & Wulff), yet serve as background rather than as the focus of discussion in scholarly publications. Sometimes interviews are integral to the overall research methodology, as in a mixed-methods design, and represent rich one-on-one data collection opportunities. Initial qualitative inquiries may serve to ground successive research in real-life situations and observations, thus enhancing the later processes and final outcomes (Padgett, 1998; Ulin, Robinson, & Tolley, 2005b). Documenting these early steps allows researchers to trace initial origins of items or language, as well as later iterations of emergent or evolving concepts. In addition, recording the early views makes it possible to evaluate those views within the often larger and subsequent research study as it unfolds. Early perspectives in qualitative research thus can simultaneously encourage self-reflective research practices. Among the rich benefits of conducting initial qualitative inquiries, is the potential to focus subsequent study on interview data, coupled with social interaction, to generate new ideas from the data themselves (Darkenwald, 1980). Thus, the initial inquiry is both exploratory and holistic (Saldana, 2009). In the case of our work, our end goal was to develop instruments examining college students' images and attitudes on scientists and science careers. As suggested by others, the best way to learn about student experiences is to ask the students' themselves (e.g., Singh, 2008), so we consulted actual college students as credible experts (Creswell, 2003) in developing our items. In this way, too, we endorsed the perspective that the reality of participants is socially constructed, and largely based upon past and present interactions within their respective school, family, and community environments (Racher & Robinson, 2002). We intended that our work reflect this participant worldview and use accurate language. In this paper, we will detail some of the key features of our focus groups and some of the new insights we gleaned about employing focus groups in item generation. Specifically, we will focus on frameworks and research questions; agenda setting and determining participation; data recording and field notes; and coding and other methodological issues. We will pay particular attention to a field note format previously used in clinical settings that we employed in our research process and that we believe has promise for future research initiatives. While there is a rich and recent literature that provides guidelines for conducting research utilizing focus groups, the notion that focus groups play a critical "pre-design" role has yet to emerge. This gap provided us with an opportunity to develop our own vision for the process, tasks, and outcomes from focus group activities. Our primary goal was to develop an updated quantitative instrument to capture students' stereotypes of scientists. Initially, the focus groups were designed to explore the content of the known stereotypes, to confirm or challenge items that had been used for decades to measure stereotypes of scientists. Because our research focus was on instrument development and psychometric evaluation of a set of quantitative scales, it would have been time prohibitive, unrealistic, and not particularly useful for us to employ, for example, extensive transcription, multiple coding, and analytical processes that often accompany qualitative research utilizing focus groups. Yet, because an updated scale required exploring the sometimes contested territories of stereotypes, it quickly became clear that the details of the focus group discussions, what was said and unsaid, were of potential value to the overall project. …