Reflections on the Cognitive Interviewing Reporting Framework: Efficacy, Expectations, and Promise for the Future

Based on the experiences of three research groups using and evaluating the Cognitive Interviewing Reporting Framework (CIRF), we draw conclusions about the utility of the CIRF as a guide to creating cognitive testing reports. Authors generally found the CIRF checklist to be usable, and that it led to a more complete description of key steps involved. However, despite the explicit direction by the CIRF to include a full explanation of major steps and features (e.g., research objectives and research design), the three cognitive testing reports tended to simply state what was done, without further justification. Authors varied in their judgments concerning whether the CIRF requires the appropriate level of detail. Overall, we believe that current cognitive interviewing practice will benefit from including, within cognitive testing reports, the 10 categories of information specified by the CIRF. Future use of the CIRF may serve to direct the overall research project from the start, and to further the goal of evaluation of specific cognitive interviewing procedures. Boeije and Willis (2013) have argued that the effective conduct of cognitive interviewing requires the use of a systematic, complete, and harmonized system of reporting standards, and have introduced the Cognitive Interviewing Reporting Framework (CIRF) as a means for accomplishing this goal. As an attempt to apply and to initially evaluate the CIRF, the preceding three papers within the current issue each attempted to attend to the CIRF in a variety of ways (Bode & Jansen, 2013; Padilla, Benitez, & Castillo, 2013; Vis-Visschers & Meertens, 2013). Each of these authors adopted the CIRF categories to fashion a testing report, based on information from a cognitive testing project that had already been conducted. Based on their experiences, and on a review of their reports, we make conclusions concerning the Willis, G. , Boeije, H. Reflection on the cognitive interviewing reporting framework: efficacy, expertitions, and promise for the future. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences: 2013, 9(3), 123-128 This is a NIVEL certified Post Print, more info at http://www.nivel. eu utility of each of the 10 major CIRF checklist categories, and then turn to overall reflections on the CIRF. ASSESSMENT OF THE 10 CIRF CATEGORIES CIRF Category 1: Research Objectives It does appear that the authors of the cognitive interviewing studies within this volume – even when induced to utilize the CIRF – tended to neglect to include a complete description of the research objectives of the investigation. The major assertion made was that a questionnaire required evaluation, and it was assumed that cognitive testing was appropriate for achieving this end. Neither Bode and Jansen, nor Vis-Visschers and Meertens, explain why they felt cognitive testing was an appropriate method, but rather assumed this to be the case – perhaps because cognitive testing has become commonplace and well accepted as a means for evaluating survey questions. On the other hand, Padilla et al. began by describing the decision to include cognitive testing, perhaps because their aim was to contrast this with another method (psychometric analysis). The CIRF Research Objectives category also suggests providing a description of theoretical perspectives, and in particular to indicate how features of the evaluated questionnaire, and anticipated problems, drive the selection of the particular cognitive approach used. For example, if particular comprehension challenges are expected, the report will presumably mention that probe questions were fashioned to target these areas. However, the three reports featured in this volume tended to simply summarize their theoretical approach in a generic manner – that is, that cognitive testing will be used to examine comprehension, retrieval, judgment, and response processes, according to the standard Tourangeau (1984) model. There is of course merit to remaining open to a consideration of multiple potential sources of error, rather than overly narrowing one’s viewpoint or expectations at the outset. On the other hand, pretesting investigations can sometimes demand a degree of targeting. For example, a questionnaire on lifetime physical activity would seem to demand attention to retrieval processes, given the long reference period involved, and perhaps an approach to cognitive probing that focuses on that issue. We therefore believe that although cognitive interviewers should avoid being narrowly focused, concerning the way theory applies to their investigation, a description of particular hypotheses or expectations that are somewhat more specific may be appropriate, within the opening section of the report. As such, we continue to advocate the CIRF’s emphasis on providing information concerning why, and how, cognitive interviewing can be expected to satisfy the demands of the investigation. CIRF Category 2: Research Design The intent of the CIRF category describing Research Design is to describe the features of the overall design, and to describe why particular variants of cognitive testing parameters were selected. For example, testing of a selfadministered paper questionnaire involving complex skip instructions may call for purely retrospective verbal probing, in order to avoid interruption of navigational behavior that may occur if concurrent probing were chosen. We note that all three authors selected retrospective verbal probing. However, only Padilla et al. describe why they chose Willis, G. , Boeije, H. Reflection on the cognitive interviewing reporting framework: efficacy, expertitions, and promise for the future. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences: 2013, 9(3), 123-128 This is a NIVEL certified Post Print, more info at http://www.nivel. eu that technique; whereas Vis-Visschers and Meertens state that they chose it but without explanation as to why; and Bode and Jansen rely on a technique (the ThreeStep Test Interview, or TSTI), that appears to involve retrospective probing, but not by that name. Further, although all three investigations appear to have relied on scripted probes (i.e., cognitive probes fashioned prior to the interviews), it is not clear to what extent they also relied on more flexible forms of probing (those labeled spontaneous or emergent by Willis, 2005). Yet, the selection of such probes constitutes one of the more interesting, though contentious, issues in the field, and is important to describe in a testing report. We continue to argue that a full description of detailed design information is vital. First, this will be of use in creating a body of evidence concerning how cognitive testing is commonly conducted, in practice. Second, to the extent that authors provide information reflective of the efficacy of their investigations, it may be possible to make conclusions concerning the association between particular approaches and the type or quality of results obtained (e.g., the usefulness of concurrent probing versus retrospective when assessing navigational behavior on a web-based survey). CIRF Category 3: Ethics Although cognitive interviews generally do not present the same degree of ethical challenge as, for example a clinical trial involving disease therapy, ethical issues still pertain. Participants within cognitive interviewing studies almost never receive any direct positive benefit, as the entire endeavor is more geared toward the researcher’s goals (or those of private industry, in case of marketing surveys). Therefore, investigators should pay attention to how they respected the interests of participants. Further, they should provide information on basic issues such as whether the interviews were conducted in a private location, and whether individual responses were systematically protected. At the least, a statement should appear stating that the investigation was approved by an ethical review body such as an Institutional Review Board, or else was conducted in a manner consistent with the ethical requirements established by the testing organization. We observed that the manuscripts in the current volume were led, through use of the CIRF, to at least mention issues of research ethics. Vis-Visschers and Meertens state that attention to this topic was prompted explicitly by the CIRF, and that it would not normally be included in a report that adhered to their organization’s guidelines. Especially given the tendency of journal editors to increasingly request such information, it seems that inclusion of this topic in the CIRF is a useful contribution. CIRF Category 4: Participant Selection All three studies in this volume describe participant selection methods used in the way intended by the CIRF, although these did not universally capture both (a) how recruitment was done (procedurally), and (b) who was recruited (e.g., by providing a table of participant demographic characteristics). Further, despite the intention of the CIRF, no study commented directly (in any part of the report) on whether the individuals successfully recruited did serve to satisfy the purposes of the research, whether enough subjects were studied (e.g., whether saturation was attained), or whether there were ‘‘gaps’’ with respect to particular types of individuals who were observed. Willis, G. , Boeije, H. Reflection on the cognitive interviewing reporting framework: efficacy, expertitions, and promise for the future. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences: 2013, 9(3), 123-128 This is a NIVEL certified Post Print, more info at http://www.nivel. eu Somewhat in that vein, however, Padilla et al. note that: ‘‘. the CIRF obliged us to report on methodological issues that are often missed in cognitive interviewing studies. For example, in the Method section for Study 2 we reported demographics such as ‘‘type of household’’ and