How Valid Are the Responses to Nursing Home Survey Questions? Some Issues and Concerns

Although it is a well-recognized standard in high-quality survey development, cognitive-based testing of surveys has not been standard practice in long-term care research or has not been commonly reported in research publications. Although focus groups and pilot testing have been routinely utilized, few nursing home (NH) studies have reported including cognitive testing as part of survey development (e.g., see Frank, Flynn, & Rothman, 2001; Housen et al., 2008; Katz et al., 2009). Cognitive-based testing is the systematic testing of survey questions intended to determine how respondents understand the question and the thought process involved in providing an answer (Presser et al., 2004). This form of testing is used to establish how respondents understand and interpret questions, to determine how and why they choose specific responses, and to identify questions that are difficult for respondents for a variety of reasons. The main objectives of cognitive-based testing are to reduce measurement error, increase validity, and improve response rates (Jobe & Mingay, 1989) by identifying instances where researchers have different interpretations of terms or concepts from those of respondents. Unlike pilot testing, cognitive-based testing examines the assumptions researchers may make about how terms are defined and understood without realizing that these assumptions may not be shared. Cognitive-based testing is also useful for determining those instances where respondents may simply not possess the information the researcher is seeking to collect. Identifying differing interpretations of terms and concepts regarding philosophies and systems of care may be especially important when new trends emerge and then evolve within the field, such as NH “culture change” and “palliative care.” For example, the Consumer Assessment of Health Providers and Systems survey was originally designed to include measures of “coordination of care” and “shared decision making,” but these constructs ultimately were not included in the final survey because uniformly understood questions could not be developed (Levine, Fowler, & Brown, 2005). Furthermore, researchers, clinicians, administrators, and consumers may have different perspectives or understanding of such complex concepts. A recent study found that those in the long-term care field who were not directly involved in the provision or study of palliative care knew little about its philosophy or strategies (Lima, Miller, & Shield, 2009). Cognitive-based testing can play an important role in developing surveys that more accurately measure practitioners’ understanding of and response to emerging concepts. Determining where mismatches occur between the interpretation of terms and concepts by researchers and those of long-term care professionals may also have important implications for future research and policy and may explain discrepant findings in past research. For example, numerous studies have examined turnover among NH employees and found widely varying results (Barry, Kemper, & Brannon, 2008). A review of the turnover literature by Castle (2006) revealed a wide range of reported turnover rates for nurse aides (14% to 346%), licensed practical nurses (LPNs, 8% to 103%), and registered nurses (RNs, 19% to 64%). This may be related in part to discrepant interpretations of the concept of “turnover” between those designing surveys and those responding to surveys or may be related to other measurement issues not previously discovered. The purpose of this study was to cognitively test two instruments, one designed to be administered to Nursing Home Administrators (NHAs) and one to directors of nursing (DONs) in a national study. We sought to ensure that survey respondents understood questions as intended, to determine whether the NHA or the DON was best able to answer questions on certain topics, and to inform the answer choices provided for questions. In doing so, we also identified a number of issues that raise concerns about the validity of previous questionnaires and the assumptions implicit in some long-term care policy.