Does Qualitative Evaluation Measure Up?

 

 

Kathleen S. O’Connor1

Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics 

3311 Toledo Road, Hyattsville, MD 20782-2003, koconnor1@cdc.gov

Key words: qualitative evaluation, pretest, children with special health care needs, National Survey of Children with Special 

Health Care Needs 

Overview 

Qualitative research can be broadly characterized as the in-depth detailed study of a small number of cases primarily using 

subjective observational techniques (Ragin 2004); examples include cognitive interviews, expert review, observation of field 

interviews, and interviewer and supervisor debriefings. These techniques are frequently used to critically examine the 

application and foundation of theory, build new theories, generate or support hypotheses, investigate odd cases or cases in the

fringes of the distribution, and ‘obtain information on meaning, affect, and culture’, thus rendering ‘the facts understandable’ 

(Ragin 2004). Although qualitative evaluation methods are subjective, they can provide valuable contextual information that 

might otherwise be unattainable. 

This paper describes the role and impact of qualitative evaluation in refining the content and design of the 2001 National 

Survey of Children with Special Health Care Needs (NS-CSHCN) in preparation for a repeat survey in 2005 – 2006 

(hereafter referred to as the ‘2005-2006 NS-CSHCN’ or ‘2005-2006 survey’). In order to improve the second iteration of the 

survey, qualitative tools were employed to elicit the perceptions of survey methodologists, subject matter experts, 

researchers, and parents of children with special health care needs (CSHCN). Qualitative techniques used in the pretest phase 

of the 2005-2006 NS-CSHCN included expert review, cognitive interviews, interviewer and supervisor debriefings, and 

observation of field interviews. Through these different avenues project staff identified omissions and areas with lack of 

clarity. This revision process incorporated elements of the total survey design paradigm, which may ultimately result in 

improved CSHCN research (Lessler & Kalsbeek). 

This paper uses the following conceptual format: first, the design characteristics and impact of the 2001 NS-CSHCN are 

described; then, for each qualitative tool, the application of the technique in the pretest phase of the 2005-2006 NS-CSHCN 

and examples of problems illuminated and resolved are discussed. The goal of this paper is to illustrate examples in which 

qualitative evaluation techniques have been applied, and document the changes that have occurred as a result. Table 1 

summarizes the qualitative techniques used and the concomitant outcomes of using the techniques. 

Description and Impact of the Survey 

The first NS-CSHCN was conducted from October 2000 to April 2002 via the State and Local Area Integrated Telephone 

Survey (SLAITS) mechanism of the Centers for Disease Control and Prevention’s (CDC) National Center for Health 

Statistics (NCHS). This population-based, list-assisted random-digit-dial (RDD) telephone survey, sponsored by the 

Maternal and Child Health Bureau (MCHB) of the Health Resources and Services Administration (HRSA), screened over 

373,000 children under 18 years of age for the presence of a special health care need in over 196,000 households nationwide 

(Blumberg 2003). It was conducted to provide prevalence estimates of CSHCN, a comprehensive array of data on child and 

family experiences with the health care system, a description of the types of services needed and used by CSHCN, and 

assessment of unmet needs. The data are intended to assist program planners and policy makers and to track a variety of 

initiatives such as Healthy People 2010 and State-level Title V performance measures (Blumberg 2003, van Dyck 2002). 

 

1

 The content of this paper reflects the opinion of the author, and not that of the Department of Health and Human Services, 

the Health Resources and Services Administration, the Maternal and Child Health Bureau, the Centers for Disease Control 

and Prevention, and the National Center for Health Statistics. 

The author would like to especially thank Marcie Cynamon, Stephen Blumberg, Eve Powell-Griner, and Jennifer Madans for 

their insightful comments, as well as the Maternal and Child Health Bureau for funding the survey. 
This is the first survey to collect reliable, uniform data on special health care needs at the State and national levels using a 

standard screening battery, the CSHCN Screener (Bethell 2002). This battery operationalizes a definition of special needs 

that is contingent on the need for elevated service use and the level of functioning, rather than the presence of a diagnosed 

chronic health condition. It serves to categorize a diverse set of children with an array of conditions spanning the continuum

of severity, and includes children who may have other needs beyond medical services such as social services, mental health 

needs, early intervention services, or vocational education (McPherson 1998, McPherson 2002, Bethell 2002). 

Although the 2001 NS-CSHCN provided useful and comprehensive data, several issues emerged that prompted evaluation of 

the 2001 instrument and methods prior to future implementation. These issues included the need to broaden the description 

of the study population and the impact of the special need, and examine the validity of the CSHCN screener across 

demographic subpopulations. Because no other data source provides this breadth of information for CSHCN at the national 

and State levels, it was critical to address these areas with sound research using the full complement of techniques. A 

balanced approach considering cost, time, and consistency using multiple evaluation methods (which relied heavily on expert 

review and input) was adopted. 

Qualitative evaluation technique 1: Extensive expert review to determine the optimal method(s) to describe CSHCN 

Statement of the problem. A fundamental goal of the survey is to provide an accurate description of the CSHCN 

population, and it became clear that the ability to describe the CSHCN population needed to be broadened in the 2005-2006 

survey. Several constructs needed to be added to the survey (for example, questions on the effective control of the condition

did not exist in the 2001 survey). After the 2001 survey data were disseminated, the demand for descriptive information on 

the CSHCN population increased. Additionally, several findings from the 2001 survey seemed higher or lower than expected 

(for example, parents reported nearly 40% of CSHCN were never affected in their ability to do things other children their age 

do) (USDHHS 2004). It was felt that more information was needed to adequately address these concerns. 

Qualitative method chosen to address this problem. The qualitative technique of expert review (described below) was 

chosen to address these deficiencies, and expand the ability to describe the CSHCN population. 

Description of the expert panel. A Technical Expert Panel (TEP) was convened to review the first instrument and 

procedures. Content and survey methodology experts were assembled from academia (the medical, public health, and health 

services research communities), State and Federal Title V program directors, advocacy groups, the Association for Maternal 

and Child Health Programs (AMCHP), the American Academy of Pediatrics (AAP), and MCHB and SLAITS staff. The role 

of the TEP in preparation of the second survey was to prioritize content and offer recommendations on survey content and 

methodology. Simultaneously, the TEP was sensitive to the differing data needs of the survey’s constituent groups 

(policymakers, researchers, parents, etc.). The group developed a methodological research plan which would be 

implemented over a nine month pretest period. Considerable attention was paid to the impact of changes on survey 

estimates. 

Expert review to evaluate alternative theoretical approaches. The TEP identified two theoretical frameworks to broaden 

the description of the study population and the impact of the special health care need. The frameworks were (1) a diagnostic 

approach listing chronic conditions (similar to the approach used in the International Classification of Disease (ICD-9)); and 

(2) a functional or categorical approach using a battery of questions to characterize the broad level of functioning. A more 

detailed discussion of the issues involved in defining and classifying chronic health conditions have been written by Bethell 

(2002) and Perrin (1993). 

During this discussion TEP members evaluated the methods considering validity, utility, and credibility; as well as the 

feasibility of each approach. For the 2005-2006 NS-CSHCN, the TEP considered the inclusion of components from the 

International Classification of Functioning (ICF), specifically the International Classification of Functioning, Disability, and 

Health Children-Youth (ICF-CY). This standardized classification scheme is more inclusive and focuses on body functions, 

body structures, activities, participation, and environmental and personal factors (WHO 2001). Because experts remained 

divided on the use of a list of conditions, an abbreviated condition list was adopted in conjunction with elements from the 

ICF-CY framework to improve the description of the population. 

Expert review to evaluate the use of a referent group. The 2001 survey did not include questions on children without 

special health care needs. This was considered a shortcoming because it hampered the ability to fully describe CSHCN by 

comparing this population to children without special health care needs. After extensive discussion, the TEP recommended 

2
the use of a national referent sample of 5,000 children for the 2005-2006 NS-CSHCN for its simple design, cost, and to 

ensure adequate sample sizes for detailed analyses. The use of this ‘referent’ sample allows a direct comparison of children 

without special health care needs to CSHCN (although the screening procedure is slightly different between the referent 

group and the main NS-CSHCN sample to allow a methodological test of an alternate screening procedure). The TEP was 

primarily concerned with the salience and acceptance of some of the questions for the referent population, as well as the 

feasibility of this approach. The 2005-2006 NS-CSHCN dress rehearsal was used in part to gauge the procedure, and 

acceptance of the instrument with respondents who have children without special health care needs. The dress rehearsal is 

described in the final section of this paper. 

Results of the expert review. Two methods are employed in the 2005-2006 NS-CSHCN instrument to better describe the 

population: (1) the use of a referent group to facilitate comparison of the characteristics of CSHCN to children without

special health care needs, and (2) the use of an abbreviated list of medical conditions with a list of functional status questions 

that address components of the ICF-CY. In this example, the use of qualitative evaluation identified two complementary 

robust methods that will be used to enhance the description of CSHCN. 

Qualitative evaluation technique 2: Extensive stakeholder and expert review to prioritize and resolve instrument 

revisions 

Statement of the problem. The data user and researcher communities, as well as in-house data analyses conducted using the 

2001 data file, identified a number of problematic questions and questionnaire sections in the 2001 instrument. 

Qualitative method chosen to address this problem. The technique of extensive stakeholder and expert review was chosen 

to address this problem. The previously described TEP, a multidisciplinary panel of experts, provided expert individual and 

group consultation to prioritize and resolve proposed instrument revisions, in addition to determining the optimal methods to 

describe the CSHCN population as described in the first section. Comments were also solicited from individual data users 

and researchers as described below. 

Stakeholder and expert review of the instrument. As plans became more concrete for the second administration of the 

survey, SLAITS staff felt it would be useful to assemble comments from stakeholders on what content was critical to 

preserve and what could change, what needed enhancement, and what could be abbreviated. A questionnaire was sent to 

SLAITS data users and AMCHP members to solicit structured input for the TEP. Although the questionnaire used in the first 

survey was comprehensive in content, the stakeholders identified a number of analytic and substantive limitations. 

Throughout the revision process TEP members were invited to provide expert review of the instrument. Small TEP subgroup 

discussions were eventually convened to allow for more frequent focused discussions on selected topics than could be 

accommodated in a full TEP meeting. During the individual and subgroup review processes, TEP members were instructed 

to mull over the instrument keeping in mind the following general criteria: 

• validity 

• analytic use of the data by researchers and analysts 

• length, scope, and order of questions and sections of the instrument 

• overall interview length and the cost of adding survey questions 

• utility of the question and/or section from a State-level perspective 

• parsimonious word choices to capture complex topics 

• context effects, and 

• skip patterns. 

TEP members were careful to preserve trend analyses when practicable, and provided justification for recommended changes. 

Whole-scale changes to the instrument were entertained only if the questions or section yielded exceedingly poor quality data 

in 2001. 

Results of the stakeholder and expert review. The short questionnaire circulated to data users and AMCHP members 

identified many minor issues with questions throughout the instrument. Major issues were also identified and centered on a 

need to reconceptualize three concepts assessed in the survey: care coordination, transition to health care providers who treat 

adults, and community based services. Specific conceptual deficiencies included: 

3
• Care coordination: The assessment of care coordination resulted in lower-than-expected findings when examining 

the 2001 data. The 2001 survey revealed that only 1 in 9 children have parents who reported that care coordination 

was needed. Additionally, it was felt that the 2001 care coordination section did not adequately measure key 

constructs. 

• Community based services: This section did not collect adequate information. 

• Transition to care providers who treat adults: This section in the 2001 instrument did not address key constructs. 

Specific section deficiencies included the following: 

o Four questions were added to the 2001 NS-CSHCN after nearly nine months of data collection to assess 

services necessary for youth ages 13 – 17 years old to make the transition to health care providers who treat 

adults. Because of this lag, the 2001 survey can only be used to examine a small number of children with 

affirmative responses which led to unstable estimates at the State level. 

o In the 2001 instrument, the transition section contained a vocational education question that behaved 

differently than the other three questions. 

Stakeholders preferred that the 2005-2006 survey questions ask about the presence or absence of a written transition care 

plan, instead of just asking about communication regarding these topics (as phrased in the 2001 survey). 

4

Before sending the questionnaire to data users and AMCHP members, project staff was unaware of some of the issues which 

were raised in this review. If this short questionnaire had not been circulated, it is very unlikely that some of these revisions 

would have been made (especially regarding the transition section). These changes to the instrument improved the 

description of this population. Content validity was improved (that is, that the question batteries adequately measure the 

domain that it is supposed to), as well as construct validity (that is, that the respondents understand the overall 

question/battery intent). In this example, the use of this qualitative evaluation technique identified missing key constructs,

the need to reconceptualize key concepts, and identified problematic questions and question batteries from the data user and 

expert perspectives. 

Qualitative evaluation technique 3: Cognitive interviews to assess the translation and cultural validity 

Statement of the problem. Analysis of data from the 2001 NS-CSHCN revealed that the prevalence of special health care 

needs was lower among Hispanic children than for all other ethnic and racial groups with the exception of Asian children. 

Additional findings of concern include (Read & Bethell (2004)): 

• Hispanic children whose parents were interviewed in Spanish had a much lower positive screening rate for the 

CSHCN screener than parents who responded in English. 

• The language of interview appeared to play a larger role than ethnicity in how parents responded to the screening 

questions, as language was a significant and strong predictor of special health care need status even after accounting 

for variations in demographics, health status, and health care access. 

• Results for a screener question on prescription medication varied by ethnicity; it was the only screening component 

in which ethnicity and language of interview were both significant. 

Qualitative method chosen to address this problem. The cognitive interview technique was chosen to better understand 

and address these problems. Individual interviews were conducted by experienced trained interviewers with a small number 

of volunteer limited English proficient Hispanic parents of CSHCN. The use of the cognitive interview technique permitted 

the extensive cognitive testing of the CSHCN screener and related concepts, as well as an in-depth assessment of the

instrument translation. 

Description of the cognitive interviews. Nineteen face-to-face cognitive interviews were conducted with limited English 

proficient Hispanic parents of CSHCN in two sites (Boston, MA and Portland, OR). The cognitive interviews were 

conducted to: 

• learn how parents of Hispanic children understand and respond to the CSHCN Screener, 

• assess the translation of the Screener, 

• probe possible explanations regarding the screener results found among Hispanic CSHCN, 

• examine concordance between the researcher and parent interpretation of various screener items, and 

• assess cultural validity of various topics (such as ‘health’, ‘health problem’, ‘prescription medicine’, ‘functional 

limitations’, and ‘elevated service need/use’). 
Participants were asked a standardized list of questions, and various cognitive interview techniques such as ‘think aloud’, 

paraphrasing, and specific, general, and open-ended probes were also employed during the one-hour interviews (Willis 2005, 

Beatty 2004, Read & Bethell 2004). A limited bilingual researcher collected behavior coding data on hesitations, confusion, 

and requests to repeat questions. Two bilingual native Spanish speaking interviewers (a physician-researcher, and a 

professional Spanish interpreter and radio commercial voice actor) conducted all of the interviews. After each cognitive 

interview, the researchers gathered for a debriefing interview to confirm findings, clarify points, and discuss the implications 

of the findings for future interviews (Read & Bethell, 2004). The qualitative results described below cannot be extrapolated 

to the entire population of CSHCN or Hispanic CSHCN, and describe only the subpopulation from which they originated. 

5

Results of the cognitive interviews. Selected results from the cognitive interviews include (Read & Bethell 2004): 

• Many limited English proficient Hispanic parents of CSHCN stated they would be less likely to talk about their 

child’s health over the phone due to the sensitive nature of the overall topic (i.e., their child’s health). 

• Mirroring a 2001 NS-CSHCN result, CSHCN identified during the Spanish language interviews were less likely to 

qualify on the prescription medication screening component. 

• One of the screener questions (“Does your child/Do any of your children need or use more medical care, mental 

health, or educational services than is usual for most children of the same age?) is a complex question especially for 

Hispanic parents with low levels of education. The length and wording of this question was problematic for some

parents. 

• Parents did not express any difficulties or problems with the following items: 

o the current Spanish translation 

o the definition of ‘prescription medication’ 

o the concepts of ‘need’ for services versus ‘use’ of services 

o comparing each child’s abilities to those of other children the same age 

o the screening question regarding their child’s need or use of physical, occupational, or speech therapies, or 

the emotional, developmental, or behavioral component, and 

o the distinction between medication use for acute versus chronic illnesses. 

The cognitive interview findings were assessed by considering face validity, clarity, utility, and credibility. These findings

provided an illustrative contextual backdrop to aid decision-making by the methodologists and survey sponsor. In order to 

preserve the ability to conduct trend analysis, the TEP did not recommend revision of the screener questions. In this 

example, this qualitative technique revealed illustrative findings that most likely would not have been revealed without the 

use of this method. 

Qualitative evaluation techniques 4 and 5: Interviewer and supervisor debriefings and observation of field interviews 

to assess question revisions, and acceptance of the instrument among parents of children without special health care 

needs 

Statement of the problem(s). Project staff needed to test the instrument and determine whether parents of CSHCN would 

understand the questions. Additionally, we needed to know if parents of children without special health care needs in the 

referent sample would understand, accept, and answer the questions even though they feel the questions may not apply to 

their child. 

Qualitative methods chosen to address these problems. Two techniques were chosen to better understand the acceptance 

of the questions among parents of children without special health care needs, as well as to observe issues that arise among 

parents of CSHCN: observation of field interviews and interviewer and supervisor debriefings. Three experts observed field 

interviews conducted during ‘real-time’ interviewing. This option allowed experts to receive immediate unfiltered feedback 

directly from the respondents on problem questions, transitions, sections, and/or procedures. This technique can also be used 

to assess task difficulty, monitor response time, and gauge respondent interest. 

The interviewer and supervisor debriefings were focus group discussions conducted during the dress rehearsal. This 

technique allows for the rapid collection of data to identify problems from the interviewer and supervisor perspectives. This 

technique also can be used to identify problematic questions, sections, transitions, translations, and task difficulty, response 

time, and respondent interest. 

Description of the dress rehearsal. Both techniques were implemented during the November 2004 dress rehearsal. The 

dress rehearsal for the 2005-2006 NS-CSHCN was conducted to test the revised instrument, procedures, and computer 
assisted telephone interviewing (CATI) programming; provide interviewers and supervisors hands-on experience with the 

revisions; monitor acceptance of the instrument among respondents regarding children without special health care needs in 

the referent sample; and provide preliminary estimates of timings and data distributions that might be expected from the main 

survey. The dress rehearsal was particularly ideal because of the immediate feedback that can be garnered, i.e., interviews 

are conducted in ‘real-time’ and are unfiltered for observers. 

During the dress rehearsal, five hundred ninety interviews were completed in the main sample and 195 interviews were 

completed in the referent sample. The respondent was the person who was most knowledgeable about the health and health 

care of the child (in almost all cases this was a parent or guardian). 

Observation of field interviews. Three expert researchers from the National Center for Health Statistics observed the 

interviewer and supervisor trainings, as well as several days of field interviews conducted at the contractor’s telephone 

calling centers located in Chicago, Illinois and Las Vegas, Nevada. The interviews were monitored at the beginning of the 

dress rehearsal data collection. 

Results of the observation of field interviews. The observation of field interviews revealed the following selected 

findings: 

• parents pondered and answered the screener questions very deliberately; 

• most of the parents were easily able to answer the condition questions; 

• the health condition question allowing verbatim responses at the end of the listed conditions gathered data on rare 

congenital diseases; 

• several parents expressed difficulty answering screener questions regarding whether the condition would last 12 

months or longer (for example, if the child was receiving psychological or psychiatric counseling to handle a 

traumatic situation that happened in the past); 

• the functional difficulty questions were easily answered by most parents; however, this was not the case if the child 

had health care needs that change frequently, or if the child experiences varying levels of difficulty (for example, a 

child who does not take medication on a particular day may have more difficulties, or a child with environmentally 

induced allergies to food may have occasional but severe limitations); and 

• interviewers needed to be trained to: 

o answer questions from parents of children without special health care needs in the referent sample who 

expressed concern, boredom, or exasperation; 

o handle protests from parents of a child/children with special health care needs if a child without special 

health care needs who also lives in the household was chosen as the subject of the referent interview; 

o be prepared to handle respondents who are upset about a health-related current event not related to the 

survey (For example, the dress rehearsal was conducted during the midst of the influenza vaccination 

shortage. Various respondents vented to the interviewers especially during the screening process; and 

although experienced interviewers handled the situation in a very professional manner, this could have been 

an unnerving experience for a novice interviewer). 

6

Results of the interviewer and supervisor debriefings. The interviewer and supervisor debriefings, conducted in late 

November 2004 by trained contractor project staff, revealed the following selected findings: 

• some interviewers thought it was easier to complete an interview in the main sample than the referent sample, but 

others felt it was easier to complete an interview in the referent sample; 

• the vast majority of parents of children without special health care needs did not tire of answering questions targeted 

to CSHCN; 

• multiple definitional and interpretation issues were revealed related to access to care, utilization, and unmet needs, 

satisfaction with care, and health insurance; 

• parents were eager to tell the interviewer the ‘story’ of their child; listening to those stories helped to build rapport 

between the interviewer and the parent, and helped the interviewer recognize when parents needed clarification to 

accurately report their children’s experiences; 

• interviewers thought that it was critical for respondents to understand why this research is so important; and 

• most of the interviewers and supervisors had experience working on either the 2001 NS-CSHCN, the National 

Immunization Survey, or other SLAITS modules. 
Overall results of these techniques. The main complementary finding of the interviewer and supervisor debriefings and the 

observation of field interviews is that the instrument seemed to be accepted by parents of children without special health care

needs in the referent sample. Some transition statements had to be added to the instrument to accommodate this group. 

Parents of CSHCN identified a few issues with questions and transitions; changes to address these minor issues were 

incorporated into a subsequent version of the instrument. These findings were assessed using face validity, utility, and 

clarity. These qualitative techniques elucidated information that would have otherwise been unattainable. 

7

Conclusions 

In the pretest phase of the 2005-2006 NS-CSHCN, did qualitative evaluation measure up? In this instance, the question 

should be answered in the affirmative. The various stages of qualitative assessment complemented, but did not replace, 

quantitative analyses. Qualitative methods were used to provide rich information which significantly altered the final 

product. Although qualitative work may be considered by some to be subjective and not as rigorous as quantitative methods, 

useful information that was otherwise elusive was gleaned by employing these techniques. Results from the dress rehearsal 

determined that modifications implemented as a result of the qualitative findings produced an improved instrument and 

methodology and a stronger overall survey, although the establishment of trend lines had to be sacrificed for several variables

as a result. The qualitative techniques discussed above appear to have enhanced content and construct validity. The use of a 

referent group is a novel addition that will help address some of the limitations identified in the 2001 survey. Each 

qualitative technique contributed a critical piece of information, such as: 

• the cognitive interviews bolstered our understanding of key concepts in the Hispanic population, and verified the 

quality of the translation; 

• the TEP discourse provided a broad philosophical, research-oriented, clinical, and practical backdrop and allowed all 

of the proposed instrument, question, and methodology options and revisions to be considered in a very thorough 

manner with both technical and subject matter experts; 

• the expert and stakeholder reviews pointed to specific questions and sections for targeted improvement, and 

identified theoretical approaches that would enhance the survey; 

• interviewer and supervisor debriefings confirmed that the survey questions were acceptable (for the most part) to 

parents of children without special health care needs, and revealed cognitive issues with various questions; and

• the observation of field interviews conducted during the dress rehearsal confirmed the interviewer and supervisor 

debriefing findings; offered a real-time practical evaluation of the revisions; and affirmed the dedication and 

commitment of the interviewers and project staff. 

The 2005-2006 NS-CSHCN is currently in the field and data will continue to be collected through 2006. The impact of the 

pretest revisions, as well as the overall data file, will continue to be vetted to monitor quality, validity, reliability, and 

consistency. 
Table 1. Summary of the qualitative techniques used in the pretest phase of the 2005-2006 NS-CSHCN. 

8 

Cognitive interviews Interviewer and Supervisor 

Debriefings 

Extensive expert review Observation of field interviews 

Definition • Individual interviews conducted 

by experienced specially trained 

interviewers with a small 

number of volunteers from the 

target population of interest 

• Used to assess the cognitive 

process, and how potential 

respondents understand, 

interpret, and answer individual 

questions, batteries, and/or key 

constructs 

• Also used to assess task 

difficulty, hidden problems, 

response time, and respondent 

interest 

• Includes the use of exhaustive 

think-aloud and various verbal 

probing techniques 

• Focus group discussion with 

interviewers and supervisors 

conducted during the time 

period of the dress rehearsal 

• Allows for the rapid collection 

of data to identify problems 

from the perspective of the 

‘front line’, i.e., the 

interviewers and supervisors 

• Used to solicit impressions of 

the questionnaire; identify 

problematic questions and 

issues; also used to identify 

what works well 

• Also used to assess task 

difficulty, response time, and 

respondent interest 

• Includes the use of various 

verbal probing techniques 

• Convened panel of 

multidisciplinary experts to 

evaluate the draft instrument 

and methodology 

• Prioritize options to address 

problems in the proposed 

instrument and methodology 

• Provide overall expert guidance 

• Field monitoring of interviews 

(while the interviewer is 

conducting real-time telephone 

interviews) 

• Used to monitor interviewer 

behavior (adherence to protocol 

and interview script, flow of the 

interview, behavioral coding, 

etc.) and respondent 

perceptions, reactions, and 

responses 

Brief 

non-exhaustive 

literature review 

Converse & Presser (1986) 

Fowler (2001) 

Beatty (2004) 

Groves et al (2004) 

Presser et al (2004) 

Willis (2005) 

Converse & Presser (1986) 

Groves et al (2004) 

Presser et al (2004) 

DeMaio & Landreth (2004) 

Groves et al (2004) 

Nelson (1985) 

Groves et al (2004) 

Presser et al (2004) 

How we used this 

technique 

• Permitted extensive cognitive 

testing of the CSHCN Screener 

and related concepts among 

limited English proficient 

Hispanic parents of CSHCN 

• Solicited opinions and 

suggestions from native 

Spanish-speaking bilingual and 

English-language interviewers 

and supervisors on the 

translation, topical areas, 

questions, transitions, etc. 

• Explored operational and 

computer program issues (skip 

pattern and text fill errors, etc.) 

• Provided overall expert guidance 

• Provided expert individual and 

group consultation for proposed 

revisions to the content and 

methodology 

• Small TEP subgroup discussions 

allowed for more frequent 

focused and detailed discussions 

on selected topics than could be 

accommodated in a full TEP 

meeting 

• Offered and prioritized 

recommendations related to the 

content and methods 

• Provided rich unfiltered 

observations directly from the 

interviewers, supervisors, and 

survey respondents 

• Allowed ‘real-time’ test of actual 

procedures and the computer 

program 

• Identified problem questions, 

transitions, sections, and/or 

procedures 
• Assessed findings from the last 

survey to identify areas of concern 

• Reviewed and prioritized 

comments from data users 

• Provided oversight for the pretest 

phase and dress rehearsal 

• Provided strategic guidance 

• Matched survey objectives with 

data needs and the instrument to 

ensure the survey will fulfill its 

objectives and provide useful data 

9 

Outcome (impact 

on the instrument 

and/or survey 

methodology) 

• Provided rich contextual data 

• Bolstered methodological 

understanding regarding the 

population of limited English 

proficient Hispanic parents of 

CSHCN 

• Revealed useful explanatory 

findings that would have been 

elusory otherwise 

• Verified that respondents 

understood and accepted the 

instrument 

• Assessed acceptance of 

questions and the instrument 

among respondents of children 

without special health care 

needs in the referent group 

• Identified problem questions, 

topical areas, and transitions 

• Allowed for extensive 

discussion of the translation 

• Major questionnaire revisions 

centered on three sections: care 

coordination, transition to adult 

care providers, and use of 

community based services 

• Other minor changes were made 

to questions throughout the 

instrument 

• Confirmed questions could be 

asked of parents of children 

without special health care needs 

in the referent sample (with slight 

modifications in some 

introductory statements) 

• Offered realistic feedback 

regarding question length, 

wording, transitions, and 

operational procedures 
References 

10

Beatty, P. (2004). Paradigms of cognitive interviewing practice, and their implications for developing standards of best 

practice. ZUMA-Nachrichten Spezial Band 9:8-25. 

Bethell, C.D., Read, D., Stein, R.E.K., Blumberg, S.J., Wells, N., Newacheck, P.W. (2002). Identifying children with 

special health care needs: development and evaluation of a short screening instrument. Ambulatory Pediatrics 2:38-48. 

Blumberg, S.J., Olson, L., Frankel, M., Osborn, L., Becker, C.J., Srinath, K.P., Giambo, P. (2003). Design and operation of 

the National Survey of Children with Special Health Care Needs, 2001. National Center for Health Statistics. Vital Health 

Stat 1 (41). 

Converse, J.M., and Stanley Presser. (1986). Survey questions: handcrafting the standardized questionnaire. Sage 

University Paper series on Quantitative Applications in the Social Sciences, Series no. 63. Sage Publications: Thousand 

Oaks, CA. 

DeMaio, T., Landreth, A. (2004). Examining expert reviews as a pretest method. ZUMA-Nachrichten Spezial Band 9:60-

73. 

Fowler, Jr., F.J. (2001). Survey research methods. Applied Social Research Methods Series, Volume 1. Sage Publications: 

Thousand Oaks, CA. 

Groves, R.M., Fowler, Jr., F.J., Couper, M.P., Lepkowski, J.M., Singer, E., Tourangeau, R. (2004). Survey Methodology. 

John Wiley & Sons: Hoboken, NJ. 

Lessler, J.T., and William D. Kalsbeek. (1992). Nonsampling error in surveys. John Wiley & Sons: New York, NY. 

McPherson, M., and Lynda Honberg. (2002). Identification of children with special health care needs: a cornerstone to 

achieving Healthy People 2010. Ambulatory Pediatrics 2: 22-23. 

McPherson, M., Arango, P., Fox, H., Lauver, C., McManus, M., Newacheck, P.W., Perrin, J.M., Shonkoff, J.P., Strickland, 

B. (1998). A new definition of Children with Special Health Care Needs. Pediatrics 102:137-140. 

Nelson, D. (1985). Informal testing as a means of questionnaire development. Journal of Official Statistics 1:179-188. 

Perrin EC, Newacheck PW, Pless IB, Drotar, D., Gortmaker S.L., Leventhal, J., Perrin, J.M., Stein R.E.K., Walker, D.K., 

Weitzman, M. (1993). Issues involved in the definition and classification of chronic health conditions. Pediatrics 91:787-

793. 

Presser, S., Couper, M.P., Lessler, J.T., Martin, E., Martin, J., Rothgeb, J.M., Singer, E. (2004). Methods for testing and 

evaluating survey questions. Public Opinion Quarterly 68:109-130. 

Ragin, C.C., Nagel, J., White, P. (2004). Report of the Workshop on Scientific Foundations of 

Qualitative Research. Conducted July 11-12, 2003. National Science Foundation: Arlington, Virginia. 

Read, D., Bethell, C. (2004). Cognitive testing the Spanish language version of the CSHCN Screener with limited English 

proficient Hispanic parents. Report prepared for CDC/NCHS, PO#211-2004-M-07923. 

U.S. Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health 

Bureau. (2004). The National Survey of Children with Special Health Care Needs Chartbook 2001. U.S. Department of 

Health and Human Services: Rockville, MD. 

van Dyck, P.C., McPherson, M., Strickland, B.B., Nesseler, K., Blumberg, S.J., Cynamon, M.L., Newacheck, P.W. (2002). 

The National Survey of Children with Special Health Care Needs. Ambulatory Pediatrics 2:29-37. 

Willis, G.B. (2005). Cognitive interviewing: a tool for improving questionnaire design. Sage Publications: Thousand Oaks, 

CA. 
World Health Organization. (2001). International classification of functioning, disability, and health. World Health 

Organization: Geneva, Switzerland. 

11