> 2001 National Survey of Veterans Design and Methodology
>
> Final Report

**Authors:**

G. Hussain Choudhry

Inho Park

Martha Stapleton Kudela

John C. Helmick

**August 12, 2002**

table of contents

Chapter Page

acknowledgements viii

executive summary ix

1 INTRODUCTION 1-1

1.1 Study Background 1-1

1.2 Questionnaire Design 1-2

1.3 Sample Design, Selection, and Management 1-4

1.4 Interviewer Recruitment and Training 1-5

1.5 Data Collection 1-5

1.6 Sample Weighting 1-8

1.7 Quality Control 1-8

1.8 Field Results 1-9

2 QUESTIONNAIRE DESIGN 2-1

2.1 1992 National Survey of Veterans 2-1

2.2 1992 National Survey of Veterans Utilization Study 2-2

2.3 Structural and Content Revisions to the NSV 2-5

3 SAMPLE DESIGN, SELECTION, AND MANAGEMENT 3-1

3.1 Sample Design 3-1

3.2 Sample Selection 3-18

3.3 Sample Management 3-23

4 INTERVIEWER RECRUITMENT AND TRAINING 4-1

4.1 Recruitment 4-1

4.2 Initial Training 4-1

5 DATA COLLECTION 5-1

5.1 The CATI System 5-1

5.2 Interviewing Operations 5-7

5.3 Special Operations 5-9

6 SAMPLE WEIGHTING 6-1

6.1 List Sample Weights 6-2

6.2 RDD Sample Weights 6-7

6.3 Composite Weights 6-17

table of contents (continued)

Chapter Page

6.4 Replicate Weights 6-22

6.5 Reliability of the Survey Estimates 6-24

6.6 Bias and Precision in the Combined Sample 6-26

7 QUALITY CONTROL 7-1

8 SURVEY FIELD RESULTS 8-1

8.1 Completed Interviews 8-1

8.2 Completed Interviews by Sample Stratum 8-1

8.3 List Sample Extended Interview Results and Response Rates 8-5

8.4 RDD Household Screening Results and Response Rates 8-7

8.5 RDD Sample Extended Interview Results and Response Rates 8-9

8.6 List Sample Location Rates 8-11

8.7 Cooperation Rates 8-12

8.8 Questionnaire Administration Timing 8-13

8.9 Telephone Call Statistics 8-14

references r-1

List of Appendixes

Appendix

A NSV 2001 RDD HOUSEHOLD SCREENING QUESTIONNAIRE A-1

B NSV 2001 LIST SAMPLE VERIFICATION QUESTION B-1

c nsv 2001 extended interview questionnaire and

cati specifications c-1

d 2001 NATIONAL SURVEY OF VETERANS

FREQUENTLY ASKED QUESTIONS AND ANSWERS d-1

e Response Categories for the 2001 National

Survey of Veterans e-1

f Chi-squared Automatic Interaction Detector f-1

g Raking G-1

H NSV 2001 Final Result Codes H-1

table of contents (continued)

List of Tables

Table Page

3-1 Distribution of total veteran population across priority groups 3-3

3-2 Allocation of NSV 2001 sample across priority groups under

"square root" allocation 3-5

3-3 Percentage of veterans in the VA files by priority group 3-7

3-4 Design effects and coefficients of variation (cv) for various
veteran

population subgroups for the alternative sample designs 3-13

3-5 Cost comparison of the alternative sample designs 3-15

3-6 Sample allocation for sample design D 3-18

3-7 Allocation of List Sample to sampling strata 3-19

3-8 RDD Sample waves 3-25

3-9 List Sample waves 3-25

3-10 Sample yield by sample wave (RDD and List Samples) 3-26

3-11 Distribution of completed sample cases by health care priority
group

within each wave (RDD and List Samples) 3-26

3-12 Distribution of completed sample cases by demographic variables

(age, gender, and race/ethnicity) within each wave (RDD and List

Samples) 3-27

3-13 Distribution of completed sample cases by level of education within

each wave (RDD and List Samples) 3-28

3-14 Distribution of completed sample cases by census region within each

wave (RDD and List Samples) 3-28

3-15 Chi-square values for testing homogeneity of distribution of the
sample

yield by various characteristics (RDD and List Samples) 3-29

5-1 List Sample advance mailout responses 5-10

5-2 Results of end-of-data collection credit bureau search 5-16

table of contents (continued)

List of Tables (continued)

Table Page

5-3 Results of mailout to refusal and hard-to-locate cases 5-25

6-1 VA key variables 6-21

6-2 Comparison between RDD and composite estimates 6-29

8-1 Target versus achieved List Sample completed extended interviews

by priority group (PRIOADJ3) 8-2

8-2 List priority (PRIOADJ3) by observed priority 8-3

8-3 Target versus achieved RDD Sample completed extended interviews

by priority group 8-4

8-4 Response status by priority group (List Sample) 8-5

8-5 Response status by age group (List Sample) 8-6

8-6 Response status by gender (List Sample) 8-6

8-7 Response status by census region (List Sample) 8-7

8-8 Household screener interview results by sample type 8-8

8-9 Distribution of telephone households by number of potential veterans

in the household 8-8

8-10 Distribution of response status by age groups (RDD Sample) 8-10

8-11 Distribution of response status by gender (RDD Sample) 8-10

8-12 Distribution of response status by census region (RDD Sample) 8-10

8-13 List Sample location rates by age group 8-11

8-14 List Sample location rates by census region 8-11

8-15 Refusal conversion rates by sample type 8-12

8-16 Cooperation rate by sample type 8-12

table of contents (continued)

List of Tables (continued)

Table Page

8-17 Questionnaire administration time in minutes by sample type 8-13

8-18 Number of completed cases by number of calls for each sample type
8-15

List of Figures

Figure

3-1 Expected RDD sample yield 3-21

6-1 Categories of response status 6-5

7-1 Flowchart of NSV 2001 pretest sample activities 7-5

List of Exhibits

Exhibit

4-1 NSV 2001 interviewer training agenda 4-3

5-1 Example of CATI specifications with hard range 5-3

5-2 Example of CATI specifications with soft range 5-3

5-3 Example of CATI logic check 5-4

5-4 VA advance letter 5-11

5-5 Westat advance letter 5-12

5-6 Letter to veterans with unpublished telephone numbers 5-14

5-7 Letter to veterans with no telephone number 5-15

5-8 Privacy Act statement 5-21

5-9 Screener refusal letter (RDD Sample) 5-22

5-10 Extended refusal letter (List Sample) 5-23

5-11 Extended refusal letter (RDD Sample) 5-24

ACKNOWLEDGMENTS

Susan Krumhaus was the Project Officer for the U.S. Department of
Veterans Affairs. In addition to her management responsibilities, she
provided valuable input into the sample and questionnaire design
processes. At Westat, Veronica Nieva was the Corporate Officer and John
Helmick was the Project Director. Katie Hubbell supervised all data
processing activities and Susan Fraser was responsible for all telephone
center operations. Contributing authors to this report were Tiandong Li,
Robin Jones, Wayne Hintze, Susan Fraser, and Susan Englehart.

EXECUTIVE SUMMARY

The 2001 National Survey of Veterans (NSV 2001) is the fifth in a series
of periodic comprehensive surveys conducted by the U.S. Department of
Veterans Affairs (VA). The survey target population was
noninstitutionalized veterans of the U.S. uniformed services living in
private households in the United States, including Puerto Rico. The NSV
2001 questionnaire reflects the needs and contributions of many VA
stakeholders. Covering a full range of topics about VA benefit programs
and services, the survey provides the VA with extensive data about
veterans' military background, education and training, health care
usage, and understanding and use of a broad array of VA benefits.

To meet the VA research objective of obtaining sufficient data from a
cross section of the veteran population, Westat designed a sampling plan
that employed a dual frame approach to obtain 20,000 completed veteran
interviews. This approach called for the completion of 13,000 veteran
interviews from randomly selected households (RDD Sample) augmented by
7,000 veteran interviews completed from a List Sample selected from the
VA Compensation and Pension, and Health Care Enrollment files. The
sample was allocated to obtain reliable estimates for each of the seven
VA health care enrollment priority groups, and for population subgroups
of particular interest such as females, Hispanics, and African
Americans.

Using a computer-assisted telephone interviewing (CATI) data collection
methodology, Westat collected the NSV 2001 data from February 12, 2001
through November 12, 2001. Nearly 300 interviewers participated in the
data collection, which resulted in 20,048 completed interviews (12,956
from the RDD Sample and 7,092 from the List Sample). Overall
administration time for the extended interview was 35.2 minutes. List
Sample veterans took an average of 38.7 minutes to complete the extended
interview, while RDD Sample veterans took an average of 33.3 minutes.
This pattern was expected because the List Sample veterans had more
medical conditions and medical experiences to report. List Sample
veterans were also more likely to have a service-connected disability
that required them to complete an additional survey module about that
disability.

Of all the households screened from the RDD Sample, 25.8 percent had at
least one potential veteran. The screener response rate was 67.6
percent. The extended interview response rate for RDD Sample veterans
was 76.4 percent. The overall RDD (combined screener and extended
interview) response rate was 51.6 percent. Tracking and tracing efforts
achieved a location rate for List Sample veterans of 73.6 percent. The
extended interview response rate for List Sample veterans was 62.8
percent. The lower response rate for the List Sample veterans can be
attributed to difficulty in locating the List Sample veterans.

The survey data were weighted so that the responses of the sampled
veterans could be properly expanded to represent the entire
(non-institutionalized) veteran population. The weight calculations took
into account the original selection probability, nonresponse, and
households with multiple residential telephone lines. We computed the
weights separately for the List Sample and the RDD Sample so that, when
fully weighted, the List Sample would represent the veterans from whom
the sample was drawn, and the RDD Sample would represent the entire
(non-institutionalized) population of veterans. In addition, the RDD
sample was benchmarked to known veteran population counts from the U.S.
Bureau of the Census 2000 Supplementary Survey (C2SS) to account for
undercoverage in the RDD Sample. The undercoverage in the RDD Sample
arises from the omission of nontelephone households and households with
unlisted telephone numbers belonging to "zero-listed telephone banks"
not covered in the list-assisted RDD methodology. The RDD and List
Samples were combined and a single database was constructed with
composite weights to represent the entire (non-institutionalized)
veteran population.

1\. INTRODUCTION

This report has been prepared by Westat for the U.S. Department of
Veterans Affairs (VA), to document the methodology employed for the 2001
National Survey of Veterans (NSV 2001). In this first chapter, we
present a comprehensive overview of the procedures we used to design and
carry out the survey. The remainder of the report provides details on
the survey design, data collection and quality control procedures,
sample weighting, management of the database, and field results. Chapter
2 focuses on questionnaire development, including the impact of previous
versions of the NSV, policy changes, and results of the 1992 National
Survey of Veterans Utilization Study. We describe the sample design,
selection, and management in Chapter 3. In Chapter 4, we discuss
interviewer recruitment and training procedures, including details of
the initial project-specific training and specialized training (e.g.,
refusal conversion, proxy interviewing) that took place after data
collection began. Chapter 5 summarizes data collection procedures,
including the computer-assisted telephone interviewing (CATI) system,
interviewing operations, and special data collection operations such as
proxy interviewing and tracing. In Chapter 6, we describe the
construction of the sample weights used to properly expand responses of
the sampled veterans to the population that the sample was selected to
represent. Chapter 7 presents the quality control procedures we
implemented throughout the various phases of the survey. Finally, we
summarize the survey field results in Chapter 8. Appendixes A, B, and C
contain the RDD screener questionnaire, List Sample verification
question, and extended interview questionnaire. A list of frequently
asked questions used by interviewers to address respondent concerns
appears in Appendix D. Appendixes E and H are the NSV 2001 response
categories and final result codes. Appendixes F and G describe the
Chi-square Hierarchical Automatic Interaction Detector (*CHAID*)
software and the raking procedures used for the NSV 2001.

# 1.1 Study Background

The 2001 National Survey of Veterans is the fifth in a series of
periodic comprehensive surveys conducted by the VA with a national
sample of all eligible veterans. The previous surveys were conducted in
1978, 1979, 1987, and 1993. All five surveys cover a full range of
topics about VA benefit programs and services. Much of the information
in these surveys is not available from any other sources that collect
information about veterans, including VA administrative files and the
U.S. Census of Population and Housing.

Since the 1992 NSV, the VA has undergone significant administrative
change. In response to the Veterans Health Care Eligibility Reform Act
of 1996, the Veterans Health Administration (VHA) reconfigured its
health care system to eliminate any distinction between inpatient and
outpatient care and to make primary care its major area of emphasis in
meeting the health care needs of veterans. This legislation also
established two health care eligibility categories and seven health care
enrollment priority groups. The Veterans Benefit Administration (VBA)
has experienced similar administrative reforms since the 1992 NSV. In
order to plan for and evaluate the newly reconfigured health care
services, programs, and policies, the VA, VHA, and VBA needed up-to-date
information about the veterans they serve.

The NSV 2001 provides the VA with extensive data about veterans'
military background, education and training, health care usage, and
understanding and use of a broad array of VA benefits. Using these data,
the VA can follow changing trends in the veteran population, compare the
characteristics of veterans who use VA programs and those who do not,
and perform policy analyses. The data also provide information about
issues that have a direct impact on veterans, such as eligibility and
health care reforms. Finally, the NSV 2001 provides information relevant
to planning and budgeting VA programs and services for veterans in
general, as well as for certain veteran subgroups of particular
interest.

# 1.2 Questionnaire Design

The final NSV 2001 questionnaire reflects the needs and contributions of
many VA stakeholders. It addresses the larger national agenda, current
legislation about who qualifies for VA benefits, and recent developments
within the VA. At the same time, it remains comparable to previous
national surveys of veterans. The NSV 2001 draws on the 1992 NSV, the
Department of Veterans Affairs Strategic Plan (FY 1998-2003), and the
1992 National Survey of Veterans Utilization Study conducted by Westat
with numerous VA stakeholders between January and March 1999.

From the Utilization Study, we knew that data users wanted more detailed
information about veterans who do not use VA programs, veterans' unmet
needs, their future plans for using VA benefits, and their communication
preferences. The new financial eligibility guidelines for certain VA
programs and legislation expanding the definition of who qualifies for
benefits created the need for more information about veterans' financial
status and military background. Data users also expressed the desire for
more detailed information about veterans' health status, health
insurance coverage, health care preferences, and use of medical
facilities. The VA and Westat began refining these and other broad
research questions into a workable survey instrument in September 2000.
As part of this process, we took into consideration several practical
issues:

-   **Computer-assisted Telephone Interviewing (CATI) Technology.**
    Using CATI technology, we were able to incorporate complex skip
    patterns into the questionnaire design.

-   **Data Collection Mode.** We accommodated the demands of telephone
    interviewing by ensuring that questions were easy to comprehend,
    that lists of response options were short but not unduly
    restrictive, and that the average questionnaire administration time
    did not exceed 30 minutes.

-   **Sample Design.** We implemented a dual frame design. One frame was
    a random national cross section of veterans drawn using a household
    screening questionnaire from a random digit dialing sample of
    telephone numbers (RDD Sample) and the second frame was a random
    sample from VA administrative files (List Sample).

-   **Respondent Cognitive Ability.** We accommodated veterans' varying
    abilities to answer survey items, including the likelihood of either
    having the requisite personal knowledge or the capacity to remember
    past events in their lives.

The final NSV 2001 instrument collected information about each
respondent's military background and sociodemographic characteristics.
As well, within each of six benefit modules (Health Care, Disability,
Mortgage Loan, Life Insurance, Education and Training, and Burial
Benefits), the survey asked about veterans' perception of need for the
benefit and their recent use of the benefit. The survey also asked about
other sources of assistance they used, access and barriers to use of the
service, their unmet needs, future plans to use benefit programs, and
how they prefer to receive information about the service. Finally, the
NSV 2001 included a separate module about veterans' communication needs.
This last section, a new feature of the NSV 2001, asked about veterans'
recent need for VA information, preferred media, and access to and use
of the Internet.

In addition to the modular questionnaire that addressed substantive
topics of interest (hereinafter referred to as "the extended
interview"), we designed a preliminary screening questionnaire to
determine whether any member of an RDD Sample household was a veteran.
Because List Sample veterans drawn from the VA files were, by
definition, eligible for the survey, it was unnecessary to conduct the
screening part of the interview with them. Instead, verification
procedures were established to ensure that contact was made with the
correct sampled veteran.

# 1.3 Sample Design, Selection, and Management

####  Sample Design

A primary VA research objective was to obtain sufficient data from a
cross section of the veteran population, from each of the seven VA
health care enrollment priority groups, and from population subgroups of
particular interest (female, Hispanic, and African American veterans).
The VA specified 95 percent confidence intervals of 5% for estimates of
proportion equal to 0.50 for each of the above cells. To meet these
objectives, Westat designed a sampling plan that called for the
completion of 13,000 veteran extended interviews from randomly selected
households (RDD Sample). This was augmented by 7,000 veteran extended
interviews completed from a List Sample selected from the VA
administrative files. By completing 20,000 veteran interviews from these
two samples, we fulfilled most of the VA precision requirements.

####  Sample Selection

The project team prepared estimates of the number of cases in each type
of sample that would be needed to yield the desired number of completed
interviews. For the RDD Sample, the sampling unit was the randomly
selected telephone number. Therefore, these estimates included the
historical rate of residential numbers among randomly sampled telephone
numbers from a list-assisted RDD sampling frame. Other factors that
affected the sample estimates were the expected success rate in
contacting and completing a screening interview with the households, the
estimated percentage of households with at least one veteran, and the
expected cooperation rate among identified veterans. Based on these
estimates, we selected a sufficient sample of telephone numbers from the
national sampling frame using a list-assisted RDD sampling methodology.
In addition, a Puerto Rico sample was selected using a naïve approach
called "RDD element sampling."

For the List Sample, we created a frame from two VA administrative
files: the Health Care Enrollment File and the Veterans Compensation and
Pension (C&P) File. From the total of 4.5 million veterans on the list
frame, a stratified systematic random sample was drawn of veterans
belonging to the mandatory health care enrollment priority groups
(groups 1 through 6).

####  Sample Management

In managing the survey sample, we ensured two things: first, that the
sample was truly representative of the universe that defined it, and
second, that we reached, as closely as possible, the target quotas for
each sample stratum. To achieve these goals, we monitored cases during
data collection, evaluated sample yields against expectations, refined
interviewing strategies, and adjusted assigned sampling rates as needed
to meet the targets. As data collection progressed, our ability to
project the number of cases needed to meet our targets became more
precise. We periodically reprojected yields based on reports comparing
actual sample performance to the original yield projections.

# 1.4 Interviewer Recruitment and Training

The interviewers we employed to conduct the NSV 2001 received 20 to 24
hours of training in areas specific to the NSV 2001 project, including
questionnaire content, sample design, contact procedures, call results
recordkeeping, problem documentation, and refusal avoidance. In
addition, before project-specific training, they received 8 hours of
training in general telephone interviewing techniques and use of the
CATI system. Because calls to List Sample veterans did not begin until
several weeks into data collection, List Sample contact procedures were
covered in a supplementary session instead of during the initial
training. The principal training tool was scripted interactive
exercises. Interviewers were not assigned to live interviewing until
they had successfully completed these exercises. Throughout the data
collection period, we held specialized training sessions on such topics
as interviewing proxies, converting refusals, tracing List Sample
veterans, and conducting callbacks to retrieve missing data.
Additionally, telephone center supervisors monitored on average
approximately 8 percent of all interviewing hours.

# 1.5 Data Collection

Westat collected the NSV 2001 data from February 12, 2001 through
November 12, 2001. As stated above, we used CATI technology to
administer the questionnaire. The following sections describe the CATI
methodology we used, along with special features and operations that
supported the data collection effort.

####  CATI System

The NSV 2001 CATI system was a customized version of Westat's Cheshire
CATI system, a state-of-the-art data collection tool developed and
refined by Westat programmers over the past 20 years. After designing
the questionnaire and sample, we developed detailed programming
specifications to create the CATI software that would operate the
screener and extended instruments, the database for recording the
response data, and the case management, contact, and call scheduling
modules. A systems analyst led a team of programmers in writing and
testing the code, with a final round of testing after all modules of the
system were integrated. The database administrator, the questionnaire
design team, project management staff, and telephone center personnel
also tested the system.

The system supported both general data collection requirements and the
special needs of the NSV 2001. For administering the substantive
questionnaire, we programmed the NSV 2001 CATI system with online range
checks for all variables, internal consistency checks between various
responses, and detailed probes for the interviewers to present to the
respondent when checks failed. The CATI case management function allowed
us to, among other things, control the release of the sample in waves
and automatically hold refusal and proxy cases until a designated
release time. Contact modules allowed interviewers to record and handle
appointments, messages, and problems, as well as verify contact with the
intended List Sample veteran. The CATI call scheduling system contained
detailed rules and algorithms for prioritizing and assigning cases to be
worked. These rules took into consideration time of day and day of the
week, callback appointments scheduled, call history of the case, type of
interview, and special interviewing skills required.

####  Telephone Interviews

Telephone interviewing for the NSV 2001 took place in six of Westat's
telephone research centers. Together, the facilities contained
approximately 240 telephone interviewing stations, all fully supported
by the central CATI system. Approximately 300 interviewers participated
in the NSV 2001 data collection. The interviewing operations were
managed by a telephone center operations manager, while a group of team
leaders were responsible for the daily operations in the telephone
centers. Using silent monitoring stations from which to hear an
interview and follow it on a CATI terminal, the telephone center project
manager and team leaders continuously assessed and reassessed
interviewer performance to ensure high quality data collection. VA
representatives and Westat project managers also monitored interviews
during training, the pretest, and main data collection. Monitoring was
the basis for direct feedback to each interviewer on his or her
performance and any areas needing improvement. It also provided project
staff with ideas for improving the questionnaire and survey procedures
after the pretest.

Various aspects of the NSV 2001 interview required particular skills.
For example, specially trained interviewers administered the RDD
household screener when we encountered language problems. We trained
other interviewers to identify and interview a knowledgeable proxy when
a veteran was unable to respond for himself or herself because of
illness, communication problems, or physical or mental incapacity. We
also trained a subset of interviewers to administer a short data
retrieval questionnaire to certain households that had already completed
the NSV 2001 screener but had missing data regarding eligibility.

####  Special Data Collection Operations 

To increase response rates, and to ensure that we interviewed the
correct sampled veteran, Westat put into place a variety of special data
collection operations. We sent all List Sample veterans an advance
letter to inform them of their selection to participate in the survey,
explain its purpose, urge their cooperation, and establish it as a valid
project of the U.S. Department of Veterans Affairs. To improve the
likelihood of locating and contacting the List Sample veterans, the
advance letter listed the addressee's current contact information and
included a form and business-reply envelope that the recipient could use
to update the contact information. We also provided a toll-free number
for the veteran to call with their contact information, questions, or
concerns. Other mail operations included mailout of a background letter
for RDD Sample cases who requested a written explanation of the survey
before proceeding (and for List Sample cases who did not receive their
advance letter), and refusal conversion letters for RDD screener
refusals and any extended interview refusals for which we could obtain
an address.

We further assigned teams of interviewers to carry out special purpose
interviewing efforts. They conducted refusal conversion attempts on RDD
screener and extended interview refusals who were not hostile. As
mentioned above, we also assigned a small group of highly skilled
interviewers to contact people who had previously been identified as
likely proxies for veterans who were not capable of responding to the
interview. Finally, we kept a special tracing operation in place
throughout the data collection period. Whenever we discovered that a
List Sample veteran was not at his or her last known telephone number or
address, the case was sent through one or more steps to locate the
veteran.

# 1.6 Sample Weighting

After we collected, edited, and cleaned the survey data, we constructed
sampling weights so that the responses of the sampled veterans could be
properly expanded to represent the entire veteran population. The weight
calculations took into account the original selection probability,
nonresponse, households with multiple residential telephone lines, and
undercoverage due to the omission of nontelephone households and
households with unlisted telephone numbers belonging to "zero-listed
telephone banks" not covered in the list-assisted RDD methodology. We
computed the weights separately for the List Sample and the RDD Sample
so that, when fully weighted, the List Sample would represent the
veterans from whom the sample was drawn, and the RDD Sample would
represent the entire population of veterans. The RDD and List Samples
were combined to construct a single database with composite weights. The
composite weights are adjusted weights that reflect the increased chance
that a veteran on the list sampling frame has of selection.

# 1.7 Quality Control

Rather than relying exclusively on external or after-the-fact quality
control procedures, Westat built quality control into the survey
operation itself. For example, we programmed real time, online data
control into the CATI instrument. Our interviewer training and
monitoring procedures also added extra measures of quality to the
operation. Whenever possible, tasks were automated. To assure quality
control in the automation process itself, every system and program, once
implemented, was tested, revised, respecified, and retested. We also
subjected sample selection and management tasks to rigorous quality
control checks. The database manager prepared and edited all data, coded
open-ended responses, handled and documented special problems, and
produced the final clean data files for weighting and analysis.

A key quality control measure was a pretest we conducted at the start of
data collection. We analyzed the data collected from the first 519
completed RDD extended interviews in order to test:

-   Questionnaire design, content, wording, and structure;

-   Length of interview administration;

-   The functioning of the computerized CATI questionnaire and the CATI
    case management and call scheduling software;

-   The interviewer training process; and

-   The interviewing process itself.

Based on this analysis, we modified our yield assumptions to reflect the
actual completion rate, and made slight changes to the interviewer
training program to increase focus on areas that presented problems for
interviewers or respondents in the pretest. The pretest also revealed
that the average length of the interview was slightly over the target of
30 minutes.

# 1.8 Field Results

The NSV 2001 had a total sample target of 20,000 completed extended
interviews. Of this total, 13,000 were to come from the household
screening of RDD telephone numbers and 7,000 were to come from the List
Sample. We achieved 100.2 percent of the overall target by completing
12,956 interviews from the RDD Sample (99.7 percent of the goal) and
7,092 interviews from the List Sample (101.3 percent of the goal).

As part of the NSV 2001 sample design, we also set interview completion
targets by sample stratum for both the List and RDD Samples. The main
objective of the List Sample stratification was to augment the sample
for veterans in the mandatory health care enrollment priority groups
(groups 1 through 6) and for female veterans. We also set List Sample
interview targets for Hispanic and African American veterans. The List
Sample completion rate for female veterans was 99.1 percent of the
target. We achieved 122.9 percent of the target for Hispanic veterans
from the List Sample and 156.3 percent of the African American veterans
targeted to be interviewed from the List Sample. We exceeded the List
Sample targets for four out of the six priority groups. We achieved
almost 96 percent of the target for priority group 5. For priority group
4, however, we reached only 59 percent. We attribute this low completion
rate to a high incidence of priority group 4 veterans being
institutionalized or deceased, making them ineligible for the survey.

For the NSV 2001 sample design, we set the RDD Sample size targets
according to our assumptions about the distribution of the veteran
population across various subgroups. However, the observed yields across
various subgroups of the veteran population depended on the true
distributions and not the assumed ones. For the RDD Sample, each case's
stratum was determined by the veteran's responses to the survey
questions about the stratification variables (priority group, age, race,
gender, etc.). For the female veterans we achieved 105.6 percent of the
target. The completion rates for Hispanic and African American veterans
were 107.3 percent and 92.2 percent, respectively, of their expected
targets.

The average interview time was 4.1 minutes per completed RDD screener
household interview. The RDD Sample veterans took 33.3 minutes on
average to complete the extended interview, while the List Sample
veterans took an average of 38.7 minutes per completed extended
interview. We expected the List Sample veterans to take longer to
complete the interview because they generally have more medical
conditions and medical treatment experiences to report. List Sample
veterans were also more likely to have a service-connected disability
that required them to complete an additional survey module about that
disability.

2\. QUESTIONNAIRE DESIGN

The 2001 National Survey of Veterans (NSV 2001) was designed to collect
information that will help the U.S. Department of Veterans Affairs (VA)
respond to many of its policy, planning, and budgetary challenges. This
survey information is crucial to the VA for program and benefit
planning. To ensure that the NSV 2001 would be useful to a large,
diverse group of stakeholders, we examined the 1992 NSV instrument and
reviewed results of the 1992 National Survey of Veterans Utilization
Study before the questionnaire design phase. (Westat conducted the
Utilization Study in 1999 to determine the data and information required
by various stakeholders within VA to manage the policies, programs, and
services that support the veteran population.) We then assessed the
feasibility of meeting those information needs and set priorities among
the needs. Finally, we integrated the results of the Utilization Study
into the design of the proposed NSV 2001 instrument and sampling
methodology. This chapter provides an overview of the 1992 NSV, presents
recommendations from the 1992 NSV Utilization Study, describes how these
findings were integrated into the NSV 2001, and details the content of
this most recent version of the NSV.

# 2.1 1992 National Survey of Veterans

The VA conducted the 1992 NSV to obtain information necessary for
planning and budgeting VA programs and services for veterans in general,
as well as for certain subgroups of veterans. It also studied
characteristics of the veteran population. The survey was designed to
collect data that would enable the VA to follow changing trends in the
veteran population, compare characteristics of veterans who use VA
programs versus those who do not, and provide a current data resource
for policy analyses. In addition, the data provided information needed
for major initiatives that have a direct effect on veterans, such as
benefit eligibility reform and health care benefit reform.

To the extent permitted by changing data needs, the 1992 NSV questions
were tailored to match questions in previous VA surveys to allow for
trend analysis. However, new information needs, new directions and
initiatives within VA, and a rapidly developing national agenda
necessitated a number

of changes to the survey. These changes were reflected in each major
stage of the instrument's development. Principal factors driving the
design changes over time were:

-   The need for more detailed information on veterans' health status,
    medical facilities usage, and health insurance coverage;

-   New financial eligibility guidelines for certain VA programs;

-   Legislation expanding the definition of military service experience
    eligible for veterans benefits; and

-   The need to represent certain subgroups in the veteran population
    through:

```{=html}
<!-- -->
```
-   A random cross section of the veterans drawn from a household survey
    (RDD Sample); and

-   A random sample from VA files of VA medical facilities users and
    veterans receiving compensation for service-connected disabilities
    (List Sample).

Another major factor driving the 1992 survey design was results of a
pretest showing the feasibility of collecting certain information by
telephone. This led to the decision to use a telephone survey as the
data collection method and to use computer-assisted telephone
interviewing (CATI) to conduct the interviews. The wording and structure
of the survey instrument accommodated this mode of data collection. This
mode required for instance, that questions be easy to comprehend on the
phone, that lists of choices be kept as short as possible without unduly
limiting likely responses, and that questionnaire administration time
meet a target of 45 minutes on average.

# 2.2 1992 National Survey of Veterans Utilization Study

As the primary means for generating key information for the VA's
planning process, the NSV needs to answer central questions about the
VA's "clients"--veterans of the U.S. uniformed services. These questions
include:

-   Who are the VA's clients?

-   How many veterans are eligible for VA benefits? What characteristics
    make them eligible?

-   What are veterans' needs? How do they perceive their needs?

-   How do veterans act on their needs and their perceived needs?

-   How many eligible veterans actually use their VA benefits?

-   How easy is it for them to access VA services?

-   What are their preferences for receiving services?

-   How are their needs likely to change in the near term?

Westat conducted the 1992 National Survey of Veterans Utilization Study
in 1999 to identify and articulate the uses to which stakeholders both
inside and outside the VA put previous NSV results and their anticipated
information needs from a future NSV.[^1] We conducted 51 interviews with
a diverse group of 102 individuals and found that while the 1992 survey
included answers to some of the above questions, it had serious gaps and
did not provide some important information to VA planners.

One important topic covered in the interviews was use of the 1992 NSV
final report and survey data set. Of the 51 stakeholder interviews, 36
indicated at least some use of the written NSV report, while only 18
indicated any use of the database of survey responses. All 18 database
users indicated they were also users of the final report. Fifteen of
those interviewed indicated using neither the report nor the database.
Their reasons included:

-   The report or database was not directly relevant to their needs.

-   Certain subpopulations were not sufficiently represented in the data
    set (e.g., female, minority, and homeless veterans).

-   The database was expensive to access (users had to access the
    database through a link with the Austin data center).

-   The database was difficult to use.

Even those who reported high initial use reported less use over time
because the data were no longer current or because other, more current,
sources of information existed (e.g., hospital utilization data in other
Veterans Health Administration \[VHA\] databases).

The interviews also asked about specific topics from the 1992 NSV. None
was identified as a candidate to be excluded from a future study. Those
interviewed suggested improvements or modifications for all but one
topic area (Respondent Information). Two topic areas (Health and
Activity Limitations, and Health Insurance) were targeted for major
changes. Suggested improvements to the Health Insurance section called
for more extensive information about other types of insurance coverage
and satisfaction with various insurance policy services.

In addition to specific topics, users also made suggestions and requests
about enhancements to the survey database. Many users requested the
ability to find the residence location for persons in the sample to
support analyses of geographic distribution of veterans and access to
services. Other requests included a requirement to link survey responses
to the actual use patterns of veterans as identified in current VA
databases. The 1992 database users also strongly supported having access
to the survey database on a CD-ROM.

Both internal and external VA stakeholders requested more detailed
information on service patterns for special subpopulations to ensure
fairness of service delivery and to meet special needs. Stakeholders
wanted more information on minority, female, and homeless veterans, as
well as other special subpopulations, such as veterans exposed to Agent
Orange in Vietnam or veterans who served in the Gulf War. Others
interviewed commented on the use of different sampling strategies (e.g.,
combining a List Sample of those receiving services with an RDD Sample).

The general pattern of the interviews showed that current users of the
information would likely continue to use the results provided by another
NSV. Many persons throughout the VA indicated an interest in national
survey data if the data also addressed their specific program and
mission responsibilities. Many in the latter group offered specific
suggestions for items and for characteristics of the report or database
that would encourage their increased use.

A primary finding was strong support for a survey effort to collect data
that would allow analysts to distinguish between those veterans who do
or do not currently use VA programs. Many of those interviewed noted an
increased need for information that could be used to plan, manage, and
evaluate programs. Although many of the organizations have their own
survey efforts focused on their program offerings, all VA organizations
indicated a desire for information on veterans who do not use VA
programs as a part of their overall planning, policy, and evaluation
efforts.

Based on the 1992 NSV Utilization Study, we made several structural and
content changes to the survey instrument to address the VA's needs and
to increase the usability of the next NSV data set and report.

# 2.3 Structural and Content Revisions to the NSV

The 1992 NSV Utilization Study provided the framework for setting
priorities among topics and items for the next NSV. As the fifth
national survey in a series, it was important that the NSV 2001 address
emerging issues while maintaining comparability in some areas with past
surveys. However, to keep the questionnaire to a reasonable length, we
had to shorten or remove some topics. We chose priority topics and items
by focusing on the planning information needs identified in the
Utilization Study. We reduced the number of items used to measure
service use -- most notably in the area of health care -- and we cut
items on awareness of specific types of benefits. We also eliminated
items that did not address planning questions and items that were or
could be available from other sources.

Compared to the previous version, the NSV 2001 has a more direct focus
on nonusers. It also addresses veterans' future plans for using
education, training, and burial benefits. We added new items that assess
veterans' communication preferences. We also expanded race and ethnicity
categories to permit more accurate classification of veterans.
Additionally, we expanded the set of items that measured combat exposure
to better identify women's combat exposure and permit more detailed
projections of future health care needs.

The most significant structural change to the NSV 2001 survey is the
modular structure of the instrument. This modular approach allowed us to
collect information about multiple program usage (and nonusage) among
veterans while minimizing respondent burden by letting them skip
questions about services they had not used. Grouping the questions by
benefit area also simplified the respondents' cognitive burden by asking
them to think about only one topic at a time. In addition, the parallel
structure of the modules allowed respondents to "learn" the interview
and anticipate questions, reducing perceived respondent burden and
interview length. Questions in the new NSV include the following:

-   Introduction;

-   Military Background;

-   Health Background;

-   Benefit-specific modules:

```{=html}
<!-- -->
```
-   Health care

-   Disability

-   Mortgage Loan

-   Life Insurance

-   Education and Training

-   Burial

```{=html}
<!-- -->
```
-   Communication; and

-   Sociodemographic Information

These modules are described in detail in the following paragraphs. The
NSV 2001 extended questionnaire (with CATI programming specifications)
appears in Appendix C.

**Introduction.** The Introduction module identified Westat as the data
collection contractor and named the study sponsor (U.S. Department of
Veterans Affairs). Here we outlined the general study objective,
informed the respondent that participation was voluntary, and provided
the respondent an option to receive the Privacy Act Statement. The
Introduction module disclosed U.S. Office of Management and Budget (OMB)
clearance, provided the study's OMB clearance number, and provided VA
contact and address information.

**Military Background.** This module collected respondent gender and
date of birth, along with information on military service, including
component (reserves or active), branch of service, active duty start and
release dates, service era, type of discharge, geographic locations of
service, and combat and other military experiences (exposure to
hazards/agents, prisoner of war). This module was used to establish the
basis of veteran eligibility (e.g., type of service, length of service,
locations) for specific veteran entitlements, services, and benefits.

The Military Background module was very similar to the Eligibility and
Description of Military Service sections of the 1992 NSV. Periods of
service were revised so as to be comparable with Census 2000, and the
set of items identifying combat exposure was expanded to ensure more
accurate assessment of female veterans' combat exposure and to permit
more detailed projections of future need. The items used to expand the
set of combat items were adapted from the combat intensity scale
developed and used by the U.S. Centers for Disease Control and
Prevention (CDC).

**Health Background.** The Health Background module collected general
information on health status and whether current health was a limiting
factor in performing selected daily activities (e.g., climbing stairs,
pushing a vacuum). This module included the VSF12 (a 12-item scale
providing measures of mental and physical well-being). It identified
veterans who were disabled and who had a service-connected disability,
and collected the veteran's service-connected disability rating. The
module asked veterans about the effects of their health problems on
their ability to perform daily activities (e.g., walking, bathing,
shopping, getting dressed, eating), and whether or not the veteran had
received medical treatment in the past year for a list of specific
health conditions (e.g., high blood pressure, stroke, heart trouble,
cancer, drug abuse/alcoholism, post-traumatic stress disorder). It also
collected information about the veteran's current health insurance plans
or programs, including Medicare (Part A, Part B), Medicare managed care,
HMO, Medigap/Medicare supplemental health insurance, Medicaid
(Medi-Cal), CHAMPUS, TRICARE, other government health care programs, or
private health insurance.

**Health Care.** The Health Care benefits module collected information
on the veteran's use of VA and non-VA health care services (number of
visits) within the past 12 months, including emergency room use,
outpatient care, and inpatient care (number of overnights hospitalized).
It asked how services were paid for (e.g., VA, CHAMPUS/TRICARE, HMO,
Medicare, private health insurance) and where veterans obtained their
prescription medications. This module also contained items that asked
about receipt of medical care due to exposure (while in the military) to
environmental hazards, in-home health care, prosthetics, psychological
counseling, therapy, and alcohol or drug treatment. Finally, it
collected information on reasons why the veteran did not use VA health
care benefits (within the past 12 months, or ever).

**Disability.** The Disability module applied to veterans who indicated
(in the Health Background module) that they had either a disabling
condition or a service-connected disability. It collected information on
application for disability benefits, reasons for not applying, and
status of most recent claim. It also collected information on receipt of
disability payments from the VA, including type of payment
(service-connected disability compensation, nonservice-connected
disability pension), receipt of aid and attendance or housebound
benefits, and how these payments helped in meeting financial needs.
Items in this module asked about other (non-VA) sources of monetary
disability benefits, use (or nonuse) of VA and non-VA vocational
rehabilitation services, how well the veteran understood VA disability
entitlements, and perceived ease in obtaining benefits.

**Mortgage Loan.** The Mortgage Loan module collected information on use
of VA or non-VA loan programs to purchase a home, make home
improvements, or refinance a home loan. It also asked reasons for not
using the VA home loan program.

**Life Insurance.** The Life Insurance module collected information on
VA life insurance coverage, whether the veteran was currently covered by
VA life insurance, main reasons for not using VA life insurance, and
life insurance coverage from other sources. It also asked how well the
veteran understood VA life insurance entitlements.

**Education and Training.** The Education and Training module collected
information on use or nonuse of VA and non-VA education and training
benefits. (This module did not include vocational rehabilitation
benefits, which were addressed in the Disability Module). It contained
items that asked for what purpose the VA educational benefit was used
(college/university coursework leading to a degree, correspondence
courses, flight training, teacher certification, or business, technical,
or vocational training), how important the VA educational benefits were
in meeting the veteran's educational goals or preparing the veteran for
a better job, and how likely the veteran would be to use VA educational
benefits in the future.

**Burial.** The Burial benefits module asked veterans whether they had
heard of specific burial benefit programs (VA headstones and markers,
burial in a national or state veteran's cemetery, Presidential Memorial
Certificates for next of kin). It also collected information on veteran
burial plans and preferences, reasons for wanting/not wanting to be
buried in a veteran's cemetery, whether they intended to use a
VA-provided headstone or marker, how well the veteran understood veteran
burial benefits, and how difficult the veteran thought it would be to
obtain information about veteran burial benefits.

**Communication.** The Communication module collected information on
where veterans "think they would go" to obtain information about VA
benefits. It asked how much the veteran knew about veteran benefits, how
satisfied the veteran was with the ability to get information about
veteran benefits, whether the veteran has needed any information about
benefits in the past year, and whether the veteran had access to the
Internet.

This module was a new feature of the revised NSV questionnaire. It
collected general information about respondents' recent need for VA
benefit information, preferred sources and media for VA benefit
information, Internet access, and recent Internet use.

Questions in the communication section were adapted from the 1992 NSV
and from materials used by the Centers for Medicare and Medicaid
Services (formerly known as the Health Care Financing Administration) in
its market research projects. Information in this module will help VA
researchers identify effective media for reaching users and nonusers,
and track changes in communication preferences.

**Sociodemographic Information.** The Sociodemographic Information
module collected information on employment status (working, retired,
disabled, looking for work), operation of a small business (including
Federal Employer Identification Number), race/ethnicity, education,
marital status, number of dependents, income (total combined family
income), income sources, assets, market value of primary residence,
amount of mortgage on primary residence, zip code, Social Security
number, and interest in participation in future VA studies. If the
veteran expressed an interest in participating in future surveys, his or
her full name and address were collected.

The Sociodemographic Information module is similar to several sections
of the 1992 NSV questionnaire (the Demographics and Income and the
Respondent Information sections). Several items were revised based on
discussions with VA staff and a desire to ensure comparability with
Census 2000. Taken together with the information on Military Background,
this information will be useful for:

-   Assessing program eligibility;

-   Analyzing current use;

-   Projecting future use;

-   Identifying emerging needs;

-   Identifying subpopulations with special needs; and

-   Modeling the effects of changes in program eligibility criteria.

3\. SAMPLE DESIGN, SELECTION, AND MANAGEMENT

The National Survey of Veterans (NSV 2001) was intended to provide
estimates for the entire non-institutionalized U.S. population of
veterans, as well as for veteran population subgroups of special
interest to the Department of Veterans Affairs (VA). The subgroups of
primary interest were the seven health care enrollment priority groups.
The VA was also particularly interested in data for female, African
American, and Hispanic veterans. In addition, the survey was required to
provide information needed for major initiatives that would have a
direct effect on veterans, such as benefit eligibility reform and health
care benefit reform. The sample design had to accommodate these policy
issues.

# 3.1 Sample Design

The VA desired to obtain 95 percent confidence intervals of ±5 percent
or smaller for estimates of proportion of 0.5 for each of the veteran
population subgroups. The resulting design called for 20,000 interviews
to be completed by random selection of veterans. As discussed later in
this section, we evaluated a number of alternative sample design options
and adopted a dual frame design consisting of a random digit dialing
sample (RDD Sample) and a List Sample. The cost-variance optimization
resulted in sample allocation of 13,000 completed interviews with random
digit dialing method and 7,000 completed interviews from the List
Sample. The List Sample design used the VHA Healthcare enrollment file
and the VBA Compensation and Pension (C&P) file to construct the
sampling frame. The VA administrative files alone could not be used for
the sample design because the coverage from these files was only about
21 percent.

Veterans living in institutions were included in the survey target
population only if they were in the institution for less than 6 months
and also had a principal residence elsewhere. Such veterans were
included in the survey as part of the RDD Sample only. Although the list
frame contained institutionalized veterans, they were not interviewed as
part of the List Sample because these would have to be screened for
eligibility. Veterans living abroad and in the territories were also
excluded from the survey target population. Therefore, a veteran sampled
from the list frame was not eligible for the survey if the address was
outside of the continental United States and Puerto Rico.

####  Allocation of Sample across Priority Groups

According to year 2000 projections of the veteran population provided by
the VA, approximately 25 million veterans were living across the
country. The VA manages its provision of health care services by
assigning veterans who enroll in their health care system to one of
seven health care enrollment priority groups, outlined as follows:

-   [Priority 1]{.underline}. Veterans with service-connected[^2]
    conditions rated 50 percent or more disabling.

-   [Priority 2]{.underline}. Veterans with service-connected conditions
    rated 30 to 40 percent disabling.

-   [Priority 3]{.underline}. Veterans who are former POWs. Veterans
    with service-connected conditions rated 10 to 20 percent disabling.
    Veterans discharged from active duty for a condition that was
    incurred or aggravated in the line of duty. Veterans awarded special
    eligibility classification under 38 U.S.C., Section 1151.

-   [Priority 4]{.underline}. Veterans who receive increased pension
    based on a use of regular aid and attendance or by reason of being
    permanently housebound and other veterans who are catastrophically
    disabled.

-   [Priority 5]{.underline}. Veterans with nonservice-connected and
    veterans with noncompensated service-connected conditions who are
    rated zero percent disabled, and whose income and net worth are
    below an established threshold.

-   [Priority 6]{.underline}. All other eligible veterans who are not
    required to make co-payments for their care. This includes:

```{=html}
<!-- -->
```
-   World War I and Mexican Border War veterans;

-   Veterans solely seeking care for disorders associated with exposure
    to a toxic substance, radiation, or for disorders associated with
    service in the Persian Gulf; and

-   Veterans with service-connected conditions who are rated zero
    percent disabled but who are receiving compensation from the VA.

```{=html}
<!-- -->
```
-   [Priority 7]{.underline}. Veterans with nonservice-connected
    disabilities and veterans with noncompensated service-connected
    conditions who are rated zero percent disabled, and who have income
    or net worth above the statutory threshold and who agree to pay
    specified co-payments.

The distribution of the total veteran population across the seven
priority groups is given in Table 3-1. Further, the law defines two
eligibility categories: mandatory and discretionary. Priority groups 1
through 6 are termed as mandatory, whereas priority group 7 is termed as
discretionary.

Table 3-1. Distribution of total veteran population across priority
groups

  ------------ ----------- ------- ------- ------- -------- ------- ---------------
               Mandatory                                            Discretionary

  Priority     1           2       3       4       5        6       7
  group                                                             

                                                                    

  Percent of   2.31        2.06    5.01    0.73    29.96    0.34    59.59
  total                                                             
  ------------ ----------- ------- ------- ------- -------- ------- ---------------

Note: These distributions do not reflect actual veteran health care
enrollments. These distributions were provided by VA analysts as
estimates of what the veteran population would look like if it was
segmented into the seven priority groups.

####  Three Approaches to Sample Allocation

The VA required that the sample design produce estimates of proportions
for veterans belonging to each of the seven priority groups and for
female, Hispanic, and African American veterans. Therefore, different
sampling rates had to be applied to the seven healthcare enrollment
priority groups. In particular, priority groups 4 and 6 had to be
sampled at relatively higher sampling rates to produce estimates with
the required levels of reliability.

We considered three approaches to allocate the total sample across the
seven priority groups: (1) equal allocation, (2) proportional
allocation, and (3) compromise allocation.

####  *Approach I -- Equal Allocation*

Under this approach, the sample is allocated equally to each of the
seven priority groups. The equal allocation approach achieves roughly
the same reliability for the priority group estimates of proportions. In
other words, it achieves almost the same coefficient of variation for
all priority group estimates. Because the veteran population varies
across priority groups, choosing this approach would have meant that the
selection probabilities of veterans would have also varied across
priority groups. As a result, the variation between the sampling weights
would have been very large and would have resulted in large variances
for the national level estimates. We therefore did not choose this
allocation because it would not have been very efficient for the
national level estimates.

####  *Approach II -- Proportional Allocation*

For this approach, the sample is allocated to the priority groups based
on the proportion of the veteran population that each priority group
represents. Under the proportional allocation approach, the priority
groups with larger veteran populations would have received the larger
share of the sample. In particular, priority group 7 would have received
a very large sample, while the sample sizes for priority groups 4 and 6
would have been too small to produce reliable survey estimates. The
proportional allocation would be the most efficient allocation for the
national level estimates because the probabilities of selection are the
same for all veterans irrespective of the priority group. We did not
choose this allocation because reliable priority group estimates would
only have been possible for the three largest groups (priority groups 3,
5, and 7).

####  *Approach III -- Compromise Allocation*

As the name implies, the compromise allocation is aimed at striking a
balance between producing reliable priority group estimates (*Approach
I*) and reliable national level estimates (*Approach II*). A number of
procedures are available to achieve this compromise. The actual
procedure to be applied depends on the exact survey objectives. The
simplest and most commonly used allocation is the so-called "square
root" allocation. Under this allocation, the sample is allocated to the
priority groups proportional to the square root of the population of the
priority groups. Under the "square root" allocation, the sample is
reallocated from very large priority groups to the smaller priority
groups as compared with what would have been under the proportional
allocation. A more general compromise allocation is the "power
allocation" discussed by Bankier (1988) under which the sample is
allocated proportional to ![](media/image1.wmf), where *x* is the
measure of size and the parameter ![](media/image2.wmf) can take values
between zero and 1. The value ![](media/image3.wmf) corresponds to the
"square root allocation." The two extreme values of
![](media/image4.wmf) give the "equal allocation" and the "proportional
allocation." In other words, ![](media/image5.wmf) corresponds to
*Approach I*, which is "equal allocation" and ![](media/image6.wmf)
corresponds to *Approach II*, which is "proportional allocation." Kish
(1988) has also considered a number of compromise allocations including
the "square root" allocation.

Because we were interested in both national level estimates and the
estimates for each of the priority groups, we used the "square root"
compromise allocation to allocate the sample across the seven priority
groups. The sample allocation across the seven priority groups under the
"square root" compromise allocation is shown in Table 3-2. The sample
allocation under the proportional allocation is identical to the
distribution of the veteran population across priority groups (Table
3-1) and that under the equal allocation would assign 14.3 percent of
the sample to each of the priority groups. In order to achieve the
"square root" allocation for minimum cost we chose a dual frame design.

Table 3-2. Allocation of NSV 2001 sample across priority groups under
"square root" allocation

  ------------- -------- ------- -------- ------- -------- ------- ---------
  Priority      1        2       3        4       5        6       7
  group                                                            

                                                                   

  Percent of    7.66     7.25    11.29    4.32    27.61    2.92    38.95
  sample                                                           
  ------------- -------- ------- -------- ------- -------- ------- ---------

####  Dual Frame Sample Design

Although it would have been theoretically feasible to select an RDD
Sample with "square root" allocation of the sample across priority
groups, such a sample design would have been prohibitively expensive.
The RDD Sample design is an Equal Probability Selection Method (epsem)
design, meaning that all households are selected with equal probability.
Thus, a very large RDD Sample would have to be selected in order to
yield the required number of veterans in priority group 6, the priority
group with the smallest proportion of veterans. The alternative was to
adopt a dual frame approach so that all of the categories with
insufficient sample size in the RDD Sample could be directly augmented
by sampling from the VA list frame. The corresponding survey database
would be constructed by combining the List and the RDD Samples with a
set of composite weights. This approach allowed us to use both samples
to achieve the desired level of precision for subgroups of interest to
the VA.

####  *RDD Sample Design*

We used a list-assisted RDD sampling methodology to select a sample of
telephone households that we screened to identify veterans. This
methodology was made possible by recent technological developments
(Potter et al., 1991, and Casady and Lepkowski, 1991 and 1993). In
list-assisted sampling, the set of all telephone numbers in an operating
telephone exchange is considered to be composed of 100-banks. Each
100-bank contains the 100 telephone numbers with the same first eight
digits (i.e., the identical area code, telephone exchange, and first two
of the last four digits of the telephone number). All 100-banks with at
least one residential telephone number that is listed in a published
telephone directory, known as "one-plus listed telephone banks," are
identified. We restricted the sampling frame to the "one-plus listed
telephone banks" only and then selected a systematic sample of telephone
numbers from this frame. Thus, the RDD sampling frame consisted of all
the telephone numbers in the "100-banks" containing at least one
*listed* telephone number.

The nonlisted telephone numbers belonging to "zero-listed telephone
banks" were not represented in the sample. However, nonlisted telephone
numbers that appeared by chance in the "one-plus listed telephone banks"
were included in the list-assisted RDD sampling frame.

Therefore, the list-assisted RDD sampling approach has two sources of
undercoverage. The first is that nontelephone households are not
represented in the survey. The second is the loss of telephone
households with unlisted telephone numbers in the banks having no listed
telephone numbers, known as "zero-listed telephone banks." Studies have
been carried out on these potential losses, and the undercoverage from
the two sources is estimated to be only about 4 to 6 percent (Brick et
al., 1995). As discussed in Chapter 6, an adjustment to correct for the
undercoverage was applied by use of a raking procedure with estimated
population counts from the Census 2000 Supplementary Survey (C2SS)
conducted by the U.S. Bureau of the Census.

####  *List Sample Design*

The VA constructed the list frame from two VA administrative files, the
2000 VHA Healthcare enrollment file and the 2000 VBA Compensation and
Pension (C&P) file. The files were crossed against each other, and a
single composite record was created for each veteran by matching the
Social Security numbers. The list frame included information about the
priority group to which each veteran belonged. Table 3-3 lists the total
veteran population and the percentage of population represented by the
list frame for each of the priority groups.

Table 3-3. Percentage of veterans in the VA files by priority group

+------------------+-------------------------+------------------------+
| Priority group   | Veteran population      | Percentage of veterans |
|                  | (thousands)             | in the list frame      |
+------------------+-------------------------+------------------------+
|                  |                         |                        |
+------------------+-------------------------+------------------------+
| > 1              | 577.5                   | 100.0                  |
+------------------+-------------------------+------------------------+
| > 2              | 516.4                   | 100.0                  |
+------------------+-------------------------+------------------------+
| > 3              | 1,254.1                 | 100.0                  |
+------------------+-------------------------+------------------------+
| > 4              | 183.6                   | 94.7                   |
+------------------+-------------------------+------------------------+
| > 5              | 7,501.4                 | 25.5                   |
+------------------+-------------------------+------------------------+
| > 6              | 83.8                    | 100.0                  |
+------------------+-------------------------+------------------------+
| > 7              | 14,920.3                | 5.9                    |
+------------------+-------------------------+------------------------+
| > All veterans   | 25,037.1                | 21.6                   |
+------------------+-------------------------+------------------------+

As observed in Table 3-3, the two largest priority groups (groups 5 and
7) have very low coverage of the veteran population in the list frame,
whereas four out of the remaining five priority groups (groups 1, 2, 3,
and 6) have 100 percent coverage. The list frame provides almost 95
percent coverage for priority group 4 (the second smallest priority
group). This feature of the list frame was advantageous for the dual
frame sample design because the sample could be augmented from the list
frame for the smaller priority groups. The VA lists covered 21.6 percent
of the overall veteran population including the priority group 7
veterans. Because of the very large proportion of priority group 7
population, no List Sample was required to augment this group of
veterans. After excluding priority group 7 veterans, the list frame
contained a total of over 4.5 million veterans, accounting for 44.7
percent of the veteran population belonging to the mandatory health care
groups (priority groups 1 through 6).

The list frame was stratified on the basis of priority group (groups 1
through 6) and gender. Thus, the veterans on the list frame were
assigned to one of 12 design strata and a systematic sample of veterans
was selected independently from each stratum.

####  *Allocation of Sample to List and RDD Frames*

Because it was less costly to complete an interview with a case from the
List Sample than the RDD Sample, the goal was to determine the
combination of List and RDD Sample cases that would achieve the highest
precision at the lowest cost. The higher RDD unit cost was due to the
additional screening required to identify telephone households with
veterans.

The largest proportion of veterans is in priority group 7, which
accounts for 59.6 percent of the total veteran population. The proposed
"square root" sample allocation scheme meant that we would allocate 38.9
percent of the total sample to priority group 7 veterans. Let
![](media/image7.wmf) be the total sample size and ![](media/image8.wmf)
be the proportion of the total sample that will be allocated to the RDD
frame. Then the expected RDD sample in priority group 7 would be
![](media/image9.wmf). The sample required for priority group 7 under
the square root allocation was equal to ![](media/image10.wmf). Because
no sample augmentation from the list frame was required for priority
group 7 the RDD sample in priority group 7 must be equal to the sample
required for the priority group, i. e. ![](media/image11.wmf), which
gives ![](media/image12.wmf). Thus, we needed to allocate 65.3 percent
of the total sample to the RDD frame. Any smaller proportion allocated
to the RDD frame would have had an adverse impact on the reliability of
the estimates, and a larger RDD proportion would have increased the
cost. Thus, 65.3 percent was the optimum allocation that minimized the
cost while achieving square root allocation of the total sample across
priority groups. The proportion was rounded to 65 percent for allocation
purposes, that is, 65 percent of the total sample was allocated to the
RDD frame.

The NSV 2001 cost assumptions were based on the previous RDD studies and
the assumption that about one in four households would be a veteran
household. We determined from these assumptions that it would be 1.3
times as expensive to complete an interview from an RDD household
veteran as compared with a List Sample veteran. As discussed later in
this chapter, a number of alternate sample designs were evaluated for
the total cost and the design effects for various veteran population
subgroups of interest.

####  Sample Size Determination

The decision on the sample size of completed extended interviews was
guided by the precision requirements for the estimates at the health
care priority group level and for the population subgroups of particular
interest (namely, female, African American, and Hispanic veterans). The
95 percent confidence interval for a proportion equal to 0.5 was
required with 5 percent or smaller confidence interval half-width for
these population subgroups.

The sample size required for the 95 percent confidence interval with
desired half-width (*w*) for a proportion of *p*=0.5 can be determined
by solving the following equation for the sample size
![](media/image13.wmf)

![](media/image14.wmf)

where *deff* is the design effect for the corresponding survey estimate.
As discussed later in this chapter, the *deff* of a complex sample
design is the ratio of the variances under the complex design and the
simple random sample design with the same sample sizes. For example, the
sample size for each priority group would be 768 for 95 percent
confidence interval with 5 percent margin of error for a sample design
with *deff* equal to 2.0. In order to assign a sample of 768 completed
interviews to priority group 6 (the priority group with smallest
proportion of veterans), while maintaining "square root" allocation
across priority groups, we would have to complete more than 26,000
interviews. This sample size was larger than VA was prepared to select
and it was decided to reallocate the sample across priority groups by
departing slightly from the proposed "square root" allocation and
accepting larger sampling errors for some veteran population subgroups.
As a result, the sample size of 20,000 completed interviews was
sufficient to satisfy the new precision requirements.

####  Alternative Sample Design Options 

We evaluated six sample design options with respect to cost and design
efficiency for a fixed total sample of 20,000 completed interviews. Two
of the sample designs were based on RDD sampling alone, whereas the
remaining four designs were based on a dual frame methodology using RDD
and list sampling. For each of the sample designs considered, we
compared the coefficients of variation of the national estimates and
those of a number of veteran population subgroups as well as the
corresponding design effects. The design effects were computed to
evaluate the efficiency of each of the alternative sample designs. We
also obtained the cost estimates for the alternative sample designs
using linear cost models incorporating screener and extended interview
unit costs.

####  *Sample Design A*

This sample design is based on RDD sampling of the number of veteran
households that would yield a sample of 20,000 completed extended
interviews. The sample sizes across the seven veteran health care
priority groups are random, and the expected sample sizes would be
distributed according to their proportion in the population. Similarly,
for the population subgroups of particular interest (female, African
American, and Hispanic veterans), the sample sizes are also random and
the expected sample sizes would be distributed in proportion to the
population sizes of the respective subgroups.

####  *Sample Design B*

This is also an RDD Sample design but the sample is allocated to the
priority groups according to the "square root" allocation scheme. The
fixed sample sizes across priority groups are achieved through
screening. The number of veteran households to be screened is determined
by the sample size allocated to priority group 6, the smallest priority
group, with 0.34 percent of the veteran population. The resulting sample
design is a stratified sample design with "square root" allocation of
the sample across the seven priority groups.

####  *Sample Design C*

This is the dual frame sample design discussed earlier, with an RDD
Sample of 13,000 and a List Sample of 7,000 completed extended
interviews. The list frame sample design is a stratified sample design.
The first level of stratification is on the basis of priority groups 1
through 6. As noted previously, no List Sample is allocated to priority
group 7. The List Sample is allocated across the remaining six priority
groups to achieve the "square root" allocation of the total sample (RDD
and List) across the seven priority groups. The next level of
stratification is by gender within each priority group and the sample is
allocated so that the sampling rate for female veterans is twice that
for male veterans. The stratification by gender allowed us to increase
the sample size for female veterans by sampling at a higher rate. This
strategy could not be adopted for Hispanic and African American veterans
because the variables to identify race/ethnicity were not available on
the VA files used to construct the list frame.

####  *Sample Design D*

This sample design is essentially the same as sample design C but the
List Sample is reallocated across the six priority groups by
oversampling priority groups 6 and 4 and correspondingly reducing the
sample size for priority group 5. Priority group 7 veterans are not
selected from the list frame.

####  *Sample Design E*

This is a dual frame design with an RDD Sample of 10,000 and a List
Sample of 10,000 completed extended interviews. To achieve the "square
root" allocation of the total sample across the priority groups, the
List Sample must be allocated to priority group 7 veterans as well. As
before, the List Sample is a stratified sample and the sampling rate for
female veterans is twice the sampling rate for male veterans within each
priority group.

####  *Sample Design F*

This is also a dual frame design with an RDD Sample of 15,000 and a List
Sample of 5,000 completed extended interviews. To achieve the "square
root" allocation of the total sample across the priority groups, the RDD
Sample must be screened for priority groups. The List Sample is
allocated to priority groups 1 through 6. As in the case of other dual
frame designs, the List Sample design is a stratified sample design, and
the sampling rate for female veterans is twice the sampling rate for
male veterans within each priority group.

####  Efficiencies of the Alternative Sample Designs

To evaluate the precision of the survey estimates, we computed the
standard errors of these estimates, where the standard error of an
estimate is defined as the square root of the variance of the estimate.
The ratio of the estimated standard error to the survey estimate itself,
called the coefficient of variation (cv) of the survey estimate can also
be used to evaluate a sample design.

Another way to evaluate the efficiency of a sample design and the
procedure used to develop the survey estimates is by using the design
effect. Design effect is defined as the ratio of the variance of an
estimate for a complex sample design and the variance of the estimate
under the simple random sample design with the same sample size. Kish
(1965) introduced the concept of design effect to deal with complex
sample designs involving stratification and clustering. Stratification
generally leads to a gain in efficiency over simple random sampling, but
clustering usually leads to deterioration in the efficiency of the
sample design due to positive intracluster correlation among units in
the cluster. To determine the total effect of any complex design on the
sampling variance in comparison to the alternative simple random sample
design, the design effect *(deff)* is defined as

![](media/image15.wmf).

We used the design effects for various survey estimates, including the
estimates for priority groups, to evaluate the alternative sample
designs that we considered for the NSV 2001. The smaller the design
effect the more efficient is the sample design without taking the cost
into consideration.

We also computed the cv of the survey estimates to check the precision
requirements for the survey estimates. The precision requirement was
specified in terms of margin of error of the 95 percent confidence
interval, which is given by 1.96 times the cv of the estimate. Table 3-4
provides the design effects and the cv of estimates of proportions equal
to 0.5 for various population subgroups, including the priority groups
for the alternative sample design options.

The following sections of this chapter discuss the comparative costs of
the six alternative sample design options and the cost-variance
efficiencies of the alternative sample designs.

####  Cost Comparisons

The choice of a survey design involves comparing costs and the design
efficiencies in terms of sampling variances. To estimate the total
survey costs for the six alternative sample designs, we used a linear
cost model. A general disposition of the total sampling cost for each of
the alternative sample designs can be described as follows.

Table 3-4. Design effects and coefficients of variation (cv) for various
veteran population subgroups for the alternative sample designs

  ---------------- --------- --------- --------- --------- --------- ---------
  Characteristic   Design                                            
                   Effects                                           

                   Design A  Design B  Design C  Design D  Design E  Design F

  All veterans     1.00      1.27      1.48      1.48      1.92      1.30

  Priority 1       1.98      1.00      1.13      1.13      1.10      1.21

  Priority 2       1.98      1.00      1.12      1.12      1.10      1.20

  Priority 3       1.95      1.00      1.18      1.18      1.14      1.32

  Priority 4       1.99      1.00      2.09      2.47      2.42      1.97

  Priority 5       1.70      1.00      2.07      1.92      2.62      2.02

  Priority 6       2.00      1.00      1.04      1.04      1.04      1.07

  Priority 7       1.40      1.00      1.39      1.39      1.75      1.21

  Male             1.05      1.34      1.52      1.52      1.95      1.35

  Female           1.95      2.45      2.98      2.96      4.15      2.57

  African American 1.92      2.44      2.50      2.52      3.20      2.36

  Hispanic         1.96      2.50      2.55      2.57      3.26      2.41
  ---------------- --------- --------- --------- --------- --------- ---------

  ---------------- -------------- --------- --------- --------- --------- ---------
  Characteristic   Coefficients                                           
                   of Variation                                           

                   Design A       Design B  Design C  Design D  Design E  Design F

  All veterans     0.35           0.40      0.43      0.43      0.49      0.40

  Priority group 1 3.30           1.28      1.36      1.36      1.34      1.40

  Priority group 2 3.44           1.31      1.38      1.38      1.37      1.43

  Priority group 3 2.19           1.05      1.14      1.14      1.12      1.20

  Priority group 4 6.84           1.83      2.65      2.48      2.85      2.57

  Priority group 5 0.86           0.68      0.97      0.98      1.10      0.96

  Priority group 6 9.44           2.15      2.16      1.80      2.19      2.23

  Priority group 7 0.54           0.56      0.66      0.66      0.74      0.62

  Male             0.37           0.42      0.45      0.45      0.51      0.42

  Female           2.18           2.47      2.43      2.44      2.77      2.32

  African American 1.71           1.93      1.95      1.96      2.21      1.89

  Hispanic         2.47           2.80      2.82      2.84      3.19      2.74
  ---------------- -------------- --------- --------- --------- --------- ---------

Selection of an RDD Sample considered for the NSV 2001 involved two
interviewing processes: a household screening interview and an extended
interview. Depending on an allocation method adopted in the sample
design, the screening interview would be administered to screen only for
the veteran households (level I screening) or for the priority groups
(level II screening). Under the proportional allocation, sample sizes to
be realized across seven priority groups will be random variables, but
their expected sample sizes will be proportional to the population sizes
of the priority groups. The interviewer would need to screen only for
the veteran households (level I screening) to get the required sample
size, say ![](media/image16.wmf), for the RDD Sample. The RDD Sample for
sample design A, C, D, and E were classified into this category, and the
general expression for such screening costs is given by

![](media/image17.wmf)

where ![](media/image18.wmf) is the unit cost for the level I screening.

On the other hand, the square root allocation used in the RDD Sample for
sample design B necessitated an additional step to screen for priority
group (level II screening) as well. The RDD Sample for sample design F
also must go through this step because the corresponding RDD Sample must
be screened for the priority groups. That is, once the interviewer found
a veteran household, he or she would need to further screen for the
priority group. Determination of priority group that a veteran belongs
to would require asking questions on income, assets, debts, number of
dependents, and disability rating. Consequently, the corresponding unit
cost ![](media/image19.wmf) for the level II screening was much larger
than the unit cost ![](media/image20.wmf)for the level I screening.
Also, to obtain the designated sample sizes, the screening process would
need to continue until the designated sample size for priority group 6
(the smallest category) is obtained. Thus, the number of telephone
numbers to be screened, denoted by ![](media/image21.wmf), would be much
larger than the corresponding total number of telephone numbers to be
sampled for the design, ![](media/image22.wmf). The higher unit
screening cost ![](media/image23.wmf) for level II screening and the
larger number ![](media/image24.wmf), of households to be screened would
result in a very large total screening cost. The total screening cost
for level II screening is given by

![](media/image25.wmf).

On completing the screening interview, the extended interview would be
administered to the fixed number, ![](media/image26.wmf), of veterans.
Letting ![](media/image27.wmf) denote the unit cost of the RDD extended
interview, the total cost for the RDD extended interview is

![](media/image28.wmf)

This equation applies to all RDD Samples.

For the List Sample, relevant information for selecting veterans is
available at the outset from the frame itself. Thus, no screening cost
is incurred but there is a cost associated with tracking and tracing the
List Sample veterans. Moreover, the average time to administer the
extended interview to List Sample veterans would be higher that that for
the RDD Sample veterans. Therefore, the unit cost of the List extended
interview would be higher than the RDD extended interview unit cost. The
extended interview would need to be administered to the List Sample
cases for all of the alternative sample designs using a dual frame
methodology. If we denote by ![](media/image29.wmf) the unit cost of the
List Sample extended interview then the total interview cost of the List
extended interview is

![](media/image30.wmf).

Under a linear cost model, total sampling cost based on the above
disposition can be obtained as the sum of the relevant components. For
example, ![](media/image31.wmf) is the total survey cost for the sample
design A, ![](media/image32.wmf) is the total survey cost for sample
design B, ![](media/image33.wmf) is the total survey cost for sample
design C, and so on. Table 3-5 provides the relative total sampling
costs for the alternative sample designs, where we have set the total
cost for sample design A equal to 100 to allow a standard comparison
across sample designs.

Table 3-5. Cost comparison of the alternative sample designs

  --------------------------------- -------------------------------------
  Sample design option              Cost relative to option A

                                    

  A                                 100

  B                                 519

  C                                 93

  D                                 93

  E                                 89

  F                                 104
  --------------------------------- -------------------------------------

As expected, the cost of the sample designs C and D are the same because
of the same sample allocation between the list and RDD frames. On the
other hand, sample design B is very expensive due to higher screening
cost (level II screening).

####  Cost-Variance Efficiency of the Alternative Sample Designs

We considered six sample designs to meet the survey objective, which was
to obtain sufficient data about a cross section of veterans for each
health care priority group, while at the same time obtaining reliable
survey estimates at the national level. Reliable survey estimates were
also required for female veterans and for Hispanic and African American
veterans. The design effects and coefficients of variation for the six
sample design options were obtained and these are given in Table 3-4 for
various subgroups of the veteran population. The reliabilities of the
survey estimates of proportions equal to 0.5 for various domains of
interest were considered for evaluating the alternative sample design
options.

Although the design effects would always be less than 2 for sample
design A, this design does not satisfy the survey objectives due to very
small sample sizes for smaller priority groups. The design effect for
sample design A is equal to ![](media/image34.wmf) when estimating a
proportion equal to 0.5 for a population domain of size
![](media/image35.wmf) relative to the entire population. On the other
hand, sample design B would satisfy the survey objectives but such a
design would not be feasible because of very high screening costs.
Hence, neither of the two RDD Sample designs is suitable for the NSV
2001, and we used a dual frame sample design with the allocation of the
total sample to the RDD and list frames as discussed earlier in this
chapter.

The cost of sample design C reduces to 93 percent of sample design A
because it is a dual frame design. The precision requirement is also
satisfied for most veteran population subgroups. It turned out that for
sample design C, the highest design effects were for priority groups 4
and 5 because coverage of these two groups by the list frame was less
than 100 percent. In spite of the high design effect for priority group
5, the precision requirement was satisfied because of the larger sample
size. The sample size required to satisfy the desired reliability for
the priority group 4 estimates could not be achieved by the proposed
"square root" sample allocation across priority groups with the total
sample size of 20,000 completed interviews. Although the proportion of
veterans in priority group 6 was less than that in priority group 4, the
precision requirement for priority group 6 was satisfied because of its
100 percent overlap with the list frame.

The design effects for female, African American, and Hispanic veterans
were also larger than 2. The female veterans account for 5.4 percent of
the total veteran population, and Hispanic and African Americans are
respectively 4.0 percent and 8.2 percent of the total veteran
population. The precision requirements for the estimates for female
veterans were achieved through oversampling the list frame. The higher
sampling rate for female veterans increased the design effects slightly
for the national level estimates because of increased variability in the
survey weights. The precision requirements for the estimates for
Hispanic veterans could not be satisfied. The List Sample size for
Hispanic veterans could not be increased because the variables to define
race/ethnicity were not available on the list frame. The precision
requirement for the estimate for African Americans was met because of a
larger sample size.

We reallocated the sample as proposed under sample design D to improve
the precision of the two smallest priority groups (groups 6 and 4).
Under sample design D we reallocated the sample from priority group 5 to
priority groups 4 and 6 from the allocation under sample design C.
Although the precision requirement for priority group 6 was satisfied
because the list frame covered this group completely, the List Sample
was also reallocated from priority group 5 to priority group 6. Under
sample design D the 95 percent confidence interval half-width would be
wider than 5.0 percent for Hispanic veterans only. Although the design
effect for priority group 4 increased, the precision level improved
because of an increase in the sample size due to sample reallocation.
The increase in the design effect for priority group 4 resulted from
fixed RDD variance for the nonoverlap part. Priority group 6 has 100
percent coverage by the list frame, and hence the design effect does not
change if the List Sample size increases. Overall, when the List Sample
is reallocated, the precision of the estimates for both priority groups
4 and 6 improves without significantly deteriorating the precision of
the estimate for priority group 5. The precision of the estimate for
Hispanic veterans decreases somewhat, but the effect is almost
negligible (the design effect for Hispanic veterans increased from 2.55
under sample design C to 2.57 under sample design D). The survey cost
for sample design D is the same as under sample design C due to the same
sample allocation between the list and the RDD frames.

The other two dual frame designs (sample designs E and F) differ from
sample design C in sample allocation between the RDD and list sampling
frames. Under design E, the cost is reduced only slightly (89 for design
E versus 93 for design C) by over-allocating the sample to the list
frame, but the reliability of the survey estimates is affected for
several categories of estimates. Sample design F achieves better
precision levels for some of the categories and worse for others, but
the overall cost increases because of higher screening costs (level II
screening). Sample design F requires screening for priority group in
order to achieve "square root" allocation of the sample at the priority
groups. The cost of sample design F is 104 as compared with 93 for
sample designs C and D. Therefore, sample design D provides a solution
that satisfies the survey objectives of producing reliable estimates and
controlling the overall cost of the survey.

The sampling parameters of sample design D (sample allocation and sample
sizes) are given in Table 3-6. The table also gives the effective sample
size, defined as the total sample size divided by the design effect. The
minimum effective sample size must be 384 in order to achieve the
required 5 percent half-width for 95 percent confidence interval of the
estimate of proportion equal to 0.5. Thus, for sample design D, the only
veteran population subgroup for which the precision requirement could
not be met was Hispanics.

Table 3-6. Sample allocation for sample design D

+---------------+---------+---------+---------+---------+-----------+
| C             | Sample  |         |         | Design  | Effective |
| haracteristic | size    |         |         |         |           |
|               |         |         |         | Effect  | sample    |
|               |         |         |         |         | size      |
+---------------+---------+---------+---------+---------+-----------+
|               | RDD     | List    | Total   |         |           |
+---------------+---------+---------+---------+---------+-----------+
| All veterans  | 13,000  | 7,000   | 20,000  | 1.48    | 13,489    |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 295     | 1,240   | 1,535   | 1.13    | 1,357     |
| group 1       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 271     | 1,199   | 1,470   | 1.12    | 1,308     |
| group 2       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 661     | 1,636   | 2,296   | 1.18    | 1,939     |
| group 3       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 69      | 931     | 1,000   | 2.47    | 405       |
| group 4       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 3,731   | 1,231   | 4,962   | 1.92    | 2,589     |
| group 5       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 36      | 764     | 800     | 1.04    | 773       |
| group 6       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Priority      | 7,937   | 0       | 7,937   | 1.39    | 5,712     |
| group 7       |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Male          | 12,338  | 6,419   | 18,757  | 1.52    | 12,344    |
+---------------+---------+---------+---------+---------+-----------+
| Female        | 662     | 581     | 1,243   | 2.96    | 420       |
+---------------+---------+---------+---------+---------+-----------+
| African       | 1,066   | 574     | 1,640   | 2.52    | 650       |
| American      |         |         |         |         |           |
+---------------+---------+---------+---------+---------+-----------+
| Hispanic      | 520     | 280     | 800     | 2.57    | 311       |
+---------------+---------+---------+---------+---------+-----------+

# 3.2 Sample Selection

The samples from the list and RDD frames were selected independently.
The RDD Sample consists of a sample of telephone households, and the
List Sample consists of veterans sampled from the VA list frame. This
section describes sampling procedures for each of the two components.

####  List Sample Selection

The List Sample is a stratified sample with systematic sampling of
veterans from within strata. The strata were defined on the basis of
priority group and gender. The first level of stratification was by
priority group and then each priority group was further stratified by
gender. Thus, the sample had 12 strata (priority group by gender).

Under the assumption of an 80 percent response rate to the main extended
interview, a List Sample of about 8,750 veterans was anticipated to
yield 7,000 complete interviews. We also decided to select an additional
50 percent reserve List Sample to be used in the event that response
rates turned out to be lower than expected. Therefore, the total sample
size selected from the list frame was 13,125 veterans. With the
systematic sampling methodology, we achieved a total sample of 13,129
veterans from the list frame, out of which a sample of 4,377 veterans
was kept as a reserve sample.

The allocation of the List Sample to the six priority groups, in
combination with the RDD Sample, corresponded to sample design D. Female
veterans were sampled at twice the rate as male veterans while keeping
the List Sample size fixed at 13,129. Table 3-7 provides the allocation
of the List Sample to the 12 sampling strata.

Table 3-7. Allocation of List Sample to sampling strata

+---------------+-------------+-----------------+---------------------+
| Priority      | Gender      | Stratum         | Sample Size         |
| group         |             |                 |                     |
+---------------+-------------+-----------------+---------------------+
| 1             | > Male      | 11              | 2,082               |
+---------------+-------------+-----------------+---------------------+
|               | > Female    | 12              | 242                 |
+---------------+-------------+-----------------+---------------------+
| 2             | > Male      | 21              | 2,027               |
+---------------+-------------+-----------------+---------------------+
|               | > Female    | 22              | 224                 |
+---------------+-------------+-----------------+---------------------+
| 3             | > Male      | 31              | 2,798               |
+---------------+-------------+-----------------+---------------------+
|               | > Female    | 32              | 270                 |
+---------------+-------------+-----------------+---------------------+
| 4             | > Male      | 41              | 1,637               |
+---------------+-------------+-----------------+---------------------+
|               | > Female    | 42              | 110                 |
+---------------+-------------+-----------------+---------------------+
| 5             | > Male      | 51              | 2,127               |
+---------------+-------------+-----------------+---------------------+
|               | > Female    | 52              | 182                 |
+---------------+-------------+-----------------+---------------------+
| 6             | > Male      | 61              | 1367                |
+---------------+-------------+-----------------+---------------------+
|               | > Female    | 62              | 63                  |
+---------------+-------------+-----------------+---------------------+
| Total sample  |             |                 | 13,129              |
+---------------+-------------+-----------------+---------------------+

Based on the sizes of the 12 sampling strata on the list frame and the
allocated sample size for each stratum given in Table 3-7, we used a
systematic random sampling procedure within each stratum to select List
Sample veterans.

We also determined release groups by assigning sequential numbers to
each veteran in the List Sample starting with 1. The release groups were
defined for sample management purposes as discussed later in this
chapter. The List Sample was divided into 15 release groups by assigning
veterans numbered *i, i+15, i+30, i+45,* etc. to the
![](media/image36.wmf) release group, where ![](media/image37.wmf). The
first 4 release groups contained 876 veterans each and the remaining 11
release groups contained 875 veterans each. From those 15 release
groups, we selected two sets (waves) consisting of 5 release groups each
for the March 2001 and the May 2001 List Sample releases. These were
selected by using two sequential systematic samples of release groups
from the 15 release groups. We used the remaining 5 release groups as a
reserve sample to be released only if the actual response rates turned
out to be lower than the assumed response rate of 80 percent.

Although the cooperation rates were quite high, the List Sample targets
could not be met with the initial two waves of the List Sample. This
happened because of out-of-scope sample cases (e.g., deceased,
institutionalized veterans) and sampled cases that could not be traced.
Thus, we released the final wave, which also consisted of 5 release
groups, in June 2001.

####  RDD Sample Selection

####  *National RDD Sample*

We selected the RDD Sample of households using the list-assisted RDD
sampling method. This method significantly reduces the cost and time
involved in such surveys in comparison to dialing numbers completely at
random. The general approach we employed was a two-stage sampling
procedure in which we initially selected a sample of telephone numbers
and successfully screened for households with veterans.

Using the list-assisted sampling methodology, we selected a random
sample of telephone numbers from "one-plus listed telephone banks." This
list-assisted RDD sampling methodology is implemented in the GENESYS
sampling system, which employs a single-stage equal probability sampling
method to select the telephone numbers. The "one-plus listed telephone
banks" are initially sorted by geographic variables, such as state,
metropolitan, and nonmetropolitan, and also by area codes and five digit
prefixes. These sorts construct the sampling frame. The frame is then
divided into implicit strata (almost) equal in size while preserving the
sort ordering. The total number of such implicit strata is the same as
the desired sample size. Then a single telephone number is selected
independently from within each implicit stratum.

Based on propensity estimates from the 1992 NSV RDD Sample, we estimated
that we needed a sample of 135,440 telephone numbers to obtain 13,000
completed extended interviews for the RDD component of the sample. Our
assumptions were:

-   Residential numbers -- 60 percent;

-   Response to screening interview -- 80 percent;

-   Households with veterans -- 25 percent; and

-   Response to extended interview -- 80 percent.

The sample yields at various steps during the RDD Sample selection are
given in Figure 3-1.

+-----------------------------------------------------------------------+
| Telephone Numbers 181,000                                             |
|                                                                       |
| Residency Rate 45%                                                    |
|                                                                       |
| Identified Residential Households 81,450                              |
|                                                                       |
| Screener Response Rate 80%                                            |
|                                                                       |
| Screened Households 65,160                                            |
|                                                                       |
| Households with Veterans 25%                                          |
|                                                                       |
| Identified Veteran Households 16,290                                  |
|                                                                       |
| Extended Interview Response Rate 80%                                  |
|                                                                       |
| Completed Extended Interviews 13,032                                  |
+-----------------------------------------------------------------------+

Figure 3-1. Expected RDD sample yield

From the above calculation, we determined that we required an RDD Sample
of 181,000 telephone numbers to yield 13,032 completed extended
interviews. We also decided to select an additional 30 percent reserve
RDD Sample to be used in the event that the yield assumptions did not
hold (in other words, if the response rate turned out to be lower than
expected). Thus, a total of 235,300 telephone numbers were to be
selected from the RDD frame but we increased the sample size to 240,000
telephone numbers so that 186,000 telephone numbers could serve as the
main RDD Sample and the remaining 54,000 as the reserve sample. We
selected the sample of 240,000 telephone numbers from the GENESYS RDD
sampling frame as of December 2000.

To maintain better control over the RDD sampling yields, the 240,000
telephone numbers selected were divided into 80 release groups, with
each group containing 3,000 telephone numbers. To determine release
groups, sequential numbers starting from 1 were assigned to the
telephone numbers in the RDD Sample. The sampled telephone numbers were
kept in the same order in which they were originally listed in the
GENESYS sampling frame to preserve the implicit stratification of the
sample. The telephone number with the sequence numbers *i, i+80, i+160,
i+240*, etc. constituted release group ![](media/image38.wmf), where
![](media/image39.wmf). From the 80 release groups, systematic samples
of 10 release groups (or waves) were selected almost every month for the
sampling process. The first release (wave) contained 20 release groups
because it included the pretest effort. The sample released in August
contained 19 release groups or 57,000 sampled telephone numbers. Early
in data collection it became clear that the sample yield assumptions
were very optimistic and even the entire RDD Sample of 240,000 telephone
numbers would not be sufficient to produce the required 13,000 completed
extended interviews. Therefore, we decided to select a supplementary RDD
Sample of 60,000 telephone numbers.

####  *Supplementary RDD Sample*

Based on the result of the interim RDD Sample yields, we selected a
supplementary sample of 60,000 telephone numbers from the GENESYS RDD
sampling frame as of June 2001. The supplementary sample was divided
into 20 release groups, and the release groups were assigned the
sequential numbers from 81 to 100. The supplementary sample of 60,000
telephone numbers and the one last release group from the initial RDD
Sample were also released in August 2001.

####  *Puerto Rico RDD Sample*

No listed household information was available for Puerto Rico. As a
result, we used a naïve RDD sampling approach called "RDD element
sampling" (Lepkowski, 1988) instead of the list-assisted RDD method that
we used for the national RDD Sample. With this methodology, all possible
10-digit telephone numbers were generated by appending four-digit
suffixes (from 0000 to 9999) to known 6-digit exchanges consisting of
3-digit area code and 3-digit prefix combinations. A commercially
available[^3] database contained 325 Puerto Rico residential exchanges.
Thus, the Puerto Rico RDD frame was constructed to have 3,250,000
telephone numbers and a systematic sample of 5,500 telephone numbers was
drawn to achieve 176 completed extended interviews.

Before sampling, the frame file was sorted by 6-digit exchange and place
name (or service name).[^4] This implicit stratification permitted a
better representation of the population of households. Also, the
required 176 completed extended interviews was determined proportionally
to the ratio of the Puerto Rico population size (3,803,610) to the U.S.
population size (281,421,906) as of April 1, 2000 from the size of the
main RDD completed interviews (13,000). To achieve this target sample
size of 176 completed extended interviews, the total of 5,500 telephone
numbers was calculated by assuming the residency rate, the screener
response rate, the veteran household rate, and the extended interview
response rate as 20 percent,[^5] 80 percent, 25 percent and 80 percent,
respectively. Unlike the latter three rates, the assumed residential
rate is much lower than that of the national RDD Sample. This is because
RDD element sampling, unlike the list-assisted RDD sampling, does not
have any information on the "one-plus listed telephone banks."

# 3.3 Sample Management

Conduct of the NSV 2001 required not only an effective sample design but
also careful management of the entire sampling process, from creating
the sampling frame to the end of data collection. Before each sampling
step, project staff identified the goals, designed the process, and
prepared detailed specifications for carrying out the procedures. At
each stage, we carried out quality control procedures that involved
checks of counts, cross-tabulations, valid values, and other measures of
correct file processing, transfer, and handling. The remainder of this
section describes the principal areas of sample management.

####  Sampling Frames

####  *List Frame*

Based on the sample design and the data available from the VA source
files, we developed specifications for constructing the list sampling
frame. It included rules for:

-   Defining each sample stratum;

-   Determining the source file and variable for each data element
    needed to construct and stratify the frame; and

-   Defining other needed data items and their sources, such as
    identification and contact information about the veteran.

The VA staff matched the two source files -- the VHA Healthcare
enrollment file and the VBA Compensation and Pension (C&P) file.

####  *RDD Frame*

The national RDD Sample was selected at two different times, and hence
two different frames were used. The RDD frames, differing in the
reference period as of December 2000 and June 2001, consisted of all
"one-plus listed telephone banks" covering the fifty states and the
District of Columbia. The GENESYS Sampling System provided the RDD
sampling frames. As of December 2000, GENESYS had supplied 2,487,468
"one-plus listed telephone banks" in the frame. Thus, the size of the
national RDD sampling frame was 248,746,800 (2,487,468×100). Westat
obtains an updated sampling frame quarterly from GENESYS for conducting
RDD surveys. The updated frame as of June 2001 contained 2,538,453
"one-plus listed telephone banks." As described earlier, the frames were
sorted in geo-metro order before sampling so that the resulting RDD
Samples represented the entire population of households.

####  Sample Release Groups

To ensure that the sample remained unbiased during the data collection
process, we partitioned both the RDD and List Samples into a number of
release groups so that each release group was a random sample. The
sample was released to data collection staff in waves. Each of these
sample waves comprised a number of release groups, which were selected
at random. The small size and independence of sample release groups gave
precise control over the sample. During data collection, we monitored
sample yield and progress toward our targets. When we noticed that a
sufficient number of sample cases from the previous waves had been
assigned final result codes, we released new waves of the sample. Tables
3-8 and 3-9 show the release dates of sample waves and the corresponding
number of cases in each wave for the RDD and List Samples.

We carefully monitored the sample to ensure that we met the data
collection goal of 20,000 completed extended interviews, and that the
sample was representative of the veteran population across the sample
waves (see Table 3-10). Tables 3-11 and 3-12 compare the sample yield
distributions with respect to priority groups and the other demographic
variables of age, gender, and race/ethnicity across sample waves.
Similarly, Table 3-13 compares the sample yield distribution with
respect to levels of education across waves and Table 3-14 compares that
with respect to census regions.

Table 3-8. RDD Sample waves

+-----------------------+-----------------------+-----------------------+
| Sample wave           | Date released         | Number of cases       |
+-----------------------+-----------------------+-----------------------+
| 1                     | > February 12, 2001   | 60,000                |
+-----------------------+-----------------------+-----------------------+
| 2                     | > March 16, 2001      | 30,000                |
+-----------------------+-----------------------+-----------------------+
| 3                     | > May 18, 2001        | 30,000                |
+-----------------------+-----------------------+-----------------------+
| 4                     | > June 11, 2001       | 30,000                |
+-----------------------+-----------------------+-----------------------+
| 5                     | > July 11, 2001       | 30,000                |
+-----------------------+-----------------------+-----------------------+
| 6                     | > August 21, 2001     | 57,000                |
+-----------------------+-----------------------+-----------------------+
| 7                     | > August 30, 2001     | 63,000                |
+-----------------------+-----------------------+-----------------------+
| Puerto Rico           | > April 16, 2001      | 5,500                 |
+-----------------------+-----------------------+-----------------------+

Table 3-9. List Sample waves

+-----------------------+-----------------------+-----------------------+
| Sample wave           | Date released         | Number of cases       |
+-----------------------+-----------------------+-----------------------+
| 1                     | > April 9, 2001       | 4,376                 |
+-----------------------+-----------------------+-----------------------+
| 2                     | > May 14, 2001        | 4,377                 |
+-----------------------+-----------------------+-----------------------+
| 3                     | > July 27, 2001       | 4,376                 |
+-----------------------+-----------------------+-----------------------+

In Table 3-10 the sample yield is defined as the ratio of the number of
completed extended interviews and the number of sampled cases expressed
as a percent. The sample yield was quite uniform over the List Sample
waves but it decreased monotonically over the RDD Sample waves. The
reason for decreasing sample yield for the RDD Sample was that
interviewers had less calling time for the sample waves released later
during the data collection. At various times during data collection we
closed earlier sample waves. Had we not done so, the difference in
yields would have been even larger. Puerto Rico is not included in these
analyses because the Puerto Rico sample was not released in waves.

Table 3-10. Sample yield by sample wave (RDD and List Samples)

+-------+-------+-----------------+-----------------+-----------------+
| S     | Wave  | Cases sampled   | Cases completed | Percent yield   |
| ample |       |                 |                 |                 |
|       |       | \(A\)           | \(B\)           | (B/A) \* 100    |
+-------+-------+-----------------+-----------------+-----------------+
| > RDD | > 1   | 60,000          | 2,840           | 4.73            |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 2   | 30,000          | 1,374           | 4.58            |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 3   | 30,000          | 1,318           | 4.39            |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 4   | 30,000          | 1,307           | 4.36            |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 5   | 30,000          | 1,241           | 4.14            |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 6   | 57,000          | 2,394           | 4.20            |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 7   | 63,000          | 2,431           | 3.86            |
+-------+-------+-----------------+-----------------+-----------------+
|       | >     | 300,000         | 12,905          | 4.30            |
|       | Total |                 |                 |                 |
+-------+-------+-----------------+-----------------+-----------------+
|       |       |                 |                 |                 |
+-------+-------+-----------------+-----------------+-----------------+
| >     | > 1   | 4,376           | 2,384           | 54.48           |
|  List |       |                 |                 |                 |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 2   | 4,377           | 2,317           | 52.94           |
+-------+-------+-----------------+-----------------+-----------------+
|       | > 3   | 4,376           | 2,391           | 54.64           |
+-------+-------+-----------------+-----------------+-----------------+
|       | >     | 13,129          | 7,092           | 54.02           |
|       | Total |                 |                 |                 |
+-------+-------+-----------------+-----------------+-----------------+

Table 3-11. Distribution of completed sample cases by health care
priority group within each wave (RDD and List Samples)

+-------+-------+------+------+------+------+------+------+------+
|       |       | He   |      |      |      |      |      |      |
|       |       | alth |      |      |      |      |      |      |
|       |       | care |      |      |      |      |      |      |
|       |       | prio |      |      |      |      |      |      |
|       |       | rity |      |      |      |      |      |      |
|       |       | g    |      |      |      |      |      |      |
|       |       | roup |      |      |      |      |      |      |
+-------+-------+------+------+------+------+------+------+------+
| S     | Wave  | 1    | 2    | 3    | 4    | 5    | 6    | 7    |
| ample |       |      |      |      |      |      |      |      |
+-------+-------+------+------+------+------+------+------+------+
| > RDD | > 1   | 2.75 | 3.27 | 7.99 | 0.14 | 1    | 1    | 5    |
|       |       |      |      |      |      | 6.97 | 1.76 | 7.11 |
+-------+-------+------+------+------+------+------+------+------+
|       | > 2   | 3.28 | 1.82 | 8.81 | 0.00 | 1    | 1    | 5    |
|       |       |      |      |      |      | 7.54 | 1.28 | 7.28 |
+-------+-------+------+------+------+------+------+------+------+
|       | > 3   | 3.19 | 3.11 | 9.18 | 0.00 | 1    | 1    | 5    |
|       |       |      |      |      |      | 8.06 | 1.46 | 5.01 |
+-------+-------+------+------+------+------+------+------+------+
|       | > 4   | 3.14 | 2.52 | 8.80 | 0.08 | 1    | 1    | 5    |
|       |       |      |      |      |      | 5.46 | 2.55 | 7.46 |
+-------+-------+------+------+------+------+------+------+------+
|       | > 5   | 3.46 | 1.93 | 9.11 | 0.00 | 1    | 1    | 5    |
|       |       |      |      |      |      | 8.53 | 0.72 | 6.24 |
+-------+-------+------+------+------+------+------+------+------+
|       | > 6   | 3.63 | 2.72 | 1    | 0.08 | 1    | 1    | 5    |
|       |       |      |      | 0.03 |      | 7.63 | 0.57 | 5.35 |
+-------+-------+------+------+------+------+------+------+------+
|       | > 7   | 3.54 | 2.26 | 9.13 | 0.08 | 1    | 9.91 | 5    |
|       |       |      |      |      |      | 8.92 |      | 6.15 |
+-------+-------+------+------+------+------+------+------+------+
|       | >     | 3.27 | 2.60 | 8.98 | 0.07 | 1    | 1    | 5    |
|       | Total |      |      |      |      | 7.63 | 1.09 | 6.36 |
+-------+-------+------+------+------+------+------+------+------+
|       |       |      |      |      |      |      |      |      |
+-------+-------+------+------+------+------+------+------+------+
| >     | > 1   | 2    | 1    | 2    | 1.47 | 1    | 6.63 | 7.13 |
|  List |       | 1.48 | 8.62 | 6.13 |      | 8.54 |      |      |
+-------+-------+------+------+------+------+------+------+------+
|       | > 2   | 2    | 1    | 2    | 0.91 | 1    | 6.65 | 7.03 |
|       |       | 1.92 | 9.16 | 6.15 |      | 8.17 |      |      |
+-------+-------+------+------+------+------+------+------+------+
|       | > 3   | 2    | 1    | 2    | 1.25 | 1    | 5.98 | 6.32 |
|       |       | 2.12 | 8.44 | 7.48 |      | 8.40 |      |      |
+-------+-------+------+------+------+------+------+------+------+
|       | >     | 2    | 1    | 2    | 1.21 | 1    | 6.42 | 6.82 |
|       | Total | 1.84 | 8.74 | 6.59 |      | 8.37 |      |      |
+-------+-------+------+------+------+------+------+------+------+

The List Sample allocated to female veterans was 8.3 percent and Table
3-12 shows that the proportion of female veterans in the completed List
Sample was 8.2 percent. Thus, the proportions allocated and completed
were very close and the distribution by gender showed no bias. The
female veterans in the List Sample would have been 9.5 percent if the
proportion of female veterans on the list frame had been the same as
that in the overall veteran population. In fact, the proportion of
female veterans on the list frame was about 4.3 percent as compared to
5.4 percent in the overall veteran population. Therefore, the proportion
of female veterans in the completed List Sample cases was 8.2 percent
because the sampling rate for female veterans was twice that for male
veterans.

Table 3-12. Distribution of completed sample cases by demographic
variables (age, gender, and race/ethnicity) within each wave (RDD and
List Samples)

  -------- ------- ---------------- ------- ------- ------- -------- -------- ------- ------- -------
                   Demographic                                                                
                   variables                                                                  

                   Race/Ethnicity                           Gender            Age             

  Sample   Wave    Hispanic         Black   Other   White   Male     Female   Under   50-64   Over 64
                                                                              50              

  RDD      1       3.73             8.63    4.08    83.56   94.96    5.04     25.92   34.65   39.44

           2       4.15             6.48    5.24    84.13   94.54    5.46     25.62   34.93   39.45

           3       4.02             7.36    4.25    84.37   94.23    5.77     24.89   33.92   41.20

           4       3.90             6.81    4.59    84.70   94.49    5.51     25.48   34.35   40.17

           5       4.83             7.57    3.79    83.80   94.20    5.80     29.90   33.76   36.34

           6       3.68             7.89    5.22    83.21   94.65    5.35     27.78   35.76   36.47

           7       3.91             7.36    4.90    83.83   94.61    5.39     26.66   34.43   38.91

           Total   3.95             7.61    4.61    83.83   94.60    5.40     26.60   34.65   38.74

                                                                                              

  List     1       4.82             12.37   5.96    76.85   91.99    8.01     24.83   34.27   40.90

           2       5.48             12.56   7.25    74.71   91.80    8.20     22.57   34.57   42.86

           3       4.27             13.01   6.27    76.45   91.59    8.41     23.00   34.96   42.03

           Total   4.85             12.65   6.49    76.02   91.79    8.21     23.48   34.60   41.92
  -------- ------- ---------------- ------- ------- ------- -------- -------- ------- ------- -------

Table 3-13. Distribution of completed sample cases by level of education
within each wave\
(RDD and List Samples)

+---------+---------+-----------+-----------+-----------+-----------+
|         |         | Education |           |           |           |
|         |         | level     |           |           |           |
+---------+---------+-----------+-----------+-----------+-----------+
| Sample  | Wave    | No High   | High      | Some      | B         |
|         |         | School    | School    | College   | achelor's |
|         |         |           | Diploma   |           | Degree or |
|         |         |           |           |           | Higher    |
+---------+---------+-----------+-----------+-----------+-----------+
| > RDD   | > 1     | 13.35     | 31.16     | 28.27     | 27.22     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 2     | 14.05     | 29.55     | 29.84     | 26.56     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 3     | 12.67     | 30.05     | 29.51     | 27.77     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 4     | 11.94     | 28.16     | 30.76     | 29.15     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 5     | 14.26     | 30.86     | 28.28     | 26.59     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 6     | 13.74     | 29.62     | 28.91     | 27.74     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 7     | 13.57     | 30.52     | 28.92     | 26.98     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > Total | 13.41     | 30.14     | 29.06     | 27.39     |
+---------+---------+-----------+-----------+-----------+-----------+
|         |         |           |           |           |           |
+---------+---------+-----------+-----------+-----------+-----------+
| > List  | > 1     | 20.64     | 27.60     | 30.03     | 21.73     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 2     | 21.10     | 26.15     | 30.08     | 22.66     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 3     | 19.62     | 27.56     | 31.45     | 21.37     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > Total | 20.45     | 27.12     | 30.53     | 21.91     |
+---------+---------+-----------+-----------+-----------+-----------+

Table 3-14. Distribution of completed sample cases by census region
within each wave\
(RDD and List Samples)

+---------+---------+-----------+-----------+-----------+-----------+
|         |         | Census    |           |           |           |
|         |         | region    |           |           |           |
+---------+---------+-----------+-----------+-----------+-----------+
| Sample  | Wave    | Northeast | Midwest   | South     | West      |
+---------+---------+-----------+-----------+-----------+-----------+
| > RDD   | > 1     | 18.70     | 24.08     | 36.90     | 20.32     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 2     | 18.85     | 22.56     | 37.77     | 20.82     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 3     | 21.17     | 22.31     | 37.03     | 19.50     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 4     | 18.44     | 24.71     | 36.88     | 19.97     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 5     | 17.32     | 25.38     | 35.86     | 21.43     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 6     | 18.71     | 25.10     | 36.51     | 19.67     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 7     | 16.95     | 25.96     | 35.50     | 21.60     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > Total | 18.48     | 24.47     | 36.57     | 20.48     |
+---------+---------+-----------+-----------+-----------+-----------+
|         |         |           |           |           |           |
+---------+---------+-----------+-----------+-----------+-----------+
| > List  | > 1     | 15.94     | 20.43     | 43.16     | 20.47     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 2     | 14.98     | 21.71     | 42.73     | 20.59     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > 3     | 14.39     | 20.83     | 44.42     | 20.37     |
+---------+---------+-----------+-----------+-----------+-----------+
|         | > Total | 15.10     | 20.98     | 43.44     | 20.47     |
+---------+---------+-----------+-----------+-----------+-----------+

We used chi-square statistics to test for homogeneity of distributions
of the sample yield by priority group (Table 3-11), demographic
variables (Table 3-12), level of education (Table 3-13), and census
region (Table 3-14) across waves and found that none of the chi-square
values was significant at 5 percent level of significance (Table 3-15).
Thus, the time effect produced no evidence of bias across different
sample waves.

Table 3-15. Chi-square values for testing homogeneity of distribution of
the sample\
yield by various characteristics (RDD and List Samples)

+---------+----------------+----------+-------------------+-----------+
| Sample  | Characteristic | Ch       | Degrees of        | Pr        |
|         |                | i-square | freedom           | obability |
+---------+----------------+----------+-------------------+-----------+
| > RDD   | Priority       | 44.64    | 36                | 0.15      |
+---------+----------------+----------+-------------------+-----------+
|         | Age            | 19.15    | 12                | 0.09      |
+---------+----------------+----------+-------------------+-----------+
|         | Gender         | 1.53     | 6                 | 0.96      |
+---------+----------------+----------+-------------------+-----------+
|         | Race/Ethnicity | 19.03    | 18                | 0.39      |
+---------+----------------+----------+-------------------+-----------+
|         | Education      | 12.44    | 18                | 0.82      |
+---------+----------------+----------+-------------------+-----------+
|         | Region         | 22.53    | 18                | 0.21      |
+---------+----------------+----------+-------------------+-----------+
|         |                |          |                   |           |
+---------+----------------+----------+-------------------+-----------+
| > List  | Priority       | 7.30     | 12                | 0.84      |
+---------+----------------+----------+-------------------+-----------+
|         | Age            | 4.16     | 4                 | 0.39      |
+---------+----------------+----------+-------------------+-----------+
|         | Gender         | 0.25     | 2                 | 0.88      |
+---------+----------------+----------+-------------------+-----------+
|         | Race/Ethnicity | 8.07     | 6                 | 0.23      |
+---------+----------------+----------+-------------------+-----------+
|         | Education      | 4.47     | 6                 | 0.61      |
+---------+----------------+----------+-------------------+-----------+
|         | Region         | 3.77     | 6                 | 0.71      |
+---------+----------------+----------+-------------------+-----------+

4\. INTERVIEWER RECRUITMENT AND TRAINING

Westat's telephone interviewer and supervisor training program maximizes
trainee involvement, thereby providing the trainer with ample
opportunity to observe and evaluate individual performance. Given that
our interviewers come to us with different levels of skill and
experience, our training program is designed to ensure that they all
finish training with the same understanding of survey research in
general, the specific project goals, the substantive content of the
survey, and Westat's performance expectations. Project training for the
2001 National Survey of Veterans (NSV 2001) began on February 10, 2001
with an initial group of 31 interviewers. In the 5 months following the
first training session, another 267 interviewers were trained in ten
separate sessions, bringing the total number of interviewers trained for
the NSV 2001 to 298. After production was underway, we also conducted
additional training sessions on special topics, such as proxy
interviewing, refusal conversion, tracing calls, and language problem
cases. Interviewers selected to participate in these special sessions
possessed skills relevant to the operation for which they were trained.

# 4.1 Recruitment

We screened prospective interviewers over the telephone, and invited
those with clear, articulate voices for an in-person interview. During
the hiring interview, we asked about related interviewing and telephone
experience and had the potential hire conduct a mock telephone
interview. We then invited individuals with relevant skills and
experience into a training group. We also assigned veteran Westat
interviewers to the project.

# 4.2 Initial Training

Interviewers were recruited and trained in six Westat Telephone Research
Centers (TRC). The TRC operations manager conducted all training
sessions, and the project's telephone center supervisors (team leaders)
attended every session. To ensure that each training group received
identical information, session content was scripted ahead of time and
presented verbatim from the prepared text. During the training program,
the trainer and team leaders observed interviewer performance on
interactive exercises and role plays. At the end of each day of
training, the trainer and team leaders shared their observations,
evaluated interviewer progress, and identified problems to address the
next day. Individuals who consistently failed to meet performance
expectations were released from the project.

Before the project-specific training, new interviewers attended General
Interviewing Techniques training. This 8-hour training covered Westat's
basic telephone interviewing conventions, including listening skills and
probing techniques. We used interactive exercises to teach interviewers
how to answer respondent questions, establish rapport, and gain their
cooperation. Each new interviewer was also required to attend
computer-assisted telephone interviewing (CATI) training, a tutorial on
how to operate a CATI terminal, and the data entry and
response-recording conventions used in Westat's CATI system.

Interviewer training was divided into sessions devoted to specific
topics. For example, Session 1 included an introduction to the study, a
demonstration of an extended interview, a review of frequently asked
questions and answers, and detailed instructions on contact procedures
and administering the screener portion of the interview. Other topics
included how and when to select a proxy for an extended interview,
refusal avoidance, the use of study forms and materials, and the advance
notice letters mailed to the veterans from the U.S. Department of
Veterans Affairs (VA) and Westat. Trainers also explained the
differences in the procedures used for the RDD and List Samples, such as
the verification questions administered to confirm that the correct
sampled veteran had been reached from the List Sample. Exhibit 4-1 shows
the NSV 2001 training agenda.

To reinforce information presented in the first session, interviewers
next participated in scripted role plays. The CATI screening instrument
and extended questionnaire were displayed on each interviewer's CATI
terminal, as well as on overhead screens for the group to view together.
Following the sequence of the questionnaire, each interactive script
reviewed section-specific concepts and definitions. The interviewers
took turns asking the questions from their terminal screens and the
trainer, acting as respondent, gave the scripted answers. The answers
simulated specific situations and problems that interviewers would
encounter in actual interviews, and provided practice with a variety of
likely skip pattern scenarios. As another means of simulating live
interviews, interviewers were paired with each other to practice
additional scripted role plays. Finally, interviewers completed a
written exercise designed to test their knowledge of the presented
material.

Exhibit 4-1. NSV 2001 interviewer training agenda

  ---------------- -------------- ----------------------------------------
  **SESSION 1**                   

  **TIME**         **LENGTH**     **TOPIC**

  9:00-9:15 am     15 minutes     Introduction to the Study

  9:15-9:45 am     30 minutes     Demonstration of Extended Interview

  9:45-10:15 am    30 minutes     Review of Frequently Asked Questions and
                                  Answers

  *10:15-10:30 am* *15 minutes*   *Break*

  10:30-10:55 am   25 minutes     Screener Interactive 1 with Question by
                                  Question Review

  10:55-11:30 am   35 minutes     Screener Interactives 2, 3, and 4

  11:30-1:00 pm    1 hr 30        Screener Contact Procedures
                   minutes        

                                  

  **SESSION 2**                   

  2:00-2:30 pm     30 minutes     Screener Contact Exercise

  2:30-4:00 pm     1 hr 30        Extended Interactive 1 with Question by
                   minutes        Question Review

  *4:00-4:15 pm*   *15 minutes*   *Break*

  4:15-4:45 pm     30 minutes     Identifying Proxies

  4:30-5:00 pm     30 minutes     Extended Contact Procedures

  5:00-5:30 pm     30 minutes     Key Concepts Review (Glossary)

                                  

  **SESSION 3**                   

  9:00-10:00 am    60 minutes     Contact Role Plays

  10:00-11:15 am   1 hr 15        Extended Interactive 2 with Question by
                   minutes        Question Review

  *11:15-11:30 am* *15 minutes*   *Break*

  11:30-11:45 am   15 minutes     Discussion of RDD Sample and List Sample

  11:45-12:00 am   15 minutes     Use of Comments

  12:00-12:30 pm   30 minutes     Verifying List Sample Veterans

  12:30-12:45 pm   15 minutes     Review of Forms (Problem Sheet, Mailout,
                                  Q & A's, Coding Special Contacts)

  12:45-1:00 pm    15 minutes     Interviewer Questions

                                  

  **SESSION 4**                   

  2:00-3:00 pm     60 minutes     Refusal Avoidance Exercise

  3:00-4:00 pm     60 minutes     Role Plays

  *4:00-4:15 pm*   *15 minutes*   *Break*

  4:15-6:00        1 hr 45        Role Plays (cont.)
                   minutes        
  ---------------- -------------- ----------------------------------------

We provided all interviewers with two reference documents, which were
used both in training and during data collection. The *NSV Interviewer
Training Manual* contained background on the study as well as the
specific procedures to be used while on the telephone. The manual
included sections on contacting respondents, answering respondent
questions, recording the results of different contact situations, and
handling contact problems. The *NSV Question-by-Question Specifications*
contained detailed examples, explanations, and definitions for each
survey question in the extended portion of the questionnaire. Both
training documents are provided as separate volumes.

####  Language Problem Training

Bilingual Spanish-speaking interviewers were first tested for fluency.
Those we determined sufficiently fluent were trained to call households
previously identified as Spanish-speaking and determine whether an
English-speaking person lived there.

####  Proxy Interview Training

In a number of cases, we determined that veterans could not answer the
questions for themselves. If, for example, the sampled veteran had
difficulty hearing or speaking, or was ill or incapacitated,
interviewers were instructed to code the case "Proxy Needed" and obtain
the name and telephone number of the person most knowledgeable about
that veteran's health and medical situation. Cases coded this way were
automatically put into a special CATI work class, ensuring that only
interviewers trained to conduct proxy interviews were assigned to them.
Over the course of data collection, 23 highly-skilled interviewers were
selected from the NSV 2001 staff and trained to conduct proxy
interviews. With slight modifications, the same CATI questionnaire used
to interview the veterans themselves was also used for the proxy
respondents. Therefore, we carefully instructed proxy interviewers to
re-phrase each question so that it was clear to proxies that they were
being asked about the veteran, not themselves. The proxy interview
training took about 1 hour.

####  Tracing Training

We conducted a 1-hour tracing training at the Rockville telephone center
with six interviewers possessing previous tracing experience and
excellent probing skills. Training included instruction on the tracing
sheet that listed all previous contact information collected for the
veteran, the use of directory assistance to locate a good telephone
number, and the procedures for documenting all call attempts and their
results. Additionally, trainers reviewed the procedures for following up
on a potential lead and verifying that a telephone number, when not
listed under the List Sample veteran's name, was nevertheless one at
which we could reach that veteran.

####  Refusal Conversion Training

For refusal conversion training, we selected experienced interviewers
who had above average cooperation rates. In this 2-hour session,
interviewers discussed the kinds of refusals they had already
encountered and shared ideas for addressing respondent objections. To
sharpen interviewer presentation and timing of responses to refusals,
session participants conducted role play exercises. Other participants
observed the role playing pair and provided feedback and additional
ideas for responding to refusals. Interviewers made notes of their
favorite ideas for converting refusing respondents and incorporated them
into their own repertoire.

5\. DATA COLLECTION

Nearly 300 interviewers collected data for the 2001 National Survey of
Veterans (NSV 2001) over a 9-month period. This chapter describes the
computer system we used to collect the data, the procedures we employed
to ensure successful interviews with respondents, and the special
operations we instituted to enhance response rates.

# 5.1 The CATI System

####  Questionnaire Administration

Westat's proprietary Cheshire computer-assisted telephone interviewing
(CATI) system is designed to handle very large and complex survey
instruments. Not only does this system greatly enhance data quality by
automating most data collection tasks, it also reduces respondent burden
by streamlining the questions. The CATI system is particularly
appropriate for the NSV 2001, with its complicated questionnaire, List
and RDD Sample types, and household screening requirements for RDD
Sample cases. By using the CATI system, we ensured that the NSV 2001
instrument was administered correctly, the samples were managed
efficiently, and data were recorded properly. For example, the software
guided interviewers through the programmed skip patterns; performed
range, logic, and consistency checks; determined when cases should be
called and how often; kept track of which cases belonged to each sample;
and stored all data to a common database. Detailed descriptions of these
and other CATI system capabilities follow.

-   **Display.** The standardized display of survey questions,
    interviewer instructions, and response options contributed to better
    data quality by helping to ensure that each interviewer administered
    the instrument in the same manner. The program also displayed all
    relevant respondent information, automatically inserting into
    question text personalized data such as the respondent's name.

-   **Sample Management.** The CATI system differentiated between the
    RDD and List Samples, and provided the appropriate questions for the
    interviewer to ask. RDD Sample respondents automatically received
    the household screening questionnaire (see Appendix A), while List
    Sample members were administered a verification question (see
    Appendix B). The program implemented decision rules based on
    military service questions in the screening questionnaire and the
    extended questionnaire to determine eligibility. It then selected
    eligible persons for an interview and prompted interviewers to
    terminate the interview for ineligible respondents. The CATI system
    also was able to evaluate the samples separately in reference to
    their target production goals.

-   **Online Data Entry.** Data were captured instantly in the CATI
    system when interviewers typed respondent answers as they conducted
    the interview. The program offered a variety of ways to correct
    keying errors. Interviewers corrected data entry mistakes
    immediately by backing up through the CATI screens, reentering
    information, then moving forward again through the corrected
    question path. For mistakes discovered after the interview was over,
    or those too time consuming to correct during the interview,
    interviewers entered online comments for review by the database
    manager, or submitted a hard copy problem sheet to their shift
    supervisor. (See Chapter 7 for a more thorough discussion of
    database management.)

-   **Skip Patterns.** The NSV 2001 questionnaire contained numerous,
    often complicated, skip patterns. Once the questionnaire was
    programmed into the CATI system, however, these skips were
    administered automatically and consistently. As the interviewer
    entered a respondent's answers, the program determined and then
    displayed the correct question path. The CATI system accessed the
    sample file or other data collected from the respondent and
    presented specific sections of the instrument or different versions
    of the questions in accordance with programming instructions. The
    software also used existing data to determine whether or not to
    administer certain questions. For example, if the veteran provided
    exact dates of induction and release from active duty in the
    Military Background section of the main interview, the CATI system
    calculated and automatically entered that veteran's periods of
    service. This feature reduced response burden for some veterans,
    which decreased overall survey administration time.

-   **Range Checks.** The CATI system is capable of performing both hard
    and soft range checks. Hard range checks set the upper and lower
    limits of allowable item responses. Hard ranges were required for
    all variables. A question with response categories of 1 through 12
    accepted only those values, in addition to the required missing
    values of "Don't Know" and "Refused." Exhibit 5-1 is an example of a
    hard range check. Soft range checks queried implausible responses
    but allowed the entry to stand if unchanged after being probed by
    the interviewer. We used soft ranges for questions about dates or
    quantitative information. Exhibit 5-2 is an example of a soft range
    check.

-   **Logic and Consistency Checks.** Logic checks ensured that data
    collected at different points in the interview were internally
    consistent. If responses were inconsistent, interviewers were
    prompted through a series of custom screens to verify the recorded
    data. The program accepted corrected data when obtained. If the
    responses remained inconsistent, the case was flagged for review by
    the database manager and the interview continued. Exhibit 5-3 is an
    example of a logic check.

Exhibit 5-1. Example of CATI specifications with hard range

**PROGRAMMER NOTE 1:**

**IN ALL MONTH FIELDS, HARD RANGE = 1 -- 12.**

**IN ALL INSTANCES WHERE OTHER SPECIFY = YES PROVIDE 30 CHARACTER OTHER
SPECIFY FIELD.**

**IN MB0a, RANGE FOR YEAR = 1885 -- (1983).**

**VETS.DOBMM, VETS.DOBYYYY**

MB0a. First, I'd like to ask you for the month and year you were born.

> \|\_\_\|\_\_\| MONTH
>
> \|\_\_\|\_\_\|\_\_\|\_\_\| YEAR
>
> REFUSED -7
>
> DON'T KNOW -8
>
> 1\. JANUARY 7. JULY
>
> 2\. FEBRUARY 8. AUGUST
>
> 3\. MARCH 9. SEPTEMBER
>
> 4\. APRIL 10. OCTOBER
>
> 5\. MAY 11. NOVEMBER
>
> 6\. JUNE 12. DECEMBER

Exhibit 5-2. Example of CATI specifications with soft range

**PROGRAMMER NOTE 63:**

**IN (SD13) DEPEND HARD RANGE = 0 -- 15. SOFT RANGE = 0 -- 10.**

\[O6a.\]

**VETS.DEPEND**

SD13. During the year 2000, how many children depended on you for at
least half of their support?

> NUMBER \|\_\_\|\_\_\|
>
> REFUSED -7
>
> DON'T KNOW -8

Exhibit 5-3. Example of CATI logic check

> 1.1501 MB15A 100010300101 - (410) 555-7834 - 16:13
>
> I have recorded that you were released from active duty on
>
> January 5, 1934. That date is earlier than the
>
> date you began active duty. Please tell me the date you
>
> [began]{.underline} your active duty.
>
> (5 ) (5 ) (1931 )
>
> MONTH DAY YEAR
>
> 1\. JANUARY 7. JULY
>
> 2\. FEBRUARY 8. AUGUST
>
> 3\. MARCH 9. SEPTEMBER
>
> 4\. APRIL 10. OCTOBER
>
> 5\. MAY 11. NOVEMBER
>
> 6\. JUNE 12. DECEMBER

1.1502 MB15B 100010300101 - (410) 555-7834 - 16:13

And again, what was the date you were [released]{.underline} from active
duty?

(1 ) (5 ) (1945 )

MONTH DAY YEAR

1\. JANUARY 7. JULY

2\. FEBRUARY 8. AUGUST

3\. MARCH 9. SEPTEMBER

4\. APRIL 10. OCTOBER

5\. MAY 11. NOVEMBER

6\. JUNE 12. DECEMBER

-   **Data Storage.** The CATI system stored all administrative and
    substantive data collected during each interview administration. One
    segment of the software recorded each attempted contact with every
    respondent. It stored the date and time of the call attempt,
    interviewer identification, work class, disposition code, and
    termination information. Another segment recorded every keystroke
    made during every interview. If an interview was interrupted and
    data lost due to computer failure, this file was used to restore the
    original responses. The system also stored all hard range failures
    and soft range overrides.

####  CATI Case Management and Call Scheduling

Telephone numbers loaded into the CATI system became available to
interviewers through the CATI scheduler. An autodialer dialed the
numbers, reducing interviewer dialing time as well as eliminating the
possibility of dialing a telephone number incorrectly. The CATI
scheduler kept track of the number of calls made to each telephone
number and automatically closed out those that had reached the maximum
number of contact attempts without completing an interview. The CATI
scheduler also ensured that cases were called at the appropriate times,
using rules developed to minimize the number of calls to any given
household and to reduce nonresponse. For example, the week was divided
into day and time period categories through which the system moved each
case in a specified pattern of call attempts. If the first call attempt
was in the evening and resulted in no answer, the CATI scheduler
automatically set the next call attempt for another time of day and a
different day of the week. For cases where the interviewer made contact
and scheduled an appointment to complete the screener, or to begin or
continue the extended interview, the system held the case then released
it to the next available interviewer at the scheduled day and time.
Screener interviews that were interrupted were restarted at the
beginning of the screening questionnaire when the household was reached
again. Interrupted extended interviews began again at the beginning of
the questionnaire section from which the case had exited. Cases with
scheduled appointments were given priority over all other cases.

Another function of the CATI scheduler was to manage the flow and
assignment of cases to interviewers. The scheduling system analyzed a
number of factors when determining what case to assign to a particular
interviewer, including interviewer skills required, call history, time
zone of the respondent, day of the week, time of day, and priority
weighting of cases by sample group. This analysis required no indepth
knowledge on the part of the interviewer, and it maximized the number of
completed interviews while minimizing the number of nonproductive
interviewer hours. The CATI scheduling system also appropriately
assigned to specially trained interviewers those cases that needed to be
traced, refusal cases, language problems, and cases requiring proxy
interviews. Finally, the system maintained a log of all call attempts
and their results.

The CATI scheduler was customized to accommodate the two NSV 2001
samples in a variety of ways. First, we programmed the CATI system to
track multiple telephone numbers for each List Sample veteran. When an
interviewer learned of a new telephone number for a veteran, it was
entered directly into the system and dialed at the next call attempt.
Project staff or the database manager entered telephone numbers
collected offline from sources other than household members (such as the
VA and directory assistance). Multiple telephone numbers were also
tracked for RDD veterans (for example, veterans who had moved out of the
household they were living in at the time the screener survey was
completed).

We further programmed the CATI scheduler to optimize when and how often
a case was called based on sample type. Calls to List Sample members
were made primarily in the evenings and on weekends because we were much
more likely to contact a household resident during those hours. Initial
calls to the RDD Sample were made primarily during the day. While this
did not optimize household contact, interviewers were able to quickly
close out the nonworking and business numbers that remained even after
the sample had been prescreened for them.

####  Programming and Testing the CATI System

The CATI development staff was led by a systems analyst who was
responsible for coordinating all survey programming and testing
activities. The systems analyst worked closely with the questionnaire
designers to ensure that the CATI program specifications addressed every
aspect of the NSV 2001 instrument. The specifications included variable
name and location, valid response categories, and instructions for item
nonresponse. The programming staff then used those specifications to
program the instrument, verifying that the resulting instrument
conformed to the specifications and ensuring that no errors were hidden
in the specifications.

Westat thoroughly tested all features of the system, including those
associated with scheduling, interviewing, data storage, and editing.
After each section was programmed, it went through multiple rounds of
intensive internal testing. In the first round, the programmers used the
specifications to proof the screens, check the data dictionary, and
match the programmed language flow against the survey instrument's skip
patterns. The next round of testing focused on transitions from one
questionnaire section to another. We also tested restart points to
ensure that the flow between sections and topics was smooth, and that
questionnaire sections appeared in the proper sequence.

For the final round of testing, result codes, randomization routines,
standard function keys, database fields, flags, array variables, timing
variables, audit trails, and database structure were validated. We
proofed all text on the screens a final time and confirmed the accurate
collection of data into the survey database. Testing scenarios also
targeted the questionnaire delivery system (the CATI scheduler). The
focus here was on the proper delivery of cases to interviewers, whether
appointments were correctly made and kept, and how interview break-offs
were treated.

Just as during the data collection period, forms were filled out
whenever a potential problem was found. These problem sheets described
the situation, listed key variables, indicated expected results, and
documented the actual result. The database manager logged each problem
sheet and routed it to the appropriate person. Once the problem was
resolved and the changes were implemented in the testing environment,
the database manager sent the problem sheet back to the originator so
that the test could be rerun. This process ensured that at least two
people verified that a problem had been resolved. Westat kept its CATI
program and systems documentation, test plans, and test sets up to date,
thus ensuring that changes to the instrument could be easily
incorporated and tested. CATI system version control software maintained
a history of changes.

# 5.2 Interviewing Operations

Data collection for the NSV 2001 began on February 12, 2001 and ended
November 12, 2001. Interviewing was conducted from six of Westat's
Telephone Research Center (TRC) facilities. The telephone centers
operated seven days a week.

####  Management, Staffing, and Scheduling

The telephone operations manager coordinated all training and
interviewing activities with supervisory staff in the six TRCs. For
training, the operations manager prepared the agenda, produced the
manuals, scripts, and interviewer forms, and created exercises and role
plays. The operations manager also developed procedures for special
operations such as refusal conversion, proxy interviewing, and tracing.
During data collection, the operations manager reviewed and resolved
problem cases. To help evaluate the progress of data collection, the
operations manager produced and presented to the project team a weekly
summary of the status of all cases by screener and main interview, by
interim and final results, and by sample type. Finally, the operations
manager oversaw all interviewer scheduling and staffing.

Interviewers made most calls in the evenings and on weekends, when
people were most likely to be home. We assigned primary responsibility
for handling language problem cases to experienced Spanish-speaking
interviewers. A case was coded as a language problem if an interviewer
was unable to communicate with the respondent in English.
Spanish-speaking interviewers then recontacted the case to determine if
an English-speaking person lived in the household. We trained
interviewers who had clear, deep voices to follow up with hearing
problem cases. We chose skilled interviewers at all six TRC facilities
to conduct refusal conversion interviewing. We selected interviewers at
one location to perform the majority of tracing calls, and updated the
CATI system with the List Sample telephone numbers obtained through
those efforts.

####  Interviewer Monitoring

Supervisors regularly monitored interviewer performance. Because most
refusals occur at the point of contact, supervisors paid particular
attention to the initial contact. Monitoring sessions lasted a minimum
of ten minutes, during which supervisors made notes of interviewers'
strengths and weaknesses on a monitoring form. Supervisors discussed the
results with each interviewer immediately following a monitoring
session. They provided positive feedback along with pointers for
improvement in areas such as gaining cooperation and probing.
Supervisors also used a weekly interviewer productivity report to
identify candidates for training on more difficult tasks such as refusal
conversion, as well as interviewers who needed additional training.

####  Confidentiality

All Westat personnel, including interviewers and professional staff,
signed a statement that they would maintain the confidentiality of all
survey data. During data collection, interviewers assured each
respondent that his or her answers were protected under the Privacy Act
(see Exhibit 5-8) and informed respondents that a copy of the Privacy
Act statement was available upon request (see "Providing a Privacy Act
Letter" later in this chapter).

# 5.3 Special Operations

In addition to the standard data collection operations described in the
previous sections, Westat employed a variety of strategies unique to the
NSV 2001 for achieving the maximum possible response rates. These
special operations included contact information update measures, proxy
interviewing, and response rate enhancement measures such as refusal
conversion and data retrieval.

####  Contact Information Update Measures

####  *Cleaning the List Sample and Matching Addresses*

Before loading the List Sample contact information into our CATI system,
we processed the file through a series of automated and manual cleaning
procedures that standardized the appearance of all addresses. Once the
sample was cleaned, we matched cases with address information against
the National Change of Address Registry to ensure that we had the most
recent address. We also matched all cases with address and/or telephone
information against the databases of two address matching vendors. This
step allowed us to either obtain a telephone number for those cases with
no telephone numbers or to update existing telephone numbers. Those
veterans for whom we had a telephone number but no address were loaded
into the CATI system without mailing an advance letter. Those cases for
which we had a Social Security number and an address, but no telephone
number, were sent to a credit bureau for telephone information searching
by Social Security number. Those cases for which we had an address but
no telephone number were sent an advance letter, which asked for updated
contact information, including a telephone number. These cases, along
with those for which we had neither a telephone number nor address, were
sent immediately to our tracing operation. During the cleaning process,
veterans identified as institutionalized were coded as out of scope so
that no further contact attempts would be made.

####  *Sending an Advance Mailing*

After List Sample cleaning and address matching, we sent each List
Sample veteran an advance mailing that included a letter from the
Secretary of the Department of Veterans Affairs and a letter from the
Westat project director (see Exhibits 5-4 and 5-5). The letters informed
veterans about the study and the importance of their participation. We
included an address update form and postage-paid envelope so that
veterans could update their names, addresses, or telephone numbers by
mail. Veterans were also given the option to call the NSV 2001 dedicated
toll-free number to provide their current address and telephone
information. To help us identify institutionalized veterans, the address
update form also asked veterans to indicate whether the address was for
a private residence, hospital, assisted living facility, retirement
home, or nursing home. (As discussed in Chapter 3, we did not interview
List Sample institutionalized veterans.)

We mailed 13,010 advance letters in two waves. In response to the
advance mailings, we received 4,061 address update forms through the
mail. Of those, however, less than half contained contact information
updates. Those with no update did indicate whether their address was for
a private residence or some other type of residence. We reviewed all
additional information we received about a veteran, then entered it into
the CATI Update system where it became effective immediately. Table 5-1
shows the type of information received on the address update form.

Table 5-1. List Sample advance mailout responses

  -------------------------------------------------- --------------------
  Address update                                     825

  Telephone number update                            917

  Name correction                                    48

  Deceased                                           197

  Not a veteran                                      1

  Form returned with no update                       2,073

  Total responses received                           4,061
  -------------------------------------------------- --------------------

Exhibit 5-4. VA advance letter

![](media/image40.png){width="6.485416666666667in"
height="8.749305555555555in"}

Exhibit 5-5. Westat advance letter

WESTAT

National Survey of Veterans

1650 Research Blvd. • Rockville, MD 20850-3129 • 301 251-1500 • FAX 301
294-2040

> \<\<DATE\>\>

\<\<BASMID\>\>

\<\<FULLNAME\>\>

\<\<ADDR1\>\>

\<\<ADDR2\>\>

\<\<CITY\>\>, \<\<STATE\>\> \<\<ZIP\>\>

Dear \<\<FULL_LOWER\>\>:

The Department of Veterans Affairs (VA) has chosen Westat to conduct its
National Survey of Veterans. The enclosed letter from Anthony J.
Principi, Secretary of Veterans Affairs, explains that the VA is
conducting this survey to obtain information for planning benefits and
services for all veterans.

A Westat interviewer will call you in the next few weeks to conduct a
telephone interview. Your answers, and those of other participating
veterans, will give the VA an up-to-date picture of the whole U.S.
veteran population. If you wish to verify Westat's role in the survey,
please call the toll-free VA number mentioned in Secretary Principi's
letter: 1-800-827-1000.

While your participation is voluntary, you are one of only a few
thousand randomly selected veterans whose answers will represent all 25
million veterans. Your individual participation will have a significant
impact on the survey's outcome. We cannot interview another veteran in
your place. I want to assure you that the information you provide is
protected under the Privacy Act and section 5701 of Title 38 of the U.S.
Code. The VA will use the information you provide to evaluate current VA
policies, programs and services for veterans and in deciding how to help
veterans in the future.

Enclosed is a National Survey of Veterans Address Correction and Update
Form. Westat will call you at the phone number printed on this form. If
this phone number or address is incorrect, please provide the correct
information and mail the form to Westat in the postage-paid envelope
provided. If you prefer, you may call Westat toll-free at 1-888-258-2194
to provide a correct phone number and address. Ask to speak with the
Veterans Survey Manager.

Thank you. We greatly appreciate your help.

> Sincerely,
>
> ![](media/image41.png)
>
> John C. Helmick
>
> Project Director

####  *Conducting Tracing Operations*

Tracing is a process by which we attempt to find valid contact
information for veterans for whom we have no or incomplete information
in our records. For some List Sample veterans, we did not have a valid
telephone number, the existing number was nonworking, or we did not have
a telephone number at all. In an effort to contact these veterans, we
placed directory assistance calls. We pursued two leads per tracing
case. If the veteran was not listed at the address in our records, and
there were six or fewer listings with that veteran's last name in the
town or surrounding area, tracing interviewers requested the first two
telephone numbers. The interviewers called both households asking for
information about the veteran. When we identified new telephone numbers,
we entered them into the CATI system within 24 hours. Over the course of
data collection, tracing interviewers worked 5,350 cases.

####  *Contacting Cases With Unpublished Telephone Numbers*

During the tracing operation, we often encountered List Sample veterans
who had an address in our records but whose telephone number was not
published. We mailed these veterans a letter requesting their
participation in the study and providing a toll-free number for them to
contact us (see Exhibit 5-6). We mailed these letters to 555 veterans,
of whom 124 completed the extended interview. We also learned that 17 of
the 555 had died and 4 were institutionalized.

####  *Contacting Cases With No Telephone Numbers*

We had no telephone number and only a Post Office Box address for 65
List Sample veterans. Without a street address, we were unable to
process these cases through our tracing operation. Instead, we mailed a
letter to these veterans asking them to call the NSV 2001 toll-free
number for an interview (see Exhibit 5-7). Of the 65 veterans who
received this letter, 15 completed an extended interview. One was
reported deceased.

Exhibit 5-6. Letter to veterans with unpublished telephone numbers

WESTAT

National Survey of Veterans

1650 Research Blvd. • Rockville, MD 20850-3129 • 301 251-1500 • FAX 301
294-2040

> DATE

«BASMID»

«FULLNAME»

«ADDR1»

«CITY», «STATE» «ZIP»

Dear «FULL_LOWER»:

**We need your help.** The Department of Veterans Affairs (VA) has
chosen Westat to conduct its National Survey of Veterans. We would like
to interview you for the survey. Unfortunately, we have been unable to
reach you because your telephone number is unpublished.

The VA is conducting this survey to obtain information for planning
benefits and services for all veterans. While your participation is
voluntary, you are one of only a few thousand randomly selected veterans
whose answers will represent all 25 million veterans. Your individual
participation will have a significant impact on the survey's outcome. We
cannot interview another veteran in your place. Your answers, and those
of other participating veterans, will give the VA an up-to-date picture
of the whole U.S. veteran population.

I want to assure you that the information you provide is protected under
the Privacy Act and section 5701 of Title 38 of the U.S. Code. The VA
will use the information you provide to evaluate current VA policies,
programs and services for veterans and in deciding how to help veterans
in the future.

We understand your desire to keep your telephone number closely guarded.
For that reason, we invite you to call Westat toll-free at
1-888-258-2194 to begin an interview or schedule an appointment at a
time that is convenient for you. Please ask to speak with the Veterans
Survey Manager. Or, you may provide us with your telephone number on the
enclosed National Survey of Veterans Address Correction and Update Form
and mail it to Westat in the postage-paid envelope. Your telephone
number will only be used to contact you for completing the National
Survey of Veterans and will not be disclosed for any other purposes.

Thank you. We greatly appreciate your help.

> Sincerely,
>
> ![](media/image42.png)
>
> John C. Helmick
>
> Project Director

Exhibit 5-7. Letter to veterans with no telephone number

WESTAT

National Survey of Veterans

1650 Research Blvd. • Rockville, MD 20850-3129 • 301 251-1500 • FAX 301
294-2040

DATE

FNAME LNAME

ADDRESS

CITY, STATE ZIP

Dear Mr./Ms./Mrs. LNAME:

Thank you for your recent reply to our letter regarding the National
Survey of Veterans. Westat is conducting this study for the Department
of Veterans Affairs.

Based on your response, we understand that you are not able to receive a
telephone call in order to complete the survey. Since it is necessary
that all surveys be completed by telephone, and your responses are very
important to us, we would like to provide you an option that would make
it possible for you to participate in the survey.

To allow us the opportunity to complete a survey with you, we are asking
that you call our toll-free telephone number and ask to speak to one of
our telephone interviewers. Please tell our staff that you are calling
to complete the **National Survey of Veterans**. Once you have
identified the survey for which you are calling to participate, you will
need to give your survey identification number. This number is listed
below as the **ID Number**. Please try to call within the next 2 weeks
and during the hours outlined below.

Phone Number: **1-888 258-2194**

Project: **National Survey of Veterans**

ID Number: **BASMID**

Calling Hours:

> **Monday-Friday 9:00 am -- 10:00 pm (Eastern Daylight Time)**

**Saturday 10:00 am -- 4:00 pm (Eastern Daylight Time)**

**Sunday 2:00 pm -- 8:00 pm (Eastern Daylight Time)**

Thank you very much for helping us with this important study. We look
forward to hearing from you.

Sincerely,

![](media/image43.png){width="1.698611111111111in"
height="0.5659722222222222in"}

John C. Helmick, Jr.

Project Director

####  *Searching Credit Bureaus*

We sent the Social Security numbers of veterans for whom we had no
telephone number (or an invalid telephone number) to a credit bureau.
Toward the end of data collection, we performed the same operation on
all cases coded as nonlocatable and all cases in our tracing queue. Some
of these nonlocatable and tracing cases were likely the same ones that
had been processed through the credit bureau at the beginning of the
study. Most, however, were cases for which we had incomplete contact
information. The cases returned from the credit bureau with addresses
were then sent to our address matching vendors for telephone numbers.
Table 5-2 shows the results of the end-of-data collection credit bureau
search.

Table 5-2. Results of end-of-data collection credit bureau search

  --------------------------------------------------------- -------------
  Number of cases sent to the credit bureau                 3,197

  Number for which a telephone number was obtained          513

  Complete extended interviews                              150

  Deceased                                                  6

  Institutionalized                                         5
  --------------------------------------------------------- -------------

####  Proxy Interviewing

The NSV 2001 data collection plan permitted veterans with serious
health, cognitive, or communication problems to be represented in the
final sample, despite the fact that many of these veterans were unable
to respond for themselves or, at least, unable to respond to a telephone
interview. Veterans with such disabilities were:

-   Too mentally incapacitated to understand the questions or to
    remember or formulate their responses;

-   Too physically incapacitated, too ill, or too weak to listen and
    respond to the questions for the duration of the interview;

-   Too hearing-impaired to hear the interview questions; or

-   Too speech-impaired to communicate their responses to the
    interviewer.

We included the responses of veterans with these disabilities (referred
to as the sampled veterans) by allowing proxy respondents to participate
in the survey for them. Our estimates of the number of cases that would
require proxies, while large enough to confirm the need for a proxy
interview protocol, did not justify an entirely separate data collection
instrument and methodology. Instead, the proxy interview protocol used a
modified form of the existing CATI data collection instrument and
telephone methodology.

We established rules to help interviewers determine when a proxy
interview was appropriate, permitting them only under the four
circumstances described above. We did not interview proxy respondents
for any other reason, such as convenience, personal preference,
temporary unavailability of the veteran, or temporary illness from which
the veteran would recover before the end of the data collection period.
The most common way in which interviewers identified the need for a
proxy was when a friend or relative (and occasionally the veterans
themselves) reported that the sampled veteran was incapable of
participating in the study. In much rarer situations, the veteran was
willing to be interviewed, but during the course of the interview it
became apparent that he or she could not comprehend the questions or
remember any reasonable amount of information about himself or herself.

We also trained interviewers to identify a suitable proxy for veterans
who could not participate and to secure the telephone number needed to
call the proxy. Such a proxy needed to be familiar with the sampled
veteran's circumstances and possess some specific knowledge of his or
her health and medical history. In most cases, the person who informed
the interviewer of the veteran's inability to respond was the
appropriate proxy, and the proxy was often located at the same telephone
number. If the potential proxy was not knowledgeable enough about the
veteran or was unavailable, interviewers attempted to find another
proxy. If no proxy could be found or the suitable proxy refused to
respond, the final result of the case reflected the sampled veteran's
situation, not the outcome of contacting the proxy.

We trained a small group of interviewers to conduct proxy interviews. If
a proxy interviewer identified the need for a proxy in the course of
ordinary contact attempts, and the proxy was available, he or she
proceeded directly with the proxy interview. If, however, an interviewer
without proxy interview training identified such a need, an appointment
was made and the case was held in the CATI scheduler proxy work class
for a proxy interviewer. To accommodate the time needed to identify and
train proxy interviewers, we did not begin proxy interviewing until
April, 2001. All proxy cases from the beginning of data collection were
held in the proxy work class until that time, and proxy interviewing
continued through the end of data collection.

As noted earlier, we knew that relatively few veterans would require a
proxy interview. This fact, coupled with our desire for comparable data
from the proxy cases and the self-respondents, led us to modify the
existing questionnaire rather than create a separate proxy instrument.
We identified all questions in the NSV 2001 questionnaire that proxies
would likely be able to answer for the sampled veterans. We eliminated
some questions (such as the VSF12 items in the Health Background
section) that were too personal for a proxy to answer adequately.
Finally, we trained proxy interviewers to rephrase select questions in
the third person to remind proxies that while the questions were
directed at them, they were in fact about the sampled veteran. Rather
than program the CATI instrument to display all the third person
pronouns, we trained the proxy interviewers to use the third person as
they read the questions. We chose this cost effective measure because
few questions actually needed rephrasing, and any rephrasing usually
occurred within a pattern of repeating questions. Furthermore, the
interviewers doing the rephrasing, in addition to being highly skilled,
also had at least 2 months of experience administering the questionnaire
before proxy interviewing began.

We expected "Don't Know" responses to be more common among proxy
respondents than self-respondents. We found, however, that proxy
respondents were often very knowledgeable about the sampled veteran's
background information, military service, medical conditions, and
treatment. When proxies did have limited knowledge in certain areas,
interviewers encouraged them to provide the best response they could. In
a few cases, attempts to contact the proxy resulted in the discovery
that a sampled veteran could, after all, respond for himself or herself.
In these instances, we conducted the interview with the veteran, and the
case was correctly coded as a regular, nonproxy interview.

We identified 1,031 cases as needing a proxy respondent. We completed
interviews with 734 proxy respondents. Another 28 cases turned out to be
ineligible for the study. Of the cases initially identified as needing
proxies, an additional 45 interviews were eventually conducted with the
sampled veteran. Thus, only 3.7 percent of all interviews were conducted
with proxies.

####  Response Rate Enhancement Measures

####  *Establishing a Toll-free Telephone Number* 

We established a toll-free telephone number dedicated solely to the NSV
2001. Only supervisory staff answered this number. We included this
number on all mailout materials, and provided it to respondents who
wanted more information about the study before consenting to an
interview. Veterans with no telephone in the household called the
toll-free number to complete an interview (see the previous sections on
"Contacting Cases with Unpublished Telephone Numbers" and "Contacting
Cases with No Telephone Numbers"). Difficult-to-reach respondents often
called the toll-free line to participate, or at least tell us when they
would next be available for an interview.

Respondents who wished to contact the VA directly about the NSV 2001
were also provided with the VA's main toll-free number (800-827-1000).
This number offered an automated menu where veterans could obtain
information about various VA benefits, or remain on the line for the
next available counselor. Because information about the NSV 2001 was not
part of the automated menu, veterans had to wait for a counselor in
order to inquire about the legitimacy of the survey. This posed two,
somewhat isolated, problems. The first is that veterans were only able
to get help with their questions during business hours, when customer
service representatives/benefits counselors were available. The second
is that a few VA customer service representatives were unaware of the
study, and hence unable to reassure respondents of its legitimacy.

####  *Developing a Brief and Effective Introduction to the Survey*

In our experience, most telephone survey refusals occur in the first few
seconds of the call. With that in mind, we scripted an introduction that
briefly stated who Westat is, the sponsoring agency, and the importance
of the study. With such an introduction, we intended to quickly provide
respondents with enough information to decide, at the very least, to
listen to and answer our screening questions (or in the case of the List
Sample, the verification question).

####  *Preparing a List of Frequently Asked Questions*

We developed a list of what we anticipated would be Frequently Asked
Questions about the NSV 2001, Westat's role in the study, and VA goals
(see Appendix D). We then required interviewers to learn the answers to
these questions well enough that they could talk about the study with
respondents naturally, instead of sounding like they were reading the
answers.

####  *Providing a Privacy Act Letter*

Some veterans requested written information about the study, saying they
would not participate until they received it. Within one business day of
such requests, we mailed a letter that described the NSV 2001 and
respondents' rights under the Privacy Act (see Exhibit 5-8) or a
background letter (Exhibit 5-4). We assigned a "mailout needed" code to
these cases and held them for two weeks before calling them again. We
mailed letters to 3,923 households over the course of data collection.

####  *Developing Standard Refusal Conversion Procedures*

If, at the initial contact, a respondent (or anyone in the household)
refused to participate in the survey, the CATI scheduler moved that
person's case to the refusal work class and put it on hold for a period
of time. After this "cooling off" period, the CATI scheduler assigned
the case to a refusal conversion interviewer. The cases of respondents
who refused again, after one conversion attempt, were held for another
cooling off period and released again. We rereleased some cases up to
two times. We did not attempt to recontact hostile refusers. Overall,
our rate of refusal conversion was 33.0 percent for screener cases, 80.9
percent for RDD Sample extended interviews, and 82.1 percent for List
Sample interviews.

####  *Mailing Letters to Refusals and Hard-to-Locate Veterans*

About halfway through data collection, we began mailing letters to cases
that had refused twice and cases that were hard to locate. The refusal
conversion mailout was conducted weekly from late July through late
October, 2001. We mailed a letter in a business size envelope with first
class postage, addressed "To the Family At" to RDD Sample cases with
whom we had been unable to make contact, or that had refused during the
household screening interview (see Exhibit 5-9). We sent a letter by
overnight delivery to List Sample and RDD Sample cases that we had been
unable to recontact for an extended interview, or that had refused to
complete an extended interview (see Exhibit 5-10 and Exhibit 5-11). Our
address matching vendor provided addresses for 70 percent of the RDD
Sample. We did not have reliable names for these cases, though, so we
addressed the letters "To the Veteran At." We personalized the List
Sample letters. Along with the letters to extended interview refusers
(or hard-to-locate cases), we enclosed the same letter from the
Secretary of the VA that we included in the advance mailing to List
Sample members. Between 2 and 5 days after we mailed the letters, we
made available for calling these

Exhibit 5-8. Privacy Act statement

> September 5, 2001

«TRC_ID»

«F_NAME» «L_NAME»«SUFFIX»

«ADDR1A»

«ADDR1B»

«CITY1», «STATE1» «ZIP1»

Dear «F_NAME» «L_NAME»«SUFFIX»:

Recently you were contacted by a Westat interviewer as part of a study
being conducted for the Department of Veterans Affairs. At that time you
stated you would like to receive a copy of the Privacy Act Statement
that applies to this study. The text of this statement is provided
below.

**Privacy Act Statement** -- ***National Survey of Veterans 2001***

[Authority]{.underline}: 10 U.S.C 2358: E.O.9397

**[PRINCIPAL PURPOSE]{.underline}:** This survey collects [veteran
current]{.underline} demographic and socioeconomic characteristics,
veteran health status, medical services usage and needs, health
insurance coverage, and veteran knowledge about and use of various VA
programs. Social Security Numbers are requested and will be used for two
purposes. First, it will be used to ensure that each respondent is
interviewed only once. Second, it will be used to obtain additional
information from VA files and other sources to better understand veteran
needs.

**[ROUTINE USES:]{.underline}** None.

**[DISCLOSURE:]{.underline}** Voluntary. Maximum participation is
encouraged so that data will be complete and representative, but there
are no adverse consequences for individuals who decline to participate.

I hope that the information provided here answers any questions you have
regarding this study. Should you wish to obtain additional information
you may contact:

Department of Veterans Affairs

Office of Policy and Planning

Attn: Ms. Susan Krumhaus 008A Phone: (202) 273-5108

810 Vermont Avenue, NW

Washington, D.C. 20420

If you need general information on programs and benefits offered by the
Department of Veterans Affairs, you may call their toll-free hotline:
**(800) 827-1000**.

Thank you for your assistance in this important study.

> Sincerely,

![](media/image44.png){width="1.698611111111111in"
height="0.6583333333333333in"}

> John C. Helmick, Jr.
>
> Senior Study Director

Exhibit 5-9. Screener refusal letter (RDD Sample)

WESTAT

National Survey of Veterans

1650 Research Blvd. • Rockville, MD 20850-3129 • 301 251-1500 • FAX 301
294-2040

> «DATESENT»

ID #«MAILID»

To the Family At

«ADDRESS»

«CITY», «STATE» «ZIP»

Dear Respondent:

**We need your help.** Recently, one of our telephone interviewers
called and asked you or someone in your household to take part in a
survey about veterans of the United States military. At this time, we
have not finished an interview with you. This survey is sponsored by The
Department of Veterans Affairs (VA). It will help the VA to evaluate
their current policies, programs and services.

Your answers to this survey will help us determine whether there are any
veterans in the household. We can't talk to every person in the country.
Instead, we scientifically selected a few people to represent the nation
as a whole. This means no other person can take your place. Your
individual participation will have a significant impact on the survey's
outcome. Thus, even if there are no veterans in your household, we would
still like to speak with you briefly.

I want to assure you that the information you provide is protected under
the Privacy Act and section 5701 of Title 38 of the U.S. Code.

We would be happy to speak with you at a time that is convenient for
you. If you would like to begin an interview or schedule an appointment,
please call Westat toll-free at 1-888-258-2194. Ask to speak with the
Veterans Survey Manager.

Thank you. We greatly appreciate your help.

> Sincerely,
>
> ![](media/image45.png)
>
> John C. Helmick
>
> Project Director

Exhibit 5-10. Extended refusal letter (List Sample)

WESTAT

National Survey of Veterans

1650 Research Blvd. • Rockville, MD 20850-3129 • 301 251-1500 • FAX 301
294-2040

> «DATESENT»

ID #«MAILID»

«NAME»

«ADDRESS»

«CITY», «STATE» «ZIP»

Dear «LOWERNME»:

**We need your help.** Recently, one of our telephone interviewers
called and asked you to take part in the National Survey of Veterans. At
this time, we have not finished an interview with you. This survey is
sponsored by the Department of Veterans Affairs (VA). It will provide
valuable information for planning benefits and services for all
veterans.

While your participation is voluntary, you are one of only a few
thousand randomly selected veterans whose answers will represent all 25
million veterans. Your individual participation will have a significant
impact on the survey's outcome. We cannot interview another veteran in
your place. Your answers, and those of other participating veterans,
will give the VA an up-to-date picture of the whole U.S. veteran
population.

I want to assure you that the information you provide is protected under
the Privacy Act and section 5701 of Title 38 of the U.S. Code. The VA
will use the information you provide to evaluate current VA policies,
programs and services for veterans and in deciding how to help veterans
in the future.

We would be happy to arrange an interview at a time that is convenient
for you. If you would like to begin an interview or schedule an
appointment, please call Westat toll-free at 1-888-258-2194. Ask to
speak with the Veterans Survey Manager.

Thank you. We greatly appreciate your help.

> Sincerely,
>
> ![](media/image46.png)
>
> John C. Helmick
>
> Project Director

Exhibit 5-11. Extended refusal letter (RDD Sample)

WESTAT

National Survey of Veterans

1650 Research Blvd. • Rockville, MD 20850-3129 • 301 251-1500 • FAX 301
294-2040

«DATESENT»

ID #«MAILID»

To the Veteran At

«ADDRESS»

«CITY», «STATE» «ZIP»

Dear Veteran:

**We need your help.** Recently, one of our telephone interviewers
called and asked you to take part in the National Survey of Veterans. At
this time, we have not finished an interview with you. This survey is
sponsored by the Department of Veterans Affairs (VA). It will provide
valuable information for planning benefits and services for all
veterans.

While your participation is voluntary, you are one of only a few
thousand randomly selected veterans whose answers will represent all 25
million veterans. Your individual participation will have a significant
impact on the survey's outcome. We cannot interview another veteran in
your place. Your answers, and those of other participating veterans,
will give the VA an up-to-date picture of the whole U.S. veteran
population.

I want to assure you that the information you provide is protected under
the Privacy Act and section 5701 of Title 38 of the U.S. Code. The VA
will use the information you provide to evaluate current VA policies,
programs and services for veterans and in deciding how to help veterans
in the future.

We would be happy to arrange an interview at a time that is convenient
for you. If you would like to begin an interview or schedule an
appointment, please call Westat toll-free at 1-888-258-2194. Ask to
speak with the Veterans Survey Manager.

Thank you. We greatly appreciate your help.

> Sincerely,
>
> ![](media/image47.png)
>
> John C. Helmick
>
> Project Director

screener and extended cases. RDD Sample cases for which we were unable
to obtain an address received the telephone followup without a mailout.
The release of the refusal conversion mailout cases occasionally
overlapped with the release of new sample. This created a huge volume of
work such that the refusal cases sometimes did not get called until up
to several weeks after they received the letters. Table 5-3 shows the
number of RDD screener, RDD extended, and List Sample extended interview
cases to which we mailed letters, and the percentage of those cases for
which we were able to complete an interview.

Table 5-3. Results of mailout to refusal and hard-to-locate cases

  --------------------------- --------------------- ----------------------
  Interview Type              Number of cases to\   Percent of mailed
                              which a letter was    cases\
                              mailed                with a completed
                                                    interview

  Screener interview          28,482                2.2

  RDD Sample extended         2,746                 20.6
  interview                                         

  List Sample extended        1,571                 28.4
  interview                                         
  --------------------------- --------------------- ----------------------

####  *Matching Deceased Veterans to the SSA Date-of-Death File*

During the course of data collection, we noticed that a large number of
List Sample veterans were reported as deceased either on the advance
letter update form or when we attempted to contact them over the
telephone. To ensure that we had the most accurate information possible
about our List Sample cases, toward the end of data collection we
matched all List Sample cases against the Social Security Administration
(SSA) Date-of-Death File. Through this effort, we were able to identify
317 cases of previously unknown status as deceased. We also confirmed
that 202 cases we had already identified as deceased were, indeed,
deceased.

####  *Retrieving RDD Screener Data*

For 24 RDD Sample cases, we discovered that some screened interview
information required to determine whether someone in the household was
eligible for an interview was missing. We called these households again
to collect the missing information. We retrieved the missing data in all
24 cases, identifying 13 of them as eligible for the study. As a result
of this effort, we completed seven additional interviews. The other six
eligible veterans refused to participate.

####  *Scheduling Additional Releases of Cases*

The CATI scheduler followed certain rules for determining when to close
cases with which we did not make contact. At the same time, we wanted to
ensure that we made every attempt to reach respondents who might have
been out of town or otherwise unavailable during our initial contact
attempts. Therefore, we programmed the CATI scheduler to automatically
release such noncontact cases after a designated "resting" period. Doing
so improved our chances of making contact with these difficult-to-reach
cases.

####  *Conducting End-of-Study Measures*

Toward the end of data collection, we used three additional methods to
increase the number of household contacts. First, on one day during each
of the last 4 weeks of data collection, interviewers left a message
every time they reached an answering machine. Second, 2 weeks before the
end of data collection, we increased the number of times we called
outstanding List Sample cases, of which there were 498. To do this, we
removed all outstanding (noncompleted) cases from the CATI scheduler. We
continued to try to make contact using a paper call record. Doing so
permitted us to identify the cases with which we had had little or no
contact, attempt to call them at times different from the previous
calls, and call multiple times during one day. Third, on the final
Saturday of data collection, we asked interviewers to let telephones
ring longer than usual before hanging up.

6\. SAMPLE WEIGHTING

After the data collection and editing phases of the 2001 National Survey
of Veterans (NSV 2001) were completed, we constructed the sampling
weights for the data collected from the sampled veterans so that the
responses could be properly expanded to represent the entire veteran
population. The weights were the result of calculations involving
several factors, including original selection probabilities, adjustment
for nonresponse, households with multiple residential telephones, and
benchmarking to veteran population counts from external sources. We
produced a separate set of weights for the List and the RDD Samples and
then combined them to produce the composite weights for use with the
combined List and RDD Samples.

Our objectives in carrying out the sample weighting in the NSV 2001 were
to:

-   Enable the production of tabulations that provide estimates of the
    number of veterans in the population for the various categories
    selected;

-   Compensate for disproportionate sampling of various subgroups in the
    List Sample;

-   Compensate for the higher chance of selection of households with
    multiple residential telephone lines;

-   Reduce biases arising from the fact that nonrespondents may be
    different from those who participated;

-   Compensate, to the extent possible, for noncoverage in the sample
    due to inadequacies in the sampling frame or other reasons for
    noncoverage, such as veterans in households without telephones; and

-   Reduce variances of the estimates by using auxiliary variables that
    are highly correlated with the study variables.

We also constructed a set of replicate weights for each respondent
veteran and appended them to each record for use in estimating
variances. This chapter describes the calculation of the full sample
composite weights and replicate composite weights. We start with a
description of the List and RDD Sample weights because the two sets of
weights were constructed independently.

# 6.1 List Sample Weights

The List Sample weights are used to produce estimates from the List
Sample that represent the population of veterans who are on the list
frame. As described in Chapter 3, the list frame was constructed from
the VHA Healthcare enrollment file and the VBA Compensation and Pension
(C&P) file. The steps involved in constructing the List Sample weights
are the calculation of a base weight, poststratification adjustment to
known list frame population counts, and adjustments to compensate for
veterans with unknown eligibility, and for nonresponse. These steps are
described in detail below.

####  Calculation of List Sample Base Weights

The base weight for each veteran is equal to the reciprocal of his/her
probability of selection. The probability of selection of a veteran is
the sampling rate for the corresponding sampling stratum. If
![](media/image48.wmf) out of ![](media/image49.wmf) veterans are
selected from a stratum denoted by *h,* then the base weight (or design
weight) assigned to the veterans sampled from the stratum was obtained
as

![](media/image50.wmf) (6-1)

Properly weighted estimates using the base weights (as given above)
would be unbiased if the eligibility status of every sampled veteran
could be determined and every eligible sampled veteran agreed to
participate in the survey. However, the eligibility status of each and
every sampled veteran could not be determined (for example, some sampled
veterans could not be located). Moreover, nonresponse is always present
in any survey operation, even when participation is not voluntary. Thus,
weight adjustment was necessary to minimize the potential biases due to
unknown eligibility and nonresponse. In order to improve the reliability
of the estimates we also applied a poststratification adjustment.
Normally, the poststratification adjustment is applied after applying
the nonresponse adjustment, but we carried this out before the
nonresponse adjustment because determining the eligibility status of
every veteran on the list frame would not have been feasible.

####  Poststratification Adjustment

Poststratification is a popular estimation procedure in which the base
weights are adjusted so that the sums of the adjusted weights are equal
to known population totals for certain subgroups of the population. We
defined the poststrata to be the cross classification of three age
categories (under 50, 50-64, over 64), gender (male, female), and census
regions (Northeast, Midwest, South, and West), which resulted in 24
poststrata.

Let ![](media/image51.wmf) denote the number of veterans on the list
frame that belong to the poststratum denoted by *g*
![](media/image52.wmf) as obtained from the list frame, and let
![](media/image53.wmf) be the corresponding estimate obtained by using
the List Sample base weights. Then the ratio ![](media/image54.wmf) is
used as an adjustment to define the poststratified weight
![](media/image55.wmf) as

![](media/image56.wmf) (6-2)

The superscript (*pst*) denotes that it is a poststratified weight.
Because a veteran denoted by (*hi*) can belong to one and only one of
the poststrata, the poststratified weights are uniquely defined. The
advantage of poststratified weighting is that the reliability of the
survey estimates is improved. The minimum sample size for
poststratification cells was set at 30 veterans. For 2 out of the 24
poststrata, the sample sizes were fewer than 30 veterans. The two
deficient cells were female veterans in the age group 50-64 in census
regions "Northeast" and "Midwest." Their sample sizes were equal to 16
and 29, respectively. We collapsed these two cells in order to achieve
the sample size of more than 30 in the collapsed poststratum. Thus, the
poststratified weights were computed using the auxiliary veteran
population counts from the list frame for 23 poststrata.

For the sake of simplicity we will denote by ![](media/image57.wmf) the
poststratified weight of the ![](media/image58.wmf) List Sample veteran.
These weights are the input weights for adjustments for unknown
eligibility and nonresponse.

####  Adjustments for Unknown Eligibility and Nonresponse

The List Sample cases can be divided into respondents and
nonrespondents. Further, the respondents can be either eligible or
ineligible (out of scope) for the survey. The eligibility of the
nonrespondent veterans could not always be determined. For example, a
sampled veteran who could not be located could have been deceased and
hence ineligible for the survey. Or, an eligible veteran might have
moved and new contact information (address and telephone number) might
not be obtainable. Therefore, the nonrespondents were classified into
two categories: (1) eligible nonrespondents and (2) nonrespondents with
unknown eligibility. In order to apply the adjustments for unknown
eligibility and nonresponse, the List Sample cases were grouped into
four response status categories (Figure 6-1):

Category 1: **Eligible Respondents.** This group consists of all
eligible sampled veterans who participated in the survey, namely those
who provided usable survey data. The category includes the final result
codes CE and CX.

Category 2: **Ineligible or Out of Scope.** This group consists of all
sampled veterans who were ineligible or out of scope for the survey,
such as veterans who had moved abroad and were therefore ineligible for
the survey. The information that was obtained was sufficient to
determine that these veterans were indeed ineligible for the survey.

Category 3: **Eligible Nonrespondents.** This group consists of all
eligible sampled veterans who did not provide usable survey data. The
information that could be obtained was sufficient to ascertain that the
veteran was eligible for the survey.

Category 4: **Eligibility Unknown.** This group consists of all sampled
veterans whose eligibility could not be determined. For example, sampled
veterans who could not be located were placed in this category.

We used the final List Sample extended interview result codes
*(MAINRSLT)* and the variable *"MCURSECT"* to assign the sampled
veterans to one of the four response categories defined above. The
groupings of the extended interview result codes and the *"MCURSECT"*
values that define the above response categories are given in Appendix
E. A list of final extended interview result codes are included in
Appendix H. For incomplete cases, the variable *"MCURSECT"* indicates
the point at which the interview broke off and that the interview could
not be completed after that. We should note that the eligibility could
not be determined when the Military Background module of the extended
interview was not completed. Also, we interpreted cases with result code
*"IA"* (not a veteran) as "hidden" refusals and assigned them to the
category "eligible nonrespondents" irrespective of the *"MCURSECT"*
value.

![](media/image59.wmf)

Figure 6-1. Categories of response status

The nonresponse adjustment was applied in two steps. In the first step
the poststratified weights of the veterans with unknown eligibility
(Category 4) were distributed proportionally over those with known
eligibility (Categories 1, 2, and 3). As stated earlier, let
![](media/image60.wmf) denote the poststratified weight of the
![](media/image61.wmf) veteran sampled from the list frame. Then the
adjustment for unknown eligibility was obtained as

![](media/image62.wmf) (6-3)

where ![](media/image63.wmf) represents veterans who were survey
respondents (Category 1), ![](media/image64.wmf) represents out-of-scope
or ineligible veterans (Category 2), *N* represents eligible
nonrespondents (Category 3), and *U* represents the sampled veterans
whose eligibility could not be determined (Category 4). The adjustment
factor ![](media/image65.wmf) to account for unknown eligibility was
applied to the poststratified weights of the eligible respondents
(Category 1), out-of-scope or ineligible veterans (Category 2), and
eligible nonrespondents (Category 3). Thus, the List Sample weight
![](media/image66.wmf) adjusted for unknown eligibility was computed as

![](media/image67.wmf) if the ![](media/image68.wmf) veteran belongs to
response Category 1, 2 or 3. (6-4)

The weights of the veterans with unknown eligibility (Category 4) were
set to zero.

The adjustment for unknown eligibility was applied within homogeneous
adjustment classes. These adjustment classes were determined with
*CHAID* (Chi-square Hierarchical Automatic Interaction Detector)
software described in Appendix F.

In the second step, we calculated an adjustment factor to account for
the eligible nonrespondent veterans. The extended list interview
nonresponse adjustment factor was calculated as the ratio of the sum of
the weights (adjusted for unknown eligibility) for eligible respondents
and eligible nonrespondents to the sum of the weights for only the
eligible respondents. Thus, we calculated the nonresponse adjustment
factor ![](media/image69.wmf) to be the ratio of the sums as

![](media/image70.wmf) (6-5)

where ![](media/image71.wmf) is the weight obtained after applying the
adjustment for unknown eligibility, ![](media/image72.wmf) represents
eligible respondents (Category 1), and ![](media/image73.wmf) represents
eligible nonrespondents (Category 3). The adjustment factor
![](media/image74.wmf) is applied only to the weights of the eligible
respondents (Category 1) in the sample. That is, the
nonresponse-adjusted weight ![](media/image75.wmf) is computed as

![](media/image76.wmf) if the ![](media/image77.wmf) sampled veteran is
a respondent (Category 1). (6-6)

We applied the nonresponse adjustment, ![](media/image78.wmf), within
homogeneous nonresponse adjustment classes, which were also defined
using *CHAID* software. The final List Sample weight for each eligible
respondent was computed by multiplying the weight ![](media/image79.wmf)
by the appropriate nonresponse adjustment factor as defined above. The
final List Sample weight for the eligible nonrespondent veterans was set
to zero. The final List Sample weight of the out-of-scope/ineligible
veterans is the weight obtained after applying the adjustment factor for
unknown eligibility. The weights for the out-of-scope/ineligible
veterans could be used to estimate the ineligibility rate of the list
frame that we used to select the List Sample.

# 6.2 RDD Sample Weights

The calculation of the RDD Sample weights consisted of five main steps.
The steps included computing the base weight and various adjustments at
the screener interview level and the extended interview level. In
summary, we:

-   Computed base weight as the inverse of the probability of selection
    of the telephone number associated with the household;

-   Applied an adjustment to account for household level nonresponse
    during screening;

-   Applied an adjustment for multiple telephone lines as the reciprocal
    of the number of "regular residential" telephone numbers used by the
    household (excluding telephone numbers used only for business
    purposes, fax machines, cellular phones, pagers, or mobile phones);

-   Applied an adjustment to correct for the nonresponse to the extended
    interview; and

-   Benchmarked to known veteran population counts from the Census 2000
    Supplementary Survey (C2SS) that the U.S. Bureau of the Census
    conducted.

The final RDD Sample weights were obtained as the product of the base
weight and the various adjustments applied to the base weights. The
steps involved in computing these weights are described in detail below.

####  RDD Sample Base Weights

As described in Chapter 3, the RDD Sample was selected with the
list-assisted RDD sampling methodology except for the Puerto Rico RDD
Sample, for which an RDD sample of telephone numbers was selected from
all possible telephone numbers for Puerto Rico. The base weights for the
two RDD Samples were defined accordingly.

####  *List-assisted RDD Sample Base Weights*

The base weight is defined as the reciprocal of the probability of
selection. With the list-assisted RDD methodology, the telephone numbers
were selected with equal probabilities of selection. We used a
systematic sampling scheme to select telephone numbers, and the
probability of selecting a telephone number when *n* telephone numbers
from a pool of *N* numbers is selected is given by *f = n/N*. Because
the national RDD Sample was selected from two RDD frames constructed at
two different times (see Chapter 3) the selection probabilities were
computed according to whether a telephone number was eligible for
selection from both frames or from only one of the frames. Let
![](media/image80.wmf) and ![](media/image81.wmf) denote the RDD frames
constructed at the two time periods, and ![](media/image82.wmf) and
![](media/image83.wmf) be their corresponding sizes. A random sample of
![](media/image84.wmf) (=240,000) telephone numbers was selected from
the frame ![](media/image85.wmf) and a random sample of
![](media/image86.wmf) (=60,000) telephone numbers was selected from the
frame ![](media/image87.wmf). The selection probabilities of the sampled
telephone numbers were computed as follows.

![](media/image88.wmf) (6-7)

where *t* denotes a sampled telephone number. The base weight of a
telephone number selected from the RDD frames is given by the reciprocal
of the corresponding probability of selection.

####  *Puerto Rico Sample Base Weights*

The Puerto Rico RDD Sample was a pure RDD sample due to the fact that
information was not available on the telephones to construct the
sampling frame for list-assisted RDD methodology. The base weight was
defined to be the inverse of the selection probability.

####  RDD Sample Weight Adjustments

RDD Sample weight adjustments include weight adjustments for the
national (list-assisted) RDD Sample and the Puerto Rico RDD Sample.

####  *List-assisted RDD Sample Weight Adjustments*

List-assisted RDD Sample weight adjustments were applied as screener
interview nonresponse adjustment, adjustment for multiple telephone
lines, and an adjustment for nonresponse at the extended interview.

**Screener Nonresponse Adjustment.** The base weights were adjusted to
account for the households (telephones) with unknown eligibility during
the screening interview. We defined the four categories listed below and
assigned sampled telephone numbers to each based on the final screener
result (*SCRNRSLT*) codes as given in Appendix E.

Category 1: **Eligible Respondents.** This category consists of all
sample households that completed the screening questionnaire and
contained at least one veteran eligible for the extended interview. The
category includes the RDD screener final result codes CO or household
selection flag (HSF) equal to 1 (YES).

Category 2: **Ineligible or Out of Scope.** This category consists of
all sample households (telephones) that were ineligible or out of scope
for the survey. For example, these included households with no veterans
or telephone numbers that were business numbers.

Category 3: **Eligible Nonrespondents.** Although we defined an
"eligible nonrespondents" category, no cases were assigned to it because
once someone in the household responds to the screening questionnaire,
the persons enumerated in that household can be categorized as either
eligible respondents or ineligible/out of scope. Otherwise, the
household is assigned to the category "Eligibility Unknown."

Category 4: **Eligibility Unknown.** This category consists of all
sample telephones for which sufficient information could not be
collected to determine whether or not there was a veteran in the
household.

The assignment of sampled households (telephones) to the three response
categories (categories 1, 2 and 4) was based on the final screener
result codes (*SCRNRSLT*) and household selection flag (HSF) as given in
Appendix E. A list of final screener result codes is included in
Appendix H.

The base weights corresponding to the households (telephones) with
unknown eligibility (Category 4) were distributed proportionally over
those with known eligibility (Categories 1 and 2). To carry out the
adjustment for unknown eligibility, the telephones with unknown
eligibility were divided into two sub-categories: (1) those that we
could determine were residential and (2) those for which we could not
make a residential determination.

The adjustment for unknown eligibility was then applied in two separate
steps. In the first step, we adjusted for those telephones whose type --
residential, business, or nonworking -- could not be determined. The
weight adjustment was applied within homogeneous adjustment classes that
were determined through the *CHAID* analysis.

In the second step, nonworking and business telephone numbers were
removed and the weights were adjusted to account for the residential
telephone numbers for which the eligibility for the NSV 2001 could not
be determined. The adjustment for unknown eligibility in the second step
was computed as the ratio of the sum of the weights adjusted in the
first step of all residential sample cases (both with known and unknown
eligibility) to those with known eligibility. It should be noted that
the nonworking and business telephone numbers had been eliminated at
this stage. The weights of those with known eligibility were adjusted by
multiplying with the adjustment factor for the second step of unknown
eligibility, and the weights of those with unknown eligibility were set
to zero. The adjustment for the second step of unknown eligibility was
also applied within homogeneous adjustment classes defined using the
*CHAID* software.

**Adjustment for Multiple Residential Lines.** If every household had
exactly one residential telephone number, then the weight for a
household would be the same as the base weight of the corresponding
telephone number. The adjustment for multiple residential telephone
households prevents households with two or more residential telephone
numbers from receiving a weight that is too large by reflecting their
increased probability of selection. In theory, the household weight
would be obtained by dividing the base weight by the number of
residential telephone lines in the household. We assigned an adjustment
factor of ½ to the households with more than one residential telephone
number because the number of households with more than two residential
telephone numbers would be small. A weighting factor of unity was
assigned to households reporting only one telephone number in the
household, and an adjustment factor of ½ was assigned to households with
more than one residential telephone number.

**RDD Extended Interview Nonresponse Adjustment.** The RDD Sample
required administration of both a household screening questionnaire and
the extended NSV 2001 questionnaire, and included the possibility of
identifying multiple veterans in a single household. Because the
screener survey interview screened for the households with potential
veterans, a small fraction of persons who were screened in were not
actually eligible for the NSV 2001. Once the extended interview began,
it was still necessary to establish with certainty that the selected
person was indeed a veteran, so further screening took place at the
beginning of the extended interview in the Military Background module.
If the responses to the set of eligibility questions during the extended
interview indicated that the person was not an eligible veteran, the
interview was terminated. Moreover, for some cases that were screened
in, no information could be collected from the extended interview to
ascertain their eligibility (e.g., the potential veteran could not be
contacted for the extended interview). Thus, the screened-in sample
contained cases with unknown eligibility as well as eligible and
ineligible cases. Further, the eligible cases contained respondents and
nonrespondents. Therefore, the screened-in RDD Sample cases were grouped
into the same four categories as the List Sample cases.

Category 1: **Eligible Respondents.** This group consists of all
eligible sample veterans who participated in the survey, namely those
who provided usable survey data. The category includes the final result
codes CE and CX.

Category 2: **Ineligible or out of scope.** This group consists of all
sample cases that were determined to be ineligible or out of scope for
the survey, such as a screened-in person who was not a veteran and hence
was ineligible for the survey.

Category 3: **Eligible Nonrespondents.** This group consists of all
eligible sample veterans who did not provide usable survey data. The
information that could be obtained was sufficient to ascertain that the
veteran was eligible for the survey.

Category 4: **Eligibility Unknown.** This group consists of all sample
cases whose eligibility could not be determined. For example, sample
persons who could not be contacted were placed in this category.

The screened-in sample cases were assigned to the four response
categories on the basis of final extended interview result codes
(*MAINRSLT*) and the variable *"MCURSECT."* The groupings of the
extended result codes, along with the *"MCURSECT"* values corresponding
to the four response categories, are given in Appendix E. These
categories are very similar to those for the List Sample extended
interviews. We should note that the extended result code *"IA"* (not a
veteran) for the List Sample cases was interpreted as a "hidden" refusal
and hence was assigned to the category "eligible nonrespondents." The
RDD Sample cases with the result code *"IA"* were assigned to the
"ineligible" category because the eligibility status for RDD Sample
cases was determined during the extended interview.

The weights of the cases with unknown eligibility (Category 4) were
proportionally distributed over the other 3 categories (Categories 1, 2,
and 3). These adjustment factors were calculated separately for
homogeneous classes defined with *CHAID* analysis.

The next step in the RDD Sample weighting was the extended interview
nonresponse adjustment. The RDD extended interview nonresponse
adjustment factor was calculated as the ratio of the sum of weights for
eligible RDD extended interview respondents and eligible RDD extended
interview nonrespondents to the sum of the weights for only the eligible
RDD extended interview respondents. Separate nonresponse adjustment
factors were computed within homogeneous nonresponse adjustment cells.
The nonresponse adjustment cells were determined with the *CHAID*
software.

####  *Puerto Rico Sample Weight Adjustments*

We screened 96 households with potentially 102 veterans for which
extended interviews were attempted. We completed only 51 extended
interviews from the Puerto Rico RDD Sample. The nonresponse adjustment
factors for the screener interview and extended interview were computed
similarly to those for the national RDD Sample except that the screener
nonresponse adjustment was computed separately for two age groups (under
60, over 59) and a single nonresponse adjustment was computed for the
extended interviews. This was due to the small sample size for the
Puerto Rico RDD Sample.

After applying the screener interview and extended interview nonresponse
adjustments, the national (list-assisted) RDD and the Puerto Rico RDD
Samples were combined into one RDD Sample. The base weights adjusted for
nonresponse were further adjusted in a raking procedure, discussed in a
later section. The raked weights were the final RDD Sample weights that
were used to compute the composite weights for the combined List and RDD
Samples.

####  Comparison of RDD Estimates with VA Population Model Estimates

As a check, we compared the RDD Sample estimate of number of veterans
based on the weights before raking with the estimate from the Vetpop
2000 model[^6], VA population projection model. The NSV 2001 target
population includes only noninstitutionalized veterans living in the
U.S. The reference period for the NSV 2001 is the year 2000[^7]. The VA
population model estimates are also for the year 2000 and these are
based on the 1990 Census. These estimates are derived by incorporating
survival rates and information on veterans leaving military service. The
VA population model estimate for the entire veteran population is
25,372,000 veterans, whereas the estimate from the RDD Sample is
23,924,947 veterans, which is 5.7 percent lower than the VA population
model estimate. The difference of 5.7 percent can be attributed to the
combination of the differences from exclusion of the institutionalized
veterans and RDD undercoverage of nontelephone households and households
with unlisted telephone numbers belonging to "zero-listed telephone
banks."

The portion of undercoverage due to nontelephone households and
households with unlisted numbers belonging to "zero-listed telephone
banks" was addressed with the raking procedure, described in the next
section. The control total of veteran population for the raking
procedure was 25,196,036 veterans. Thus, the estimated undercoverage due
to nontelephone households and households with unlisted telephone
numbers belonging to "zero-listed telephone banks" would be only about
5.0 percent. After correcting for the undercoverage from these two
sources, the difference between the NSV 2001 and the Vetpop 2000
estimates is less than one percent, which is from institutionalized
veterans and veterans living abroad.

####  Raking Ratio Estimation/Undercoverage Adjustment

The raking ratio estimation procedure is based on an iterative
proportional fitting procedure developed by Deming and Stephan (1940),
and involves simultaneous ratio adjustments to two or more marginal
distributions of the population counts. Raking was proposed by Deming
and Stephan (1940) as a way to ensure consistency between complete
counts and sample data from the 1940 U.S. Census of population. The
methodology is referred to as raking ratio estimation because weights
are raked using ratio adjustments based on the known marginal population
totals. Typically, raking is used in situations where the interior cell
counts of cross-tabulation are either unknown or sample sizes in some
cells are too small for efficient estimation. The purpose of the raking
procedure in this survey is to improve the reliability of the survey
estimates, and to correct for the bias due to missed households, namely,
households without telephones and households with unlisted telephone
numbers belonging to "zero-listed telephone banks." As described in
Chapter 3, households without telephones and households with unlisted
telephone numbers belonging to the "zero-listed telephone banks" are not
included in the list-assisted RDD sampling frame.

The raking procedure is carried out in a sequence of adjustments. First,
the base weights are adjusted to one marginal distribution and then to
the second marginal distribution, and so on. One sequence of adjustments
to the marginal distributions is known as a cycle or iteration. The
procedure is repeated until convergence is achieved. The criteria for
convergence can be specified either as maximum number of iterations or
absolute difference (or relative absolute difference) from the known
marginal population totals.

We used a two-dimensional raking procedure for the RDD Sample. The
computational details of the two-dimensional raking procedure are given
in Appendix G. We formed the two raking dimensions from the cross
classification of veterans according to the demographic/education/region
characteristics of the veterans. These characteristics were also
obtained during the screening interview. The first dimension was formed
from the cross classification of three age categories (under 50, 50-64,
over 64) with four education levels (no high school diploma, high school
diploma, some college, bachelor's degree or higher) and four race
categories (Hispanic, Black, Other, and White), resulting in 48 cells.
The second dimension was formed from the cross classification of gender
(male, female) and the four census regions (Northeast, Midwest, South,
and West), resulting in 8 cells. By using a set of cross classified
variables for each raking dimension, the internal correlation structure
of the data could be better preserved. The sample sizes for the race
categories "Hispanics," "African American," and "Other" in the age group
under 50, and education "no high school diploma" were 21, 15, and 17,
respectively. These three cells in the first raking dimension were
collapsed to achieve sufficient cell sample size. Thus, the number of
cells for the first raking dimension was reduced to 46 after collapsing
the three cells with deficient sample sizes. The sample sizes were more
than 25 for all cells used for the raking.

We used the Census 2000 Supplementary Sample (C2SS) data from the U.S.
Bureau of the Census to define the control totals for the raking
procedure. We also included the Puerto Rico RDD Sample in the raking
procedure. Because the C2SS did not include Puerto Rico in the survey
target population, we estimated the Puerto Rico veteran population
counts for the year 2000 from the Census 1990 population counts based on
a model. The methodology for the veteran population counts to be used as
control totals for the raking procedure is discussed briefly in the next
section.

We applied the convergence criteria in terms of percent absolute
relative difference, which was specified to be no more than 0.01 percent
for all marginal population counts. The raking procedure converged in 8
iterations.

The above variables were chosen as the raking variables due to
significant differences in the telephone coverage by categories of these
variables, and hence maximum bias reduction would be achieved. The
sample sizes at the adjustment cell level would become very small if we
had used too many variables in the cross classification to define
marginal distributions for raking.

####  Veteran Population Counts for the Raking Procedure

The independent estimates of veteran population counts for the raking
procedure were obtained from the Census 2000 Supplementary Survey
(C2SS). The C2SS sample does not cover Puerto Rico, and we used the 1990
Census data to obtain model-based estimates of the Puerto Rico veteran
population counts for the year 2000. The methodology of the model-based
estimates for Puerto Rico is discussed later in this section. For the
purpose of the raking procedure Puerto Rico data were combined with the
census region "South."

####  *Estimates from the Census 2000 Supplementary Survey (C2SS)*

The U.S. Bureau of the Census conducted the Census 2000 Supplementary
Survey (C2SS). The survey covers the 50 states and the District of
Columbia. The sample for the C2SS used a two stage stratified design
with a sample of approximately 890,000 housing units designed to measure
socioeconomic and demographic characteristics of housing units and their
occupants. The C2SS sample of housing units was selected from the Master
Address File (MAF). The MAF was created by combining the 1990 Census
Control file, the Delivery Sequence File of the United States Postal
Service (USPS), and addresses listed for the Census 2000. The first
stage sampling involved dividing the United States into primary sampling
units (PSUs) and grouping these PSUs into homogeneous strata. The C2SS
design employed 1,925 PSUs. The strata were constructed so that they are
as homogeneous as possible with respect to social and economic
characteristics that are considered important by C2SS data users. A pair
of PSUs was selected from each stratum with probability proportional to
size (PPS) sampling. In the second stage of sampling, a sample of
housing units within the sampled PSUs was drawn using a systematic
sampling procedure.

The data were collected from more than 700,000 housing units. Our
assumption was that 1 in 4 households contains a veteran and hence, the
estimates of veteran population counts from the C2SS data will be based
on approximately 175,000 interviewed veterans. The Census 2000
Supplementary Survey universe is limited to the household population and
excludes the population living in institutions, college dormitories, and
other group quarters. Because the NSV 2001 also excludes the
institutionalized veteran population and veterans living abroad, the
estimated veteran population counts from the C2SS could be used to
benchmark the NSV 2001 estimates.

####  *Model-based Estimates for Puerto Rico*

The C2SS sample does not cover Puerto Rico, and external data for Puerto
Rico for the raking variables is not available from any other source. We
used the 1990 Census data to obtain the distribution of the Puerto Rico
veteran population by the variables used for raking. We made the
assumption that the distribution of the Puerto Rico veteran population
by the raking variables has not changed between 1990 and 2000. Thus, we
could use the total Puerto Rico veteran population in 2000 and the 1990
Census distribution to obtain the veteran population counts for the
cells defined for the raking procedure. We used the Puerto Rico total
veteran population for 2000 as derived from the veteran population model
developed by VA (Vetpop 2000). According to the Veteran Population
model, the Puerto Rico veteran population for 2000 was 142,680 veterans.
We used these model-based estimates of Puerto Rico veteran population
counts for 2000 to adjust the veteran population control totals obtained
from C2SS so that the raking procedure could be used with the RDD
sample, including the Puerto Rico RDD Sample. The Puerto Rico data were
assigned to the census region "South" for raking.

# 6.3 Composite Weights

Integration of samples from multiple frames into a single micro-data
file with a single weight requires, at a minimum, the ability to tell
which of the veterans had more than one chance of selection. This is
enough to create unbiased weights. The Social Security numbers (SSNs) of
all the veterans on the list frame were known. To identify the RDD
Sample veterans on the list frame, we needed to obtain their SSNs during
data collection so that the overlap RDD Sample would be identified by
matching the SSNs of the veterans in the RDD Sample with the list frame.
However, out of 12,956 completed extended RDD interviews (including
Puerto Rico), we were able to obtain an SSN from only 6,237 veterans,
which is 48.1 percent of the RDD completed extended interviews. The
veterans sampled as part of the RDD Sample could thus only be
categorized as belonging to the overlap RDD Sample or nonoverlap RDD
Sample if the SSN was reported. For others (those who did not report
their Social Security numbers), we used a prediction model to impute the
overlap status. The imputation of the overlap status and the
construction of composite weights are discussed in the following
sections.

####  Imputation of Overlap Status of Veterans Not Reporting SSN

We used the following model to predict the probability that a veteran in
the RDD Sample for whom an SSN could not be obtained would actually
belong to the overlap domain.

![](media/image89.wmf), (6-8)

where![](media/image90.wmf)is equal to the probability that a veteran in
the completed RDD Sample belongs to the overlap domain;
![](media/image91.wmf) is the probability that a veteran in the
completed RDD Sample reported the SSN; ![](media/image92.wmf) is the
conditional probability that a veteran in the completed RDD Sample with
a reported SSN belongs to the overlap domain; ![](media/image93.wmf) is
equal to the probability that a veteran in the completed RDD Sample did
not report a SSN, which is given by ![](media/image94.wmf); and
![](media/image95.wmf) is the conditional probability that a veteran in
the completed RDD Sample with unreported SSN belongs to the overlap
domain.

We needed to determine the probability of overlap that was conditional
on not reporting an SSN (i.e., ![](media/image96.wmf)). This can be
computed from the above expression because all other probabilities are
known. We used *CHAID* analysis to determine homogeneous classes of
overlap for those reporting SSNs in order to impute the overlap status
within each class for those not reporting an SSN. We used demographic
and socioeconomic variables, such as age, gender, race, education,
income, and priority group as predictor variables in the *CHAID* model.
The probability of overlap conditional on not reporting an SSN (i.e.
![](media/image97.wmf)) was determined independently for each cell, and
the overlap status was imputed by taking a random sample of the veterans
out of those who did not report an SSN. In other words, the overlap
status of the veterans with an unreported SSN within a class was imputed
as belonging to the overlap domain such that the proportion belonging to
the overlap was as close to the desired probability as possible. The
proportion belonging to the overlap domain was based on the weighted
counts. Thus, the above approach is an imputation approach that
effectively uses auxiliary variables, such as demographic variables and
enrollment priority groups, to impute the overlap status of the RDD
Sample veterans who did not provide Social Security numbers.

The veterans in the overlap RDD Sample (including the imputed cases)
also had a chance of being selected in the List Sample, and hence, had
an increased chance of selection. These RDD cases are referred to as the
overlap sample because they represent the portion of the RDD frame that
overlaps with the list frame. A composite weight was created for the
identified overlap RDD Sample (both observed and imputed) and List
Sample cases using the principles of composite estimation so that the
combined RDD and List Sample file could be used for analysis.

####  Calculation of Composite Weights

Composite weights were calculated using an approach developed by Hartley
(1962). We selected this approach because it could be adapted to take
into account the design effects of the RDD and List Sample designs when
combining the two samples. The List and RDD Samples were combined into
one file, consisting of 12,956 completed extended interviews from the
RDD Sample, and 7,092 completed extended interviews from the List
Sample, resulting in a combined sample of 20,048 completed extended
interviews.

In composite estimation, the estimates being combined are assumed to be
independent, and are unbiased estimates of the same population
parameter. In other words, the List Sample and the overlap RDD Sample
cases theoretically represent the same population (i.e., veterans on the
list frame). Therefore, a linear combination of the two independent
estimates would also produce an unbiased estimate. The parameter for
constructing the composite weights is chosen so that the variance is
minimized. The composite weight for each veteran in the RDD Sample and
List Sample was calculated as

![](media/image98.wmf) (6-9)

where

![](media/image99.wmf) = original List Sample weight; and

![](media/image100.wmf) = original RDD Sample weight.

The parameter ![](media/image101.wmf) (0 \< ![](media/image102.wmf)
\< 1) defines the composite weight that is used to produce the composite
estimate as a linear combination of the List Sample estimate and the
overlap domain RDD Sample estimate. The optimum value of the parameter
![](media/image103.wmf) for estimating a proportion is given by

![](media/image104.wmf), (6-10)

where

![](media/image105.wmf) = variance of a proportion from the List Sample;
and

![](media/image106.wmf) = variance of a proportion from the overlap RDD
Sample.

The composite weight gives increased weight to the estimates with
smaller variance, namely a smaller value of ![](media/image107.wmf).
Thus, the weight assigned to each of the estimates is inversely
proportional to the corresponding variance. In practice, the survey
estimates of proportions are produced for several characteristics and
each would have its own optimum value of the parameter
![](media/image108.wmf). It would not be practical to have a separate
set of weights for these characteristics and a common
![](media/image109.wmf) value is highly desirable for the sake of
internal consistency of the estimates. Therefore, the
![](media/image110.wmf) values corresponding to these estimates were
averaged according to the formula

![](media/image111.wmf), (6-11)

where

![](media/image112.wmf) = ![](media/image113.wmf) for the
![](media/image114.wmf) estimated proportion;

![](media/image115.wmf) = design effect for the ![](media/image116.wmf)
estimated proportion;

![](media/image117.wmf) = number of responding veterans;

*RDD* = overlap RDD Sample; and

*List* = List Sample.

In the above formula, the sample size when divided by the design effect
represents the effective sample size as compared with simple random
sampling because of such design features as clustering and unequal
probabilities of selection. Thus, the value of ![](media/image118.wmf)
is obtained by taking the weighted average of the individual
![](media/image119.wmf) values where the weights are proportional to the
corresponding effective sample sizes. The rationale for the above
averaging formula was that it gave more weight to the
![](media/image120.wmf) values that are based on larger effective sample
sizes.

The composite weight gives increased weight to the estimate with the
smaller variance (or larger effective sample size). There would be some
loss of variance efficiency from using a common ![](media/image121.wmf)
value for all of the characteristics instead of optimum
![](media/image122.wmf) for each of the characteristics. The increase in
the variance for a characteristic would depend on the absolute
difference between the common (average) ![](media/image123.wmf) value
and the optimum ![](media/image124.wmf) value for the particular
characteristic.

We computed the estimates of proportions and their variances for 16
statistics identified as key variables by the VA for the List Sample and
the overlap portion of the RDD Sample. These variables are listed in
Table 6-1.

Table 6-1. VA key variables

MB24: Combat or War Zone Exposure (Yes/No)

DIS1: Ever Applied for VA Disability Benefits (Yes/No)

HB21: Currently Covered by Medicare (Yes/No)

HC1: Emergency Room Care During Last 12 Months (Yes/No)

HC4a: VA Paid for Emergency Room Care (Yes/No)

HC5: Outpatient Care During Last 12 Months (Yes/No)

HC6: VA Facility for Outpatient Care (Yes/No)

HC9: Hospitalized Overnight in a VA Hospital (Yes/No)

SD14d: VA Service Connected Disability Compensation in 2000 (Yes/No)

SD14e: VA Non-Service Connected Pension in 2000 (Yes/No)

SD14j: Income Source: Public Assistance in 2000 (Yes/No)

ET1: Ever Received Any VA Education or Training Benefits (Yes/No)

ML3a: Ever Used VA Loan Program to Purchase Home (Yes/No)

ML3b: Ever Used VA Loan Program for Home Improvement (Yes/No)

ML3c: Ever Used VA Loan to Refinance Home (Yes/No)

PRIORITY: Priority Group (Mandatory/Discretionary)

The weighted average of the individual ![](media/image125.wmf)'s based
on the variables in the above table was computed according to the
formula given in equation 6-11. The average ![](media/image126.wmf)
value turned out to be 0.7272 and was used to construct the composite
weights for the combined sample. The individual ![](media/image127.wmf)
values ranged from 0.56 to 0.88.

####  Raked Composite Weights

The composite weights obtained by combining the List and RDD Samples
were also raked using the same two dimensional raking procedure that was
used for the RDD sample raking. The only difference was that we did not
need to collapse the cells in the first raking dimension, which was
defined by cross classification of age, education, and race/ethnicity.
The RDD Sample sizes for three cells in the first raking dimension were
not sufficient and these cells had to be collapsed for the raking
procedure. The combined RDD and List Sample sizes were more than 30 for
all 48 cells used for the first raking dimension and hence we did not
need to collapse cells.

The RDD Sample was raked mainly to correct for undercoverage because of
nontelephone households and households with unlisted numbers in the
"zero-listed telephone banks" that were missed in the list-assisted RDD
sampling methodology. The composite weights were raked to achieve
consistency with the C2SS estimates, and to improve the precision of the
survey estimates. The improvement in the precision of the survey
estimates would depend on the strength of correlation between the study
variable and the variables employed in the raking procedure. The raking
procedure is most beneficial if the estimation domains are defined on
the basis of the raking variables, or if these variables are highly
correlated with the study variables. We used the first raking dimension
by cross classification of the variables age, education, and
race/ethnicity to preserve the correlation structure among these
variables. Similarly, the second dimension was defined by cross
classification of the variables gender and census region. The variances
of the national level estimates of totals of the variables used in the
raking procedure would be identically equal to zero, which is an
additional benefit of the raking procedure.

# 6.4 Replicate Weights

A separate set of replicate weights was created for the RDD Sample and
the List Sample. These were then combined to construct the preliminary
composite replicate weights. The final composite replicate weights were
obtained by using the same two dimensional raking procedure with the
preliminary composite replicate weights as the input weights that were
used for the composite full sample weights.

####  List Sample Replicate Weights

A set of 51 Jackknife 1 (JK1) replicate weights was created for the List
Sample for use in variance estimation. The replicate weights were
designed for the JK1 replication method. To create the replicate
weights, the entire List Sample, including ineligible and nonresponding
veterans, was sorted by the twelve sampling strata, and by the order of
selection within strata. The strata were not explicitly used in the
assignment of replicates but the gains due to stratification were
reflected in sorting the sample cases by strata. Records 1, 1+51,
1+2\*51, 1+3\*51, and so on were assigned to the first replicate group.
Records 2, 2+51, 2+2\*51, 2+3\*51, and so on were assigned to the second
replicate group. The same approach was used with each succeeding
replicate group without regard for strata boundaries, until all records
were assigned to one of the 51 replicate groups. The replicate base
weights for the ![](media/image128.wmf) replicate were created by
setting to zero the base weights for the records in the
![](media/image129.wmf) replicate group and reweighting the base weights
in the remaining replicate groups by the factor 51/50.

The same adjustments applied to the full List Sample base weights to
obtain the full List Sample final weights were applied to the replicate
base weights to obtain the List Sample replicate final weights. This
included poststratification and the extended interview nonresponse
adjustments that were recalculated for each replicate, so that the
sampling variability in the response rates would be captured in the
replicate weights. The randomness in the number of sampled ineligible
cases is also reflected in the varying number of sampled eligible
veterans in each replicate.

####  RDD Sample Replicate Weights

A set of 51 JK1 replicate weights was also created for the veterans
identified from the RDD Sample. JK1 replicates were assigned by first
sorting the entire RDD Sample of telephone numbers, both eligible and
ineligible, in the order of selection of the 10-digit numbers that
determined each original RDD Sample. Records 1, 1+51, 1+2\*51, 1+3\*51,
and so on were assigned to the first replicate group. Records 2, 2+51,
2+2\*51, 2+3\*51, and so on were assigned to the second replicate group.
The same approach was used with each succeeding group, until all records
were assigned to one of the 51 replicate groups. The replicate base
weights for the ![](media/image130.wmf) replicate were created by
setting to zero the base weights for the records in the
![](media/image131.wmf) replicate group and reweighting the base weights
in the remaining replicate groups by the factor 51/50. The replicate
base weights for the Puerto Rico RDD Sample were computed in the same
way as those for the national (list-assisted) RDD Sample.

The replicate base weights were adjusted following the same steps as
those applied to the full sample base weights. These included the
screener level nonresponse adjustment, adjustment for multiple
residential telephone lines, extended interview level nonresponse
adjustment, and raking to the external veteran population counts
obtained from the Census 2000 Supplementary Survey. By raking the
replicate weights in the same manner as the full sample weights, the
sampling variability in the raking adjustment factors would be reflected
in the replicate weights, and hence included in the overall variance
estimate. The raking procedure was carried out on the combined national
and Puerto Rico RDD Samples.

If there were two or more veterans in a household, each respondent in
the household received the same set of replicate base weights but the
adjusted weights could differ because they could belong to different
adjustment cells.

####  Composite Replicate Weights

To create the composite replicate weights, each replicate weight from
the List Sample was multiplied by the same value of parameter
![](media/image132.wmf) (=0.7272) that was used for creating the
composite full sample weight. For the overlap RDD Sample cases, each
replicate weight was multiplied by a factor of ![](media/image133.wmf).
The remaining RDD Sample cases were assigned composite replicate weights
equal to their original RDD Sample replicate weights. Finally, the
composite replicate weights were raked to the veteran population counts
estimated from the C2SS in a two dimensional raking procedure as was
done for the composite full sample weights. The convergence criteria for
the composite replicate weights was modified so that the percent
absolute relative difference was no more than 0.1 percent for all
marginal population counts. We recall that the convergence criteria for
the composite full sample weights was that the percent absolute relative
difference was no more than 0.01 percent for all marginal population
counts.

# 6.5 Reliability of the Survey Estimates

Because estimates are based on sample data, they differ from figures
that would have been obtained from complete enumeration of the veteran
population using the same instrument. Results are subject to both
sampling and nonsampling errors. Nonsampling errors include biases from
inaccurate reporting, processing, and measurement, as well as errors
from nonresponse and incomplete reporting. These types of errors cannot
be measured readily. However, to the extent possible, each error has
been minimized through the procedures used for data collection, editing,
quality control, and nonresponse adjustment. The variances of the survey
estimates are used to measure sampling errors. The variance estimation
methodology is discussed in the next section.

####  Estimation of Variances of the Survey Estimates

The variance of an estimate is inversely proportional to the number of
observations in the sample. Thus, as the sample size increases, the
variance decreases. For the NSV 2001 the variance estimation methodology
for estimates of totals, ratios (or means) and difference of ratios is
based on the JK1 replication method, and the corresponding variance is
given as:

![](media/image134.wmf) (6-12)

where

![](media/image135.wmf) is an arbitrary parameter of interest;

![](media/image136.wmf) is the estimate of ![](media/image137.wmf) based
on the full sample;

![](media/image138.wmf) is the estimate of ![](media/image139.wmf) based
on the observations included in the ![](media/image140.wmf) replicate;

*R* is the total number of replicates formed; and

![](media/image141.wmf) is the estimated variance of
![](media/image142.wmf).

We have constructed the composite full sample and composite replicate
weights for the combined List and RDD Samples corresponding to the JK1
replication methodology. The WesVar[^8] variance estimation system can
be used to produce the survey estimates based on the composite full
sample weights and the corresponding variances of these estimates using
the variance formula given in equation 6-12.

####  Construction of Confidence Intervals

Each of the survey estimates has an associated standard error, which is
defined as the square root of the variance of the estimate. Consider the
example of estimating the proportion of veterans with a certain
characteristic, such as a service-connected disability. We denote by
![](media/image143.wmf) the estimated proportion of

veterans with the particular characteristic of interest and let
![](media/image144.wmf) be the corresponding variance estimate. Then the
standard error of the estimated proportion ![](media/image145.wmf) is
given by

![](media/image146.wmf) (6-13)

The 95 percent confidence interval is the interval such that the unknown
proportion *p* would have a 95 percent probability of being within the
interval. The 95 percent confidence interval is given by

![](media/image147.wmf). (6-14)

The lower limit of the interval is ![](media/image148.wmf), and the
upper limit of the interval is ![](media/image149.wmf) The width
![](media/image150.wmf) is known as half-width of the 95 percent
confidence interval. The factor ![](media/image151.wmf) is the *t*-value
at ![](media/image152.wmf) with 50 degrees of freedom, which is
approximately equal to 2.0. The smaller the half-width of the confidence
interval, the more precise is the survey estimate.

Alternatively, the precision of the survey estimate can also be
expressed in terms of the coefficient of variation (cv) of the estimate.
The cv of an estimate is defined as the ratio of the standard error of
the estimate and the magnitude of the estimate expressed in percent.
Thus, the cv of the estimated proportion ![](media/image153.wmf) is
given by

![](media/image154.wmf), (6-15)

where ![](media/image155.wmf) is the standard error of the estimated
proportion ![](media/image156.wmf). The smaller the cv of the estimate,
the more precise is the estimate. The percent margin of error at the 95
percent confidence level can also be obtained by multiplying the cv of
the estimate by the factor ![](media/image157.wmf).

# 6.6 Bias and Precision in the Combined Sample

We investigated two main issues associated with the use of the combined
sample versus the separate RDD and List Samples. These were: (1)
potential biases incurred in the estimates as a result of the matching
involved in creating the composite weights, and (2) the gains in
precision from the increased sample sizes of the combined sample. The
reason that both of these issues are important is that the total mean
square error (*MSE*) of a survey estimate is equal to the sum of its
variance and the square of the bias, (![](media/image158.wmf)). In
surveys with large sample sizes, the *MSE* may be dominated by the bias
term. When sample sizes are small, the variance may be a greater cause
for concern.

To address the first issue of bias, the potential risk of bias would be
due mainly to imputing the overlap status of those RDD sample
respondents who did not provide their Social Security numbers. We
obtained an SSN from only 48 percent of the RDD Sample respondent
veterans. Thus, the overlap status had to be imputed for those who did
not report their SSNs. The question arises as to whether the cases that
reported an SSN are different from those that did not. To answer this
question, statistical comparisons were made for the two groups to see
whether their distributions differed with respect to age and to other
key statistics. Pairwise t-tests showed that those not reporting an SSN
are:

-   More likely to be in the over 50 age group;

-   More likely to be in the higher income group;

-   More likely to have a higher education; and

-   More likely to belong to a discretionary priority group.

All comparisons are significant at the alpha = 0.05 level. For those who
reported an SSN, we compared the characteristics of those who were on
the list frame with those who were not on the list frame. The
significant variables for this comparison were priority group, income,
outpatient care, VA loan, and VA service-connected disability
compensation. We used these variables as predictor variables in the
*CHAID* analysis to determine homogeneous cells for imputing the overlap
status for those who did not report their SSN. Therefore, the risk of
potential bias was minimized due to imputing the overlap status within
homogenous imputation cells.

The precision of the estimates can be evaluated by comparing the
standard errors (SEs) of the estimates from the combined sample with
those from the RDD Sample alone. In this situation, the population of
analytical interest is the population of all noninstitutionalized
veterans living in the U.S. The statistics of interest for the purpose
of this analysis are proportions for various key statistics identified
by the VA. As can be seen from the comparison of SEs in Table 6-2, the
increased sample sizes of the combined sample always result in a
significant reduction in sampling variability. The standard errors of
the combined estimates are always lower than the standard errors of the
corresponding estimates from the RDD Sample alone. The design effects
for the combined sample would generally be higher than the corresponding
RDD Sample design effects due to increased variation in the sampling
weights. The standard error of a survey estimate is inversely
proportional to the square root of the effective sample size, where
effective sample size is defined as the number of cases sampled divided
by the design effect. Thus, the standard errors of the combined
estimates would be lower than the RDD estimates as long as the increase
in the design effect is less than the increase in the sample size. The
ratio of the sample sizes for the combined sample and the RDD Sample
alone is 1.54 (combined sample size divided by RDD Sample size). The
standard error of the combined estimates therefore would be less than
the standard error of the estimate from the RDD Sample alone as long as
the design effect ratio is less than 1.54. We note from Table 6-2 that
the design effect ratios for all the variables are less than 1.54. In
fact, the design effect ratios are less than 1 for priority groups 1
through 4 and the service-connected disability (SD14d).

We recall that the List Sample design is a stratified design, where
stratification is based on the health care priority groups (groups 1
through 6) and gender (male, female). The List Sample covered only the
mandatory priority groups (groups 1 through 6). The gains from
stratification for priority groups 1 through 4 more than offset the
losses due to increased variation in the combined sample weights. Hence,
the combined sample design effects are less than the RDD Sample design
effect. The gains from stratification for priority groups 5 and 6 were
not very large because of "strata jumpers." Many veterans who were
thought to belong to priority groups 5 and 6 were actually observed as
belonging to priority group 7. Therefore, the combined sample design
effects for priority groups 5 and 6 are higher than the RDD Sample
design effects. The combined sample design effect for the variable
"SD14d" (service-connected disability) is lower than the RDD Sample
design effect because of a high correlation between "SD14d" and the
priority groups.

The efficiency of the combined sample as compared with the RDD Sample
can also be defined as the ratio of the corresponding variances
expressed as percentage. We denote by ![](media/image159.wmf) the
efficiency of the combined sample as compared with the RDD Sample alone,
then

![](media/image160.wmf), (6-16)

Table 6-2. Comparison between RDD and composite estimates

  ---------- ------------- ------- -------- ------ ------- ------ ----------- ------ ------- ------ ----------- -------
  Question   Variable      Value   RDD only                       Composite                         RDD vs.     
  Number     Description                                                                            Composite   

                                   Est(%)   SE     n       Deff   Est(%)      SE     n       deff   Deff Ratio  Var
                                                                                                                Ratio

             Priority      1       3.3      0.15   425     0.95   3.2         0.11   1974    0.83   0.87        1.86
             Group                                                                                              

                           2       2.6      0.16   337     1.30   2.7         0.09   1666    0.60   0.46        3.16

                           3       9.0      0.25   1162    0.96   9.0         0.18   3048    0.82   0.86        1.93

                           4       0.1      0.03   10      1.17   0.1         0.02   96      0.55   0.47        2.25

                           5       18.5     0.31   2296    0.82   18.4        0.26   3599    0.89   1.08        1.42

                           6       10.9     0.31   1432    1.30   10.9        0.29   1887    1.77   1.36        1.14

                           1 -- 6  44.4     0.43   5662    0.99   44.3        0.38   12270   1.20   1.22        1.28

                           7       55.6     0.43   7294    0.99   55.7        0.38   7778    1.20   1.22        1.28

  MB24       COMBAT1       Yes     39.2     0.47   5145    1.18   39.2        0.46   9253    1.74   1.48        1.04

                           No      60.8     0.47   7811    1.18   60.8        0.46   10795   1.74   1.48        1.04

  ET1        EDUCTRG1      Yes     40.2     0.48   5369    1.25   40.2        0.42   8266    1.46   1.17        1.31

                           No      59.8     0.48   7587    1.25   59.8        0.42   11782   1.46   1.17        1.31

  HC1        ERYOU1        Yes     24.1     0.33   3107    0.75   24.1        0.32   5628    1.10   1.46        1.06

                           No      75.9     0.33   9849    0.75   75.9        0.32   14420   1.10   1.46        1.06

  HB21       MEDICARE1     Yes     39.3     0.23   5356    0.29   39.4        0.20   8789    0.35   1.22        1.32

                           No      60.7     0.23   7600    0.29   60.6        0.20   11259   0.35   1.22        1.32

  SD14d      VADISCMP      Yes     11.1     0.29   1436    1.10   11.2        0.18   5991    0.68   0.61        2.60

                           No      88.9     0.29   11520   1.10   88.8        0.18   14057   0.68   0.61        2.60

  SD14j      WELFARE       Yes     2.1      0.13   266     1.11   2.0         0.12   451     1.54   1.39        1.17

                           No      97.9     0.13   12690   1.11   98.0        0.12   19597   1.54   1.39        1.17
  ---------- ------------- ------- -------- ------ ------- ------ ----------- ------ ------- ------ ----------- -------

where ![](media/image161.wmf) and ![](media/image162.wmf) are,
respectively, the variances of the RDD Sample alone and the combined
sample. The efficiency values of more than 100 percent imply that the
combined sample estimates are more efficient than the estimates based on
the RDD Sample alone. We notice that efficiencies are greater than 100
percent for all variables in Table 6-2 and the efficiency values range
from 104 percent to 316 percent. Thus, the combined sample with the
corresponding composite weights should be used for all VA analyses.

7\. QUALITY CONTROL

Through all phases of the 2001 National Survey of Veterans (NSV 2001),
Westat paid strict attention to the accuracy of our work. Because we
integrated quality control procedures into every aspect of the research
process, we included descriptions of these procedures in our discussions
of each project component in the previous chapters of this report. This
chapter briefly presents the measures taken primarily to enhance
quality. It also presents detailed information about two key quality
control measures -- the NSV 2001 pretest and managing the database.

####  Designing and Programming the Questionnaire

The NSV 2001 questionnaire went through several design drafts and a
number of iterations for the computer-assisted telephone interviewing
system (CATI) program specifications. To assure that design decisions
accurately reflected study goals, we kept a detailed log of
questionnaire revisions. Westat and VA project staff thoroughly reviewed
each draft to assess its effectiveness in addressing NSV 2001 data
requirements.

The decision to use CATI was guided by the need for quality control. As
noted in Chapter 5 (Data Collection), using a CATI system to conduct
telephone interviews facilitates proper administration of the
questionnaire. However, this is the case only if we develop and
correctly implement the questionnaire specifications. We took several
steps to make certain that the specifications were accurate.
Questionnaire designers and CATI programmers worked together to develop
the specifications for programming the CATI instrument. By using this
team approach, we ensured that the CATI instrument accurately reflected
the NSV 2001 questionnaire. Once the instrument was programmed, the team
put it through numerous rounds of testing. Testing covered every aspect
of the CATI program, including screen displays, transitions from one
section to the next, skip patterns, data storage, sample management,
result codes, randomization routines, and delivery of cases to
interviewers. As part of the testing process, we documented all
potential problems and had at least two project staff members verify
that each was resolved. Finally, we maintained a history of all changes.

####  Training and Supervising Interviewers

The NSV 2001 interviewers were central in our effort to collect accurate
information, and to meet our data collection goal of 20,000 completed
interviews. As a company, Westat spends a great deal of time honing our
training techniques. Two important aspects of training are the high
level of interviewer involvement and the consistent manner in which the
training exercises are administered. Trainee involvement is critical
because it allows trainers to observe and evaluate individual
performance. Trainers can then decide early on who should be released
from the program because of failure to meet Westat performance
standards. Even more important, trainers can identify specific areas of
difficulty, and tailor the program accordingly. Scripted exercises and
role plays allow trainers to maintain a high level of consistency across
multiple training sessions (there were eleven such sessions for the NSV
2001). Scripting the mock interviews in advance also ensured that
interviewers had the opportunity to practice responding to various
scenarios they were likely to encounter during administration of the NSV
2001, better equipping them to overcome initial respondent refusals.

During data collection, the primary method of ensuring that interviewers
continued to accurately administer the NSV 2001 questionnaire was
through our monitoring sessions. Project staff and telephone center
supervisors monitored an average of 8 percent of all NSV 2001
interviews. We assessed interviewers' administration of the
questionnaire, recording of responses, probing, handling of contacts,
and professional demeanor. Monitors reported the results of each session
on a monitoring form, then shared them with interviewers (who were
unaware they were being monitored until after the session was over).
Interviewers were apprised of their strengths as well as areas needing
improvement. If needed, we provided additional training, or made
adjustments to the interviewing process in general. In conjunction with
the monitoring sessions, we used summary reports of the number of call
attempts, successful contacts, refusals, and completions to assess
interviewer performance. Based on these reports, we were able to
identify interviewers with skills that could be matched to operational
areas specific to the NSV 2001, such as proxy interviewing, tracing
calls, refusal conversion, and language problem cases. Finding the best
interviewers for these particular tasks ensured that they were carried
out with the highest levels of accuracy and skill.

####  Sampling

We prepared several sample design approaches and analyzed these before
the final design was chosen. Our detailed analyses assessed coverage of
the veteran population and subgroups of interest, precision, cost, and
operational demands. We developed detailed specifications for
constructing the list and RDD frames to assure their accuracy and
statistical validity. To minimize bias from time effects, we created
sample release groups and made each one available for calling at
separate intervals throughout the data collection period. As a means of
tracking the List and RDD Samples separately, we developed a system of
IDs that differentiated cases from each sample. Finally, we checked all
steps in the sample file creation and other sample processing stages of
data collection by designing and producing frequencies and
cross-tabulations of appropriate variables.

####  NSV 2001 Pretest

To determine if, under live interviewing conditions, our data collection
procedures would work as we had anticipated, we conducted a pretest of
the NSV 2001. The objectives of the pretest were to:

-   Test item wording, survey flow, skip patterns, ease of
    administration, respondent comprehension of the survey questions,
    and other factors related to the instrument itself;

-   Check that all functions of the CATI system, including data storage,
    call scheduling and case management, were working properly;

-   Establish the average length of time it took to administer the
    instrument;

-   Evaluate our training procedures; and

-   Solicit feedback from interviewers about all aspects of the
    interviewing process.

An additional benefit of the NSV 2001 pretest was that it afforded the
VA an excellent opportunity to observe the methodologies and procedures
planned for the main data collection phase.

The NSV 2001 pretest was conducted at one Telephone Research Center
between February 12, 2001 and March 4, 2001. During that period, the
List Sample contact procedures were still in development, so the pretest
was administered using RDD Sample cases only. The entire pretest effort
was based on an initial RDD Sample of 60,000 cases, all of which were
loaded into the CATI system. Of the 60,000 telephone numbers, 17,616
were eliminated from calling because they were either business or
nonworking telephone numbers. Therefore, 42,384 telephone numbers
remained available for dialing during the pretest.

Pretest interviewers called 21,609 telephone numbers. During the
screening portion of the telephone interviews, interviewers completed
2,928 screening questionnaires and identified 901 potential veterans in
852 households as eligible to participate in the NSV 2001. This rate of
1.06 veterans per eligible household varied little throughout the entire
data collection effort. At the conclusion of the pretest, interviewers
had completed 519 extended interviews. Figure 7-1 is a flowchart that
summarizes the magnitude of the pretest sample and workload, as well as
the outcome of calls during the pretest. On March 16, 2001, Westat
briefed the VA on these pretest call results.

The pretest revealed that the CATI instrument worked as we intended it
to. Nor did we discover any problems with the CATI system's call
scheduling, case management, or data storage functions. We did, however,
modify our yield assumptions to reflect the actual completion rates.
(See Chapter 8 for a more detailed discussion of completion rates.). We
also learned from the pretest that the average length of the interview
was slightly over the target of 30 minutes. Finally, we revised our
training program to increase the focus on one area that presented
difficulties for interviewers and respondents in the pretest --
correctly identifying current household members.

####  Managing the Database

Database managers for the NSV 2001 had two key responsibilities:
ensuring that the data collected were consistent, valid, and within the
specified ranges, and ensuring that result codes[^9] accurately
reflected the status of cases. In the course of this work, database
managers also ensured that the CATI instrument was operating correctly,
that the responses were being recorded correctly in the database, and
that interviewers were administering the questions and using the
response categories correctly. Database managers at Westat employed
several methods to carry out these responsibilities. We programmed an
automated data check, manually reviewed question by question response
frequency

![](media/image163.wmf){width="5.891666666666667in"
height="8.221527777777778in"}

Figure 7-1. Flowchart of NSV 2001 pretest sample activities

reports, reviewed and recoded open-ended responses as necessary, and
reviewed interviewer comments. This section describes each of these
methods in turn, along with the additional steps we took to prepare the
database for delivery after the close of data collection.

It is important to note that we did not impute missing data during this
phase of the project. We did, however, impute a few variables that were
required for the weighting effort. (See Chapter 6 for details about data
weighting for the NSV 2001.)

**Automated Data Checking.** To reduce survey administration time, we
programmed item MB18 to automatically determine and store the veterans'
service eras based on their responses to questions about when they began
and were released from active duty. The programming logic for this item
was complicated, using dozens of "if-then" statements. Because it would
have been difficult to identify errors by doing a manual review of this
item, not to mention unreasonably time consuming, we created a SAS^^
program that evaluated the various sets of variables defining each
respondent's entry and exit dates from the military, and used that
information to derive a corresponding date range. This date range was
then compared to the preestablished date range of each service era to
determine whether the respondent was in the military at that time.
Finally, the program compared its findings to the service era flags that
were set automatically by the CATI system and noted any discrepancies.
We ran this program every 3 to 4 weeks during data collection. On those
rare occasions when errors were found, they were easily corrected
because they were simply the result of one or two variables being missed
during a manual update.

**Review of Question by Question Response Frequency Reports.** Several
times throughout data collection we created a question by question
response frequency report. Using this report, we checked that responses
fell within the allowable range for each question, verified that the
relationships of responses across individual questionnaires were
logical, and ensured that the data accurately reflected the instrument's
skip patterns. We produced this report after the first 100 cases were
completed, at the end of the pretest, about halfway through data
collection, and after data collection closed.

The set of questions that asked veterans about the dates they were on
active duty comprised a complicated, and somewhat lengthy, series of
items. Veteran responses to those questions then determined which
categories would be displayed in survey item MB20 ("Now I'm going to
read a list of places. Please tell me if you served in, sailed in, or
flew missions over each of the following while on active duty."). To
check that the proper categories were displayed, and that data were
correctly recorded at MB20, we had to rely on more than just the
frequency report. We generated detailed cross-tabulation reports that
allowed us to examine the logical relationships among these items. Any
discrepancies we found in the expected relationships were corrected by
updating the affected variables.

The frequency report allowed us to conduct range checks of every
variable. The range checks compared the specified ranges in the CATI
data dictionary with the responses entered during interviewing or
derived by the CATI programs. The few out-of-range responses we found
were verified as representing valid answers that happened to be outside
the range we had anticipated.

In a few instances, our review of the frequency report revealed the need
to update the CATI program. In the first instance, responses entered at
the Gender verification screen as "F" for "female" rather than "2" (the
usual code for "female") resulted in eight cases in which the
respondents were not asked whether they served in branches reserved for
women (survey items MB23f -- MB23l). The variables that should have
contained valid responses in those eight cases were set to the value
"Not Ascertained."

We also updated survey item SD19 ("What is the market value of your
primary residence?") to allow a value as low as zero, made minor wording
changes to survey item SD13 ("During the year 2000, how many children
depended on you for at least half of their support?") and survey item
SD17 ("Excluding your primary residence, what is the total amount of
assets your family owns?"), and added a category to survey item BB3
("What do you want done with your ashes?"). At the request of the VA, we
changed survey items SD10 and SCD10 ("I am going to read you a list of
racial categories. Please select one or more to describe your race.") to
ensure that it matched the approach to collecting race items used by the
U.S. Bureau of the Census.

**Recoding of "OTHER (SPECIFY)".** Because all response data were
written to the CATI database directly from the interviewer data entry
function, and the instrument included no open-ended questions that
required post-coding, the NSV 2001 data posed no significant coding
requirements. However, 24 closed-ended questions gave veterans an
opportunity to add an "OTHER (SPECIFY)" response when they felt that the
precoded response categories did not adequately cover their situation.
We reviewed the "OTHER (SPECIFY)" responses weekly for the first 6 weeks
of the survey period. In subsequent weeks we reviewed them less
frequently. When data collection was complete, we conducted a final
review of all "OTHER (SPECIFY)" responses. We checked whether the "OTHER
(SPECIFY)" responses duplicated a precoded response already given by the
veteran, provided amplified or more specific information about a
precoded response already given by the veteran, or duplicated a precoded
response that had not been given by the veteran. Only in the last
instance did we reset the response category to reflect the additional
information. Overall, very few of the "OTHER (SPECIFY)" responses
required that we reset them to precoded response categories.

**Review of Interviewer Comments.** NSV 2001 interviewers had the
opportunity to record their assessments of irregular cases in four
places. Interviewers could type their reports directly into the CATI
Comments, Non-Interview Report, or Message Files. Or, they could fill
out a paper problem sheet. Sometimes, their remarks qualified or, in
some cases altered, a veteran's responses in some way that did not fit
the categorical values of the survey variables. Other times, these
remarks had more general ramifications that affected the handling of the
case or interpretation of the data.

**Comments File.** The Westat CATI system permits an interviewer to
record a comment at any point in the interview. The comment is
automatically linked to the specific case and item at which it was
recorded. Interviewers used this function to record veteran comments
that clarify or correct information entered into the questionnaire. On a
daily basis, we reviewed Comments File output for potential changes or
updates to the data. We permanently retained all comments in the file.

The majority of the comments were simply the veteran's elaboration on
answers that had been properly recorded, and therefore required no
further action. However, in some cases the comments clearly affected
responses in ways that required data updates. For example, as veterans
cycled through questions, a few realized they had not understood the
original question and had replied incorrectly. When this occurred at a
point in the interview where it was impractical to back up and change
the original response, interviewers would make a comment to change the
previous response. The database managers would then correct the original
response and, where appropriate, update all related responses.

**Non-Interview Report File (NIRF).** For all refusal cases, language or
hearing problems, or other special problem cases, interviewers recorded
their assessment of the situation in the NIRF. While it is primarily
interviewers who use the NIRF, it is an additional source of information
for database managers and project management staff to use when deciding
how to handle cases that require attention beyond that of the telephone
supervisory staff. For the NSV 2001, we periodically checked the NIRF,
but seldom had occasion to use it for this purpose.

**Message File.** At the end of every telephone contact, the Message
File gives interviewers one more opportunity to record their
observations about cases that require special attention. Interviewers
and supervisors used this file to communicate more detailed information
about cases to other interviewers, database managers, and project
management. Database managers used the file to interpret inconsistent
responses and pose general questions about the cases. As with the NIRF,
we checked this file regularly but rarely had to use it.

**Paper Problem Sheets.** Interviewers and telephone center staff used
paper problem sheets to record data entry errors and request updates to
specific cases. Problem sheets were reviewed daily and handled by
database managers in the same way as the electronic Comments File.
Interviewers also used problem sheets to note instances where they
thought the CATI system had not functioned as they expected. These notes
were checked daily and resolved by database managers and programming
staff as needed.

**Final Review of the Database.** After the close of data collection, we
conducted a final, comprehensive review of the entire database of
completed interviews. We again ran the MB18 automated data check, the
detailed cross-tabulations, and the question by question response
frequency report described earlier.

In addition, we verified that every final result code had been assigned
according to project definitions. (See Appendix H for a list of NSV 2001
final result codes.) We did this through a series of cross-tabulations
that examined sample type and responses to specific survey questions.
The majority of result codes were correct. We changed a few extended
interview result codes to reflect that a proxy rather than the sampled
veteran had completed the interview.

####  Weighting

Statistical staff prepared detailed specifications for developing the
weighting systems for the List and RDD Samples. To validate the
implementation of these specifications, the systems staff and the
statistical staff independently checked the computer codes used to
calculate and assign weights. We checked the calculation of nonresponse
adjustment factors cell by cell. We also verified the calculation of
nonresponse adjusted weights, in part by examining the variation in the
nonresponse adjustment factors and the nonresponse adjusted weights. For
the key veteran population subgroups, we designed, developed and checked
cross-tabulations of appropriate variables during each step of the
weighting process. Additionally, we compared the weighted total from the
RDD Sample to the estimate from the VA population model, Vetpop2000, to
ensure consistency between the two. For construction of the composite
weights for the combined List and RDD Samples, we prepared detailed
specifications and checked their implementation by reviewing the
computer code used to calculate and assign the composite weights. To
ensure that the weight file created for delivery to the VA was without
error, we produced several summary statistics from it and compared those
statistics to the summary statistics produced from the original Westat
file.

8\. SURVEY FIELD RESULTS

This chapter presents the field results of the 2001 National Survey of
Veterans (NSV 2001) data collection effort. We first examine the final
sample yields and response rates of completed interviews against the
sample design targets. Next, we detail the call results for each type of
instrument and sample. We also present analyses of selected operating
statistics such as location and cooperation rates.

# 8.1 Completed Interviews

The NSV 2001 had sample targets of 13,000 completed extended interviews
from the household screening of RDD telephone numbers and 7,000
completed extended interviews from the List Sample, for a total of
20,000 completed extended interviews. Overall, we achieved 100.2 percent
of that target. For the RDD sample, we achieved 99.7 percent of the
target and for the List Sample, we achieved 101.3 percent of the target.

# 8.2 Completed Interviews by Sample Stratum

####  List Sample

The NSV 2001 sample design set interview completion targets by sample
stratum for both the List and RDD Samples. The main objective of the
List Sample stratification was to augment the sample with veterans in
the mandatory health care enrollment priority groups (groups 1 through
6), and with female veterans. The List Sample target for the female
veterans was 581 completed extended interviews, and we achieved 99.1
percent of that target (576 completed interviews with female veterans).
The List Sample targets for African American and Hispanic veterans were
574 and 280, respectively. We completed 897 extended interviews with
African American veterans, and 344 extended interviews with Hispanic
veterans. Thus, we almost met the target for female veterans and
exceeded the targets for African American and Hispanic veterans. We
attribute the higher than expected number of completed extended
interviews for these two groups to the fact that the list frame
contained a higher proportion of these minority groups than exists in
the veteran population, and the latter rate is what we had assumed for
sample design purposes.

We also compared the target and achieved List Sample completed extended
interviews by priority groups (Table 8-1). PRIOADJ3 represents the
priority group as it appeared on the list frame.

Table 8-1. Target versus achieved List Sample completed extended
interviews by priority group (PRIOADJ3)

  --------------- ------------------ ------------------ ------------------
  PRIOADJ3        Target             Achieved           Percent of Target

  1               1,239              1,410              113.8

  2               1,200              1,354              112.8

  3               1,636              1,758              107.5

  4               931                549                59.0

  5               1,231              1,179              95.8

  6               763                842                110.4

  TOTAL LIST      7,000              7,092              101.3
  --------------- ------------------ ------------------ ------------------

We exceeded the target for four out of the six mandatory priority
groups. The two groups where we fell short were priority group 5, for
which we achieved almost 96 percent of the target, and priority group 4,
for which we achieved only 59 percent of the target. It turns out that
we underestimated the proportion of priority group 4 veterans that would
be ineligible for the survey because they were institutionalized or
deceased. The proportion ineligible among the priority group 4 veterans
was 30.9 percent, whereas less than 5 percent of veterans in priority
groups 2, 3 and 6 were ineligible, and just over 7 percent of priority
groups 1 and 5 veterans were ineligible (see Table 8-4).

The target List Sample counts by priority group were derived from the
list frame variable, PRIOADJ3. Information on the List frame was for a
particular point in time. It is possible that by the time we interviewed
the veteran this information had changed, thus putting that veteran in a
different priority group. Table 8-2 gives the cross-tabulation of the
priority group as it appeared on the list frame (PRIOADJ3) and that
determined from survey responses (Observed Priority). Based on the
survey responses, the number of completed extended interviews for
priority group 4 veterans was much smaller than expected based on the
number of priority group 4 veterans on the list frame. In fact, many
veterans recorded on the list frame as belonging to priority group 4
turned out to be priority group 5 veterans. As a result, only 81 of the
List Sample eligible respondents who were priority group 4 veterans on
the list frame were actually categorized as belonging to priority group
4 on the basis of their survey responses. At the same time, a large
number of veterans recorded on the List frame as priority group 5 or
priority group 6 turned out to belong to other priority groups. From
priority group 5, 253 veterans were observed to belong to priority group
7. From priority group 6, 419 veterans gave survey responses that
categorized them as belonging to priority groups 3, 5 or 7.

Table 8-2. List priority (PRIOADJ3) by observed priority

  ----------- ---------- ------- ------- ------- ------- ------- ------- --------
  PRIOADJ3    Observed                                                   
              Priority                                                   

              1          2       3       4       5       6       7\*     ALL

  1           1,308      28      14      1       16      16      27      1,410

  2           85         1,105   54      4       36      16      54      1,354

  3           23         65      1,475   1       61      34      99      1,758

  4           13         6       10      94      345     8       73      549

  5           7          4       15      4       839     51      259     1,179

  6           14         23      72      1       194     395     143     842

  TOTAL LIST  1,450      1,231   1,640   105     1,491   520     655     7,092
  ----------- ---------- ------- ------- ------- ------- ------- ------- --------

\* 113 cases with unknown "Observed Priority" have been combined with
priority 7.

We did not ask questions about catastrophically disabled veterans on the
survey. These veterans fall into priority group 4. Pensioners receiving
aid and attendance were classified into either priority group 4 or
priority group 5. Priority group 6 veterans on the List sampling frame
were those enrolled for VA health care who were coming or planning to
come to the VA solely for treatment for disorders associated with
exposure to a toxic substance, radiation, or for disorders associated
with service in the Persian Gulf. Veterans who did not meet the criteria
for priority groups 1 through 5 and indicated exposure (regardless of
whether they were presently enrolled for this type of care) were
classified into priority group 6. Priority group 6 also included
compensated veterans with a zero percent rating who did not meet the
criteria for priority groups 1 through 5.

####  RDD Sample

We made assumptions about the distribution of the veteran population
across various subgroups, and used those assumptions to arrive at our
sample design targets for each subgroup. Because of sampling
variability, we expected the observed yields to differ somewhat from
those targets, even if our assumptions were correct. However, since the
observed yields depended on the actual (not assumed) distributions of
the veteran population across subgroups, the difference could be larger.

We set an RDD Sample target of 662 completed extended interviews with
female veterans, and actually completed 699 interviews (105.6 percent of
the target). The targets for African American and Hispanic veterans were
1,066 and 520 completed extended interviews, respectively. We completed
983 extended interviews with African American veterans (92.2 percent of
the target), and 558 extended interviews with Hispanic veterans (107.3
percent of the target) .

Table 8-3 shows that, while we met the overall target for the RDD
Sample, there was considerable variability among the individual priority
groups. According to the veteran population distribution used for the
NSV 2001 sample design (see Chapter 3), the two smallest priority groups
were groups 4 and 6. However, the number of veterans in priority group 4
turned out to be only a fraction (17.4 percent) of what we projected
from our assumed distribution. On the other hand, the number of veterans
belonging to priority group 6 was 41 times what we expected. This could
be partially due to the fact that the absolute numbers for the RDD
targets for priority groups 4 and 6 were also very low, which led to
large percentage differences. Also, for Priority 6 the definition for
the target population was slightly different than what was used to count
the achieved number. Data used to estimate the target population was
only available for veterans using VA facilities solely for treatment of
environmental exposures, whereas the observed population considered
anyone stating they had been exposed to environmental hazards regardless
of treatment. The RDD Sample completed extended interviews for priority
group 5 also fell significantly short, at just over 70 percent of the
target.

Table 8-3. Target versus achieved RDD Sample completed extended
interviews by priority group

  --------------- ------------------ ------------------ ------------------
  Observed        Target             Achieved           Percent of Target
  Priority                                              

  1               295                371                125.8

  2               271                300                110.7

  3               661                741                112.1

  4               69                 12                 17.4

  5               3,731              2,636              70.7

  6               36                 1,479              4,108.3

  7\*             7,937              7,417              93.4

  TOTAL RDD       13,000             12,956             99.7
  --------------- ------------------ ------------------ ------------------

\* 61 cases with unknown "Observed Priority" have been combined with
priority 7.

# 8.3 List Sample Extended Interview Results and Response Rates

For the NSV 2001 List Sample, we attempted to complete extended
interviews with a total of 13,129 individuals. With 7,092 of these, we
were successful in our attempts. Of the remaining cases, we determined
that 1,151 were out of scope, principally because they were deceased or
institutionalized, and 427 were, for a variety of reasons, eligible
nonrespondents. We could not determine the eligibility of 4,459 cases,
mainly because the veterans could not be located.

As described in Chapter 6, we divided sample cases into four eligibility
and response categories -- eligible respondents, ineligible (out of
scope), eligible nonrespondents, and eligibility unknown. Table 8-4
shows these response status categories for the six priority groups, as
represented by the list frame variable PRIOADJ3. The proportion of
eligible respondents was much lower for priority group 4 veterans as
compared to the other priority groups. This was due to the large
proportion of priority group 4 veterans that were ineligible (out of
scope) for the survey. The proportion of eligible nonrespondent veterans
was roughly the same across all priority groups except priority group 6,
for which that proportion was slightly lower.

Table 8-4. Response status by priority group (List Sample)

  -------------- -------------- --------------- -------------- --------------
  PRIOADJ3       Eligible\      Eligible\       Ineligible     Eligibility\
                 Respondent     Nonrespondent                  Unknown

  1              60.7           3.8             7.1            28.4

  2              60.1           3.4             4.3            32.2

  3              57.3           3.0             3.9            35.8

  4              31.4           3.0             30.9           34.7

  5              51.1           3.4             7.4            38.1

  6              58.9           2.7             4.1            34.3

  TOTAL LIST     54.0           3.2             8.8            34.0
  -------------- -------------- --------------- -------------- --------------

Table 8-5 shows telephone interviewing results by age group (less than
50, 50 to 64, and over 64). The proportion of ineligible persons
increased monotonically with age because the incidence of
institutionalized and deceased persons increases with age. On the other
hand, the proportion of eligibility unknown respondents decreased
monotonically with age because it is more difficult to locate younger

Table 8-5. Response status by age group (List Sample)

+-------------+-------------+-------------+-------------+-------------+
| Age Group   | Eligible\   | Eligible\   | Ineligible  | E           |
|             | Respondent  | No          |             | ligibility\ |
|             |             | nrespondent |             | Unknown     |
+-------------+-------------+-------------+-------------+-------------+
| > Less than | 48.0        | 2.9         | 1.9         | 47.2        |
| > 50        |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| > 50-64     | 59.5        | 2.6         | 4.0         | 33.9        |
+-------------+-------------+-------------+-------------+-------------+
| > Over 64   | 53.7        | 3.9         | 16.7        | 25.7        |
+-------------+-------------+-------------+-------------+-------------+
| > TOTAL     | 54.0        | 3.2         | 8.8         | 34.0        |
| > LIST      |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

veterans. The proportion of eligible nonrespondents was slightly higher
for elderly veterans (over 64 years) mainly because these persons were
more likely to be too incapacitated to respond to the survey. For the
same reason that younger veterans (less than 50 years) were more likely
to fall into the eligibility unknown category, a lower proportion of
them were eligible respondents.

Table 8-6 gives the distribution of response status for male and female
veterans. The proportion of ineligible male veterans was almost twice
that of female veterans, mainly because males are more likely to be
institutionalized or deceased. On the other hand, the proportion of
eligibility unknown respondents was higher for female veterans because
we had less information with which to locate them. As a result, the
proportion of eligible female veteran respondents was slightly lower
than that of male veterans.

Table 8-6. Response status by gender (List Sample)

+-------------+-------------+-------------+-------------+-------------+
| Gender      | Eligible\   | Eligible\   | Ineligible  | E           |
|             | Respondent  | No          |             | ligibility\ |
|             |             | nrespondent |             | Unknown     |
+-------------+-------------+-------------+-------------+-------------+
| > Male      | 54.1        | 3.3         | 9.1         | 33.5        |
+-------------+-------------+-------------+-------------+-------------+
| > Female    | 52.8        | 3.2         | 4.8         | 39.2        |
+-------------+-------------+-------------+-------------+-------------+
| > TOTAL     | 54.0        | 3.2         | 8.8         | 34.0        |
| > LIST      |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

Table 8-7 shows the List Sample veteran response status distributed
across census regions. The proportion of eligibility unknown respondents
in the Midwest region was slightly lower and the proportion of eligible
respondents slightly higher than those in the other three regions. The
proportion of ineligible persons was slightly lower in the West,
although the proportion of eligible nonrespondents was almost the same
in that region as in all others.

Table 8-7. Response status by census region (List Sample)

+-------------+-------------+-------------+-------------+-------------+
| Census      | Eligible\   | Eligible\   | Ineligible  | E           |
| Region      | Respondent  | No          |             | ligibility\ |
|             |             | nrespondent |             | Unknown     |
+-------------+-------------+-------------+-------------+-------------+
| > Northeast | 51.6        | 3.0         | 10.5        | 34.9        |
+-------------+-------------+-------------+-------------+-------------+
| > Midwest   | 56.8        | 3.6         | 9.7         | 29.9        |
+-------------+-------------+-------------+-------------+-------------+
| > South     | 54.1        | 3.2         | 8.6         | 34.1        |
+-------------+-------------+-------------+-------------+-------------+
| > West      | 53.0        | 3.2         | 6.8         | 37.0        |
+-------------+-------------+-------------+-------------+-------------+
| > TOTAL     | 54.0        | 3.2         | 8.8         | 34.0        |
| > LIST      |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

# 8.4 RDD Household Screening Results and Response Rates

As discussed in Chapter 6, the RDD Sample screening criteria was
somewhat loose. Since we asked one household member to report military
service for all other household members, we knew that some screener
respondents would not have enough knowledge to provide definitive
information for determining veteran status. Therefore, we asked detailed
military service questions of all potential veterans (or their
knowledgeable proxies) identified and selected for an extended interview
through the screening process. The results of this approach are shown in
Tables 8-8 and 8-9. A total of 305,500 telephone numbers were sampled
for the RDD Sample. Of these, 300,000 were sampled from the 50 states
and District of Columbia using a list-assisted random digit dialing
method (national RDD Sample) and 5,500 were sampled from Puerto Rico
using a naïve RDD sampling method (Puerto Rico RDD Sample).

Of the 300,000 telephone numbers sampled for the national sample,
205,949 were dialed. We did not call the remaining 94,051 numbers
because we had identified them as nonworking or business numbers before
data collection began. Table 8-8 shows the household level screening
interview response status for the 205,949 list-assisted RDD Sample
telephone numbers and the 5,500 Puerto Rico RDD Sample telephone numbers
for which calls were attempted. As discussed in Chapter 6, there were no
households with a response status of eligible nonrespondent because, by
definition, a completed screening interview can be categorized as
eligible or ineligible. Hence, we were able to assign an eligible or
ineligible status to all completed screening interviews. If the
household did not complete a screening

Table 8-8. Household screener interview results by sample type

+-----------------+-----------------+-----------------+-----------------+
| Sample Type     | Eligible        | Ineligible      | Eligibility     |
|                 | Respondents     |                 | Unknown         |
+-----------------+-----------------+-----------------+-----------------+
| > RDD National  | 18,810          | 116,212         | 70,927          |
+-----------------+-----------------+-----------------+-----------------+
| > RDD Puerto    | 96              | 4,137           | 1,267           |
| > Rico          |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| > TOTAL RDD     | 18,906          | 120,349         | 72,194          |
+-----------------+-----------------+-----------------+-----------------+

interview, then we assigned the response status eligibility unknown. The
RDD screener response rate was 67.6 percent.[^10]

We identified 18,906 RDD Sample households (national and Puerto Rico) as
containing potential veterans and 120,349 as ineligible. We could not
determine eligibility for the remaining 72,194 numbers. Of the 120,349
ineligible telephone numbers, 54,304 were residential telephone numbers
(i.e., telephones for households with no veterans in them), and the rest
were either business or nonworking telephone numbers. Thus, a total of
73,210 telephone households were identified during RDD screening and
18,906 had at least one potential veteran.

Table 8-9 shows the comparative results of the household screening
process, in terms of the yield of households with potential veterans. Of
all households screened, 25.8 percent (18,906) had at least one
potential veteran. Because of the deliberately relaxed criteria for
identifying potential veterans at this stage of the screening, the
proportion of households with a potential veteran found during our
screening process was higher than the proportion determined by the
extended interview (22.1 percent). The assumption we made for the
purpose of the sample design was that one out of every four households
would be a veteran household (see Chapter 3).

Table 8-9. Distribution of telephone households by number of potential
veterans in the household

  --------- ------------ ------------ ------------ ------------ ------------
            Households                                          All
            with number                                         Households
            of Veterans                                         
            in the                                              
            Household                                           
            equal to                                            

            0            1            2            3 or more    

  Count     54,304       17,936       909          61           73,210

  Percent   74.2         24.5         1.2          0.1          100.0
  --------- ------------ ------------ ------------ ------------ ------------

Table 8-9 also shows that 17,936 of the 18,906 veteran households (94.9
percent) had one potential veteran, while 4.8 percent of the veteran
households contained two potential veterans. Only 61 households (0.3
percent of the veteran households) had more than two potential veterans.

# 8.5 RDD Sample Extended Interview Results and Response Rates

From the 18,906 RDD Sample screened household with potential veterans,
we identified 19,950 potential veterans. Nearly two-thirds of these
(12,956 out of 19,950 or 64.9 percent) completed the extended interview.
Only 3.1 percent were eligible nonrespondents and 11.5 percent were out
of scope (ineligible) for the survey because they were not actually
veterans. The proportion with eligibility unknown was 20.5 percent.

The RDD Sample extended interview response rate, at 76.4 percent, was
better than that for the List Sample (62.8 percent).[^11] In part this
is because a relatively higher proportion of List Sample veterans could
not be located, which also meant they could not be contacted. The
overall RDD Sample (screener and extended interview) response rate was
51.6 percent.[^12] Of the RDD Sample respondents, 64.9 percent were
eligible veterans who completed the interview and 11.5 percent responded
to the extended interview eligibility questions but were determined not
to be veterans. Of the List Sample respondents, 54.0 percent were
eligible veterans who completed the interview and 8.8 percent were
determined not to be eligible. We could not determine eligibility for
20.5 percent of the screened RDD Sample cases or for 34.0 percent of the
List Sample cases. The main reason for the higher proportion of List
Sample cases with unknown eligibility was, again, our inability to
locate them. This was not an issue for RDD Sample cases because contact
was made during the household screening interview. There were no notable
differences in the proportion of eligible nonrespondents between the RDD
and List Samples (3.1 percent for the RDD Sample versus 3.2 percent for
the List Sample). In the following sections, we compare the
distributions of response status categories across age groups (Table
8-10), gender (Table 8‑11) and census regions (Table 8-12).

Table 8-10. Distribution of response status by age groups (RDD Sample)

+-------------+-------------+-------------+-------------+-------------+
| Age Group   | Eligible\   | Eligible\   | Ineligible  | E           |
|             | Respondent  | No          |             | ligibility\ |
|             |             | nrespondent |             | Unknown     |
+-------------+-------------+-------------+-------------+-------------+
| > Less than | 59.1        | 2.9         | 17.0        | 21.0        |
| > 50        |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| > 50-64     | 63.6        | 2.6         | 12.3        | 21.5        |
+-------------+-------------+-------------+-------------+-------------+
| > Over 64   | 70.9        | 3.7         | 6.5         | 18.9        |
+-------------+-------------+-------------+-------------+-------------+
| > TOTAL RDD | 64.9        | 3.1         | 11.5        | 20.5        |
+-------------+-------------+-------------+-------------+-------------+

Table 8-11. Distribution of response status by gender (RDD Sample)

+-------------+-------------+-------------+-------------+-------------+
| Gender      | Eligible\   | Eligible\   | Ineligible  | E           |
|             | Respondent  | No          |             | ligibility\ |
|             |             | nrespondent |             | Unknown     |
+-------------+-------------+-------------+-------------+-------------+
| > Male      | 65.4        | 3.1         | 11.0        | 20.5        |
+-------------+-------------+-------------+-------------+-------------+
| > Female    | 58.1        | 2.8         | 19.6        | 19.5        |
+-------------+-------------+-------------+-------------+-------------+
| > TOTAL RDD | 64.9        | 3.1         | 11.5        | 20.5        |
+-------------+-------------+-------------+-------------+-------------+

Table 8-12. Distribution of response status by census region (RDD
Sample)

+-------------+-------------+-------------+-------------+-------------+
| Census      | Eligible\   | Eligible\   | Ineligible  | E           |
| Region      | Respondent  | No          |             | ligibility\ |
|             |             | nrespondent |             | Unknown     |
+-------------+-------------+-------------+-------------+-------------+
| > Northeast | 64.7        | 3.5         | 11.8        | 20.0        |
+-------------+-------------+-------------+-------------+-------------+
| > Midwest   | 65.6        | 2.9         | 11.5        | 20.0        |
+-------------+-------------+-------------+-------------+-------------+
| > South     | 63.7        | 3.1         | 11.9        | 21.3        |
+-------------+-------------+-------------+-------------+-------------+
| > West      | 66.7        | 2.7         | 10.8        | 19.8        |
+-------------+-------------+-------------+-------------+-------------+
| > TOTAL RDD | 64.9        | 3.1         | 11.5        | 20.5        |
+-------------+-------------+-------------+-------------+-------------+

Table 8-10 shows that the ineligibility rate decreased monotonically
with age. That is, the proportion of potential veterans identified
during the screening interview that were not actually veterans decreased
with age. The proportion of elderly veterans (over 64) that did not
respond to the extended interview was also higher than in the other two
age groups.

The ineligibility rate was also higher for potential female veterans
(Table 8-11). Household members tended to be more uncertain about
identifying potential female veterans during the screening interview
than they were about identifying potential male veterans. There was no
significant difference in the distribution of response status across
census regions (Table 8-12).

# 8.6 List Sample Location Rates

We located 73.6 percent of List Sample veterans. By definition, location
rates for the response categories eligible respondent, ineligible and
eligible nonrespondents were 100 percent. The location rate for veterans
of unknown eligibility was 22.2 percent. Although we located these
veterans, we assigned them to the eligibility unknown category because
they did not provide the information necessary to determine their
eligibility. Table 8-13 gives the location rate by three age groups
(less than 50, 50 to 64, over 64). We found it more difficult to locate
younger veterans than older ones, perhaps because younger people in
general tend to be more mobile or less likely to have a telephone or
address listing in their name.

Table 8-13. List Sample location rates by age group

+-----------------------+----------------------------------------------+
| Age Group             | Location Rate (Percent)                      |
+-----------------------+----------------------------------------------+
| > Less than 50        | 58.6                                         |
+-----------------------+----------------------------------------------+
| > 50-64               | 72.5                                         |
+-----------------------+----------------------------------------------+
| > Over 64             | 83.9                                         |
+-----------------------+----------------------------------------------+
| > TOTAL LIST          | 73.6                                         |
+-----------------------+----------------------------------------------+

The location rate was also higher for male veterans than for female
veterans (74.3 percent for male veterans versus 66.1 percent for female
veterans). We may have had difficulty locating female veterans because
they are less likely to have a telephone or address listing in their own
or maiden names.

Table 8-14 gives the List Sample location rates by census region. Our
location rate was highest in the Midwest region and somewhat lower in
the West.

Table 8-14. List Sample location rates by census region

+--------------------------+-------------------------------------------+
| Census Region            | Location Rate (Percent)                   |
+--------------------------+-------------------------------------------+
| > Northeast              | 74.1                                      |
+--------------------------+-------------------------------------------+
| > Midwest                | 77.4                                      |
+--------------------------+-------------------------------------------+
| > South                  | 73.6                                      |
+--------------------------+-------------------------------------------+
| > West                   | 69.7                                      |
+--------------------------+-------------------------------------------+
| > TOTAL LIST             | 73.6                                      |
+--------------------------+-------------------------------------------+

#  8.7 Cooperation Rates

Survey cooperation rates are one indicator of interviewers' ability to
complete an interview after contact has been established with sampled
individuals. They also demonstrate the effectiveness of interviewer
training. The calculation for cooperation rates uses completed
interviews and refusals only. Included with the completed interview
cases are all ineligible (out-of-scope) cases (as determined by
completion of the required screening questions ). We calculated
cooperation at two stages: at the initial attempt to complete an
interview and after making all attempts to persuade those who initially
refused to participate in the survey (refusal conversion). Table 8-15
shows the number of initial refusal cases, number of refusal conversion
cases, and the refusal conversion rates by sample type. One-third of
initial refusals at the RDD Sample screening interview were converted to
completed interviews while more than 80 percent of initial refusals at
the RDD and List Sample extended interview level were converted.

Table 8-15. Refusal conversion rates by sample type

+----------------+-----------------+-----------------+-----------------+
| Sample Type    | Initial         | Refusal         | Refusal         |
|                | Refusals        | Conversions     | Conversion Rate |
+----------------+-----------------+-----------------+-----------------+
| > RDD Screener | 54,781          | 18,072          | 33.0            |
+----------------+-----------------+-----------------+-----------------+
| > RDD Extended | 1,935           | 1,566           | 80.9            |
+----------------+-----------------+-----------------+-----------------+
| > List         | 983             | 807             | 82.1            |
| > Extended     |                 |                 |                 |
+----------------+-----------------+-----------------+-----------------+

Table 8-16 presents the initial cooperation rates, the conversion rates
for those who initially refused, and the net cooperation rates after
both stages for the RDD Sample screening interview, and RDD and List
Sample extended interview. The percentages are based only on those cases
for which a telephone interview was attempted.

Table 8-16. Cooperation rate by sample type

+-----------------+-----------------+-----------------+-----------------+
| Sample Type     | Initial\        | Refusal\        | Final\          |
|                 | Cooperation     | Conversion Rate | Cooperation     |
|                 | Rate            |                 | Rate            |
+-----------------+-----------------+-----------------+-----------------+
| > RDD Screener  | 50.1            | 33.0            | 66.5            |
+-----------------+-----------------+-----------------+-----------------+
| > RDD Extended  | 86.5            | 80.9            | 96.4            |
+-----------------+-----------------+-----------------+-----------------+
| > List Extended | 85.8            | 82.1            | 95.1            |
+-----------------+-----------------+-----------------+-----------------+

The RDD screening interview had an initial cooperation rate of 50.1
percent, a conversion rate of 33.0 percent and a net cooperation rate of
66.5 percent. Among individuals selected for the RDD extended interview,
the initial cooperation rate was 86.5 percent, the conversion rate was
80.9 percent, and the net cooperation rate was 96.4 percent.

The List Sample veterans' initial cooperation rate, at 85.8 percent, was
slightly lower than that of the RDD Sample. On the other hand, the List
Sample extended interview refusal conversion rate of 82.1 percent was
slightly higher than the RDD Sample rate. The List Sample net
cooperation rate (95.1 percent) for the extended interview was slightly
lower than the RDD Sample extended interview net cooperation rate (96.4
percent).

# 8.8 Questionnaire Administration Timing

Table 8-17 presents selected statistics about the amount of time it took
veterans to complete the screening and extended interviews. The average
and median administration times for the RDD screening interview were 4.1
and 3.8 minutes, respectively. The average administration time is higher
than the median time because the distribution is skewed to the right.
List Sample veterans took an average of 38.7 minutes to complete the
extended interview, while the RDD Sample took 33.3 minutes. It is not
surprising that List Sample veterans took longer, since they generally
have more medical conditions and medical treatment experiences to
report, and those sections of the interview were the most time-consuming
to complete. Table 8-17 also shows that the median time for the List
Sample extended interviews was 36.1 minutes as compared with 31.3
minutes for the RDD Sample extended interviews.

Table 8-17. Questionnaire administration time in minutes by sample type

  --------------- ----------- --------- ---------- --------- ---------- ---------
  Sample Type     Number      Mean      Lower      Median    Upper      Maximum
                  Completed             Quartile             Quartile   

  RDD Screener    18,906      4.1       3.3        3.8       4.5        27.2

  RDD Extended    12,956      33.3      26.6       31.3      37.6       136.5

  List Extended   7,092       38.7      30.7       36.1      43.8       137.6
  --------------- ----------- --------- ---------- --------- ---------- ---------

The upper and lower quartile values of interview administration time and
the maximum value are also shown in Table 8-17 for the RDD screener, RDD
and List Sample extended interviews. Seventy five percent of the RDD
screener interviews were completed in less than four-and-half minutes.
While 75 percent of the RDD extended interviews were completed in less
than 38 minutes, the comparable figure for the List Sample extended
interviews was almost three-quarters of an hour. Although a few extended
interviews took over two hours, 95 percent of the extended interviews
took under 52 minutes for the RDD Sample and just over an hour for the
List Sample. Overall (both RDD and List Sample cases), 654 extended
interviews took more than an hour. The NSV 2001 cooperation rates in
themselves are testimony to the sampled veterans' support of the survey
effort. The fact that 654 veterans were willing to devote over an hour
of their time to the interview further substantiates that support.

# 8.9 Telephone Call Statistics

Table 8-18 presents dialing statistics for completed interviews. It
shows, for each sample type, how many calls had to be made to complete
an interview. Overall, just over 30 percent of the RDD screener cases
were completed in one call and almost half of these cases were completed
in two calls. More than 95 percent of the RDD screener cases were
completed within 12 calls. The average number of calls per screened
household was 3.9 calls.

Over 48 percent of the RDD extended interviews were completed in one
call and just over 23 percent of the List Sample extended interviews
were completed in one call. Almost 62 percent of extended interviews
were completed within 2 calls for the RDD Sample, and just over 40
percent of List Sample extended interviews were completed within 2
calls. When the count rises to 3 calls, there was still a significant
difference in the proportion of completed extended interviews by sample
type, at 70 percent for the RDD Sample and 53 percent for the List
Sample. After five calls, the completion rates for the RDD and List
Sample cases were 80 percent and 71 percent, respectively. The average
number of calls per completed extended interview was 3.6 for the RDD
Sample and 4.9 for the List Sample. It took fewer calls to recontact the
RDD Sample cases, for which contact had already been established at the
screening interview stage, than the List Sample cases, where the
extended interview was our first contact. For the combined RDD and List
Samples, 91 percent of the extended interviews were completed within 10
calls, 94 percent within 12 calls, and 96 percent within 14 calls.

Table 8-18. Number of completed cases by number of calls for each sample
type

  ------------ ---------- --------- ---------- --------- ---------- ---------
  Number of    Sample                                               
  Call         Type                                                 

               RDD                  RDD                  List       
               Screener             Extended             Extended   

               Number     Percent   Number     Percent   Number     Percent

  1            5,771      30.5      6,243      48.2      1,654      23.3

  2            3,609      19.1      1,734      13.4      1,201      16.9

  3            2,594      13.7      1,041      8.0       884        12.5

  4            1,727      9.1       751        5.8       731        10.3

  5            1,264      6.7       592        4.6       529        7.5

  6            870        4.6       471        3.6       353        5.0

  7            641        3.4       359        2.8       301        4.2

  8            523        2.8       291        2.2       250        3.5

  9            379        2.0       255        2.0       214        3.0

  10           303        1.6       176        1.4       141        2.0

  11           244        1.3       171        1.3       149        2.1

  12           190        1.0       140        1.1       127        1.8

  13           154        0.8       131        1.0       109        1.5

  14           116        0.6       124        1.0       79         1.1

  15 or more   521        2.8       477        3.7       370        5.2

  Total        18,906     100.0     12,956     100.0     7,092      100.0
  Completed                                                         
  ------------ ---------- --------- ---------- --------- ---------- ---------

references

Bankier, M.D. (1988). Power Allocations: Determining Sample Sizes for
Sub-national Areas. *American Statistician*, Vol. 42, pp. 174-177.

Brick, J.M., Waksburg, J., Kulp, D.W., and Starer, A.W. (1995). Bias in
List-assisted Telephone Samples. *Public Opinion Quarterly,* Vol. 59,
pp. 218-235.

Casady, R.J., and Lepkowski, J.M. (1991). Optimal Allocation for
Stratified Telephone Survey Designs. *Proceedings of the American
Statistical Association,* Section on Survey Research Methods, pp.
111-116.

Casady, R.J., and Lepkowski, J.M. (1993). Stratified Telephone Survey
Designs. *Survey Methodology,* Vol. 19, pp. 103-113.

Deming, W.E., and Stephan, F.F. (1940). On a Least Square Adjustment of
a Sampled Frequency Table when the Expected Marginal Totals are Known,
*Annals of Mathematical Statistics,* Vol. 11, pp. 427-444.

Hartley, H.O. (1962). Multiple Frame Surveys. *Proceedings of the Social
Statistics Section of the ASA Meetings,* pp. 203-206.

Kish, L. (1965). *Survey Sampling*. John Wiley & Sons: New York.

Kish, L. (1988). Multipurpose Sample Designs. *Survey Methodology,* Vol.
14, pp. 19-32.

Lepkowski, J. M. (1988). Telephone Sampling Methods in the United
States. *Telephone Survey Methodology* (Eds. Groves, R. M., Biemer, P.
P., Lyberg, L. E., Massey, J. T., Nicholls II, W. L., and Waksberg, J).
New York: Wiley, pp. 73-98.

Potter, F.J., McNeill, J.J., Williams, S.R. and Waitman, M.A. (1991).
List-assisted RDD Surveys. *Proceedings of the American Statistical
Association,* Section on Survey Research Methods, pp. 117-122.

Westat. (April 22, 1999). *1992 National Survey of Veterans Utilization
Study* (Prepared under contract to the U.S. Department of Veterans
Affairs). Rockville, MD: Westat.

[^1]: See 1992 National Survey of Veterans Utilization Study (Westat,
    1999), for more detailed information about study methodology and
    results.

[^2]: The term "service-connected" refers to a VA decision that the
    veteran's illness or injury was incurred in, or aggravated by,
    military service.

[^3]: The 643 exchange numbers were obtained from the "Terminating Point
    Master Vertical & Horizontal Coordinates Data (abbreviated as
    TPM-VHCD)" file provided by Telcordia Technologies. As of December
    7, 2000, 325 of them were associated with a regular
    ([P]{.underline}lain [O]{.underline}ld) [T]{.underline}elephone
    [S]{.underline}ervice (abbreviated as POTS). No such numbers shared
    with paging for Puerto Rico.

[^4]: This is a character field in the TPM-VHCD file, which identifies
    the general location or service of each area code and is used by
    many customer-billing processes to appear on bills. Refer to
    TPM-VHCD -- Data Set/File Specification, Appendix A-1, Page 5.

[^5]: According to Lepkowski (1988, p. 83), fewer than 25% of all
    potential telephone numbers (generated by appending 4-digit random
    numbers to known area-prefix combinations) are assigned to a
    household.

[^6]: The Vetpop 2000 is a veteran population projection model developed
    by the office of the Actuary, Department of Veterans Affairs. It is
    the official VA estimate and projection of the number and
    characteristics of veterans as of September 30, 2000. Details of all
    aspects of the development and content of the model are available
    from the office of the Actuary, Department of Veterans Affairs, 810
    Vermont Avenue NW, Washington DC 20420.

[^7]: The data collection field period for the survey was February
    through November 2001. Nearly all of the survey items that address
    use or nonuse of VA Health Care Services use a reference period of
    "during the past 12 months," and individual and household income
    questions are for the year 2000.

[^8]: WesVar is software for analyzing data from complex surveys. The
    software was developed by Westat and can be downloaded from Westat's
    website ([www.westat.com/wesvar](http://www.westat.com/wesvar)) for
    a 30-day free trial.

[^9]: A result code is the disposition that reflects the case's entire
    call history, as well as the status of that case at the time the
    last contact was made.

[^10]: RDD screener response rate = (eligible respondents + ineligible
    respondents)/total dialed telephone numbers.

[^11]: Extended interview response rate = eligible respondents +
    ineligible respondents/total \[List or RDD\] Sample.

[^12]: Overall RDD Sample response rate = screener response rate \*
    extended interview response rate.