United States General Accounting Office

GAO Report to the Ranking Minority Member,

Committee on Science, House of

Representatives

June 1995 NUTRITION

MONITORING

Data Serve Many

Purposes; Users

Recommend

Improvements

GAO/PEMD-95-15

GAO United States

General Accounting Office

Washington, D.C. 20548

Program Evaluation and

Methodology Division

B-260632 

June 20, 1995

The Honorable George E. Brown, Jr.

Ranking Minority Member

Committee on Science

House of Representatives

Dear Mr. Brown:

The National Nutrition Monitoring and Related Research Program

(NNMRRP) is a network of surveys, surveillance systems, and research

activities designed to serve multiple purposes. It provides researchers and

decisionmakers with data for assessing the safety of the nation’s food

supply, targeting food assistance to low-income families, and studying the

relationship between diet and disease, among other uses. However, past

evaluations of federal nutrition monitoring have criticized it on several

counts, including the lack of coordination among the various activities and

its poor coverage of populations at risk of nutritional problems. Through

the National Nutrition Monitoring and Related Research Act of 1990 (P.L.

101-445), the Congress established objectives for addressing these

problems.

This report is the second in a series of three responding to your request for

information on the NNMRRP. In our first report, published last year, we

detailed the activities that make up the NNMRRP, the history of concerns

about the data collection systems, and agency progress toward meeting

the objectives of the NNMRR Act.1

 In this report, we summarize the results

of our survey of users of nutrition monitoring data. Specifically, the

objectives of this report are to (1) describe users and major uses of

nutrition monitoring data and (2) summarize the satisfaction of users with

selected nutrition monitoring activities and the changes that users

identified as likely to increase their use of or confidence in the data. The

survey results presented here serve as a foundation for our follow-up

report on the features of a model nutrition monitoring program.2

Results in Brief The data users who responded to our survey were located in a variety of

settings, including governmental, academic, and business. These users

reported that they provide analyses to the general public as well as to

1

Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23; May 27,

1994).

2

See GAO/PEMD-95-19, forthcoming.

Page 1 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

decisionmakers in their organizational settings and to other audiences.

They also reported using nutrition monitoring data for an extensive range

of purposes, from identifying nutrition-related problems and designing

programs to address the problems to informing basic research.

Although most of the data users who responded to our survey were

satisfied with the degree to which the data meet their information and data

quality needs, a majority also suggested changes that would increase their

use of or confidence in the data. Their recommendations include the need

for improved dietary intake methods, more continuous data collection,

better coverage of subpopulations and small geographic areas, improved

timeliness and documentation of the data, and increased dissemination of

the data in formats that facilitate access and analysis.

Background The U.S. nutrition monitoring system has included more than 70 separate

data collection activities conducted by several different federal agencies.

Major components of the system include the national health and nutrition

surveys administered by the National Center for Health Statistics (NCHS)

within the Centers for Disease Control and Prevention (CDC), the

state-based surveillance systems managed by the National Center for

Chronic Disease Prevention and Health Promotion (NCCDPHP), and national

surveys operated by the Agricultural Research Service (ARS). Table 1 lists

the data collection activities addressed in our survey.

Although the system has been praised for being comprehensive, it has also

been criticized for the redundancy of some of the monitoring activities, the

prolonged data collection and delays in data release, the poor coverage of

subpopulations, and the lack of compatibility in data assessment and

sampling methods across different surveys.3

 In response to these

concerns, the National Nutrition Monitoring and Related Research Act of

1990 required the Secretaries of the Departments of Agriculture (USDA) and

Health and Human Services (HHS) to implement a coordinated program of

nutrition monitoring and established an Interagency Board to facilitate the

process. The Board developed preliminary plans for meeting the goals of

the legislation and published them in a 10-year comprehensive plan in

June 1993.4

3

See GAO/PEMD-94-23 for a discussion of these concerns and the NNMRRP activities intended to

address them.

4

Ten-Year Comprehensive Plan for the National Nutrition Monitoring and Related Research Program,

58 Fed. Reg. 111 (June 11, 1993), pp. 32752-806.

Page 2 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

Table 1: Data Collection Systems in Our Surveya

Agency Data collection system Time period covered Short name

HHS/PHS/CDC/

NCHS

National Health and Nutrition Examination Survey I:

Epidemiological Follow-up

1982-84, 1986, 1987,

1992

NHEFS

National Health and Nutrition Examination Survey II 1976-80 NHANES II

National Health and Nutrition Examination Survey III 1988-94 NHANES III

Hispanic Health and Nutrition Examination Survey 1982-84 HHANES

National Health Interview Survey Supplement on Vitamin

and Mineral Supplements

1986 NHIS-Vitamin

National Health Interview Survey Supplement on Cancer

Epidemiology and Cancer Control

1987, 1992 NHIS-Cancer

HHS/PHS/CDC/

NCCDPHP

Behavioral Risk Factor Surveillance System Continuously since

1984

BRFSS

Pregnancy Nutrition Surveillance System Continuously since

1988

PNSS

Pediatric Nutrition Surveillance System Continuously since

1973

PedNSS

HHS/PHS/IHS Navajo Health and Nutrition Survey 1991, 1992 Navajo HNS

HHS/PHS/FDA Health and Diet Survey Biannually since 1982 Health and Diet

USDA/ARS Nationwide Food Consumption Survey Every 10 years since

1936, 1977-78,

1987-88

NFCS

Continuing Survey of Food Intake by Individuals 1985-86, 1989-91 CSFII

Diet and Health Knowledge Survey 1989, 1990, 1991,

1993

DHKS

aFor more information on the scope and design of these systems, see Directory of Federal and

State Nutrition Monitoring Activities, prepared by the Interagency Board for Nutrition Monitoring

and Related Research, HHS Pub. No. (PHS) 92-1255-1 (1992).

Page 3 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

Scope and

Methodology

To obtain information on the current uses of nutrition monitoring data and

identify the kinds of changes that are needed to increase the utility of the

data, we conducted a survey of potential users of nutrition monitoring

data. The survey focused on the 14 NNMRRP data collection activities listed

in table 1, selected because they are major activities or because they

addressed a major concern, such as the need for data on subpopulations.

These activities collect three kinds of nutrition data: nutritional and health

status; food consumption and dietary intake; and dietary knowledge,

attitudes, and behavior. Two other areas of nutrition monitoring—food

composition and food supply and demand—were not addressed by the

survey.

Because we found no comprehensive list of people who use these data, we

developed a complex sampling plan to obtain lists of potential users from

a variety of sources.5

 Our focus was on obtaining the views of primary data

users, defined as those who have conducted analyses in the past 5 years

rather than relied on information already processed and interpreted by

others. We limited our focus to these users because we expected them to

have a greater familiarity with the strengths and limitations of each data

collection system. (The data collection and sampling design are detailed in

appendix I.)

Our sample design cast a wide net with the intention of obtaining

information from a variety of users. However, because we aggregated

samples of different sizes from multiple lists, the survey results cannot be

used to characterize the average user in general or the typical user in each

of the organizational settings. Moreover, we asked users to consider their

experiences with individual data systems only, rather than with the NNMRRP

as a whole. Users first identified which of the 14 data collection activities

they had used in the last 5 years and then focused on the two they used

most frequently.

We conducted our review between December 1993 and December 1994 in

accordance with generally accepted government auditing standards.

5

Out of a total of 1,585 potential data users sampled from the lists, 1,180 (or 74.5 percent) responded;

an industry group helped us find another 10 respondents. Among all respondents, 93 provided

insufficient information and 123 pooled their responses with those of another respondent. Among the

rest of the respondents, 344 were not users of nutrition monitoring data, 190 were secondary users

(using information that had already been analyzed), and 440 were primary users.

Page 4 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

Principal Findings

Data Are Used in a Variety

of Settings for Multiple

Purposes

The 440 primary users who responded to our survey worked in federal,

state, and local government; academic institutions; for-profit businesses,

such as food industries; and other settings, such as hospitals. These users

also represented a variety of occupations. While data users in federal or

academic settings were more likely to identify themselves as engaged in

basic research, those in state or local settings were more likely to indicate

program planning and management as their primary occupation. (As noted

above, these data users who responded to our survey are not necessarily

representative of users in general or of the users in each organizational

setting.)

Some use of almost every data collection system was reported in each of

the organizational settings. (The exception is the Navajo HNS, which was

reported as used by only a small number of respondents in the federal

government and “other” category.) While state and local government

respondents were more likely to use the state-based surveillance systems,

federal government and academic respondents were more likely to report

using the national surveys. (Appendix IV presents more information on

users, the data sets they use, and their organizational setting.)

Across the different settings, the nutrition monitoring data supported a

variety of uses from identifying nutritional problems to planning programs

to address the problems, evaluating food-and nutrition-related programs

and policies, informing basic and methodological research, and supporting

state and local surveillance activities. Table 2 provides specific examples

of the decisions respondents stated they made based on the data. (Tables

showing the percent of respondents indicating a specific purpose for

which they used a data collection system are provided in appendix II.)

Page 5 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

Table 2: Respondents’ Examples of Decisions Made Based on the Data

Category Reported use

Problem identification Determine prevalence of high blood cholesterol in U.S. adults

Calculate exposure estimates for regulatory issues involving food additives

Assess damage from Exxon Valdez oil spill

Policy-making or program planning Refocus on diabetes in minority populations by the American Diabetes

Association

Support goals and activities for improved nutrition status of population in

state cancer plan

Develop national guidelines for screening and management of

iron-deficiency anemia

Confirm need for addition of calcium to infant and toddler foods

Decide the size of target populations for new pharmaceutical products

Place breast-feeding coordinators in areas of greatest need

Policy or program management and evaluation Increase funding for Healthy Heart Programs

Modify year 2000 objectives for blood pressure to include Mexican

Americans

Document the need for use of iron-fortified infant formula and then

document the success of the policy implementation

Calculate sales tax consequences of cashing out food stamps

Conclude that children have too much fat in their diets, but the excess is

not caused by participation in child feeding programs

Research related to nutrition Implement a clinical trial to prevent diabetes through diet modification

Plan study of unusually high anemia levels in Alaskan Natives, which led

to new cause of iron-deficiency anemia (bacterial)

Determine which foods to include on a food-frequency questionnaire for

Puerto Rican elderly

Support of monitoring activities

by states and localities

Choose knowledge and attitude indicators for a state survey because

reference values from national surveys are available

Revise weighing and measuring policy to increase accuracy in clinics

Use data in community needs assessment for counties to develop plans

for services

The primary users who responded to the survey also identified the

customers for their analyses. As shown in table 3, users in each

organizational setting identified a range of end users of the data. In

general, customers in their own organizational setting were most

commonly indicated, but the general public was also frequently identified

as a customer for the primary users’ analyses of the nutrition data.

Page 6 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

Table 3: End Users of Most Frequently

Used Data Systems Organizational setting of primary data users

End users Federal State Local Academic Business Othera

Federal government 79% 14% 0 17% 22% 18%

State government 38 81 32% 19 14 23

Local government 26 80 75 7 3 23

Universities 55 50 11 70 14 40

Hospital or health care 28 50 25 17 16 50

Researchers 68 34 14 74 38 48

For-profit business 27 10 11 14 68 20

Nonprofit, noncharitable 26 49 18 15 16 45

Charitable organization 13 24 18 8 5 18

Media 43 40 21 18 22 30

General public 50 53 43 30 30 53

Otherb 9 10 11 7 11 8

aOther settings include hospitals, nonprofit organizations, and other charitable organizations.

bOther end users include trade associations, labor groups, tribal governments, community action

agencies, medical practitioners, minority groups, regulators, university students, and so on.

Despite General

Satisfaction, Users

Suggest Changes

The majority of the primary users responding to our survey reported that

the data collection systems meet their information and data quality needs

to at least a moderate extent. However, despite this and the evidence that

the data are used for a variety of purposes, a majority of respondents

stated that changes are needed to increase their confidence in or

substantially increase their use of the data. A somewhat higher proportion

of users of USDA data systems than of HHS systems indicated a need for

change. (Users’ satisfaction is summarized in table IV.3.)

We asked primary users to identify what changes are needed in the

systems that they use most frequently. Common themes in their comments

were

• continuous or more frequent data collection;

• more detailed information on racial, ethnic, and age groups;

• data that can support estimates for small geographic areas;

• improved timeliness and documentation of the data; and

• increased dissemination of the data in general and in formats that facilitate

access and analysis.

Page 7 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

Specific comments from the users that illustrate these themes are

provided in table 4. More detailed summaries are in appendix III.

Table 4: Comments Illustrating Users’ Suggestions for Change

Category Comment Data system

Frequency of data collection Consider sampling subsets of variables in NHANES and NFCS more

frequently and conducting full survey occasionally—i.e.,

every 5-10 years.

NHANES

NFCS could be done every 5 years since food availability changes

so much.

NFCS

Coverage of racial, ethnic, and age groups Include very young and old and enough minorities to make conclusions

about the different groups.

NHANES

Expand sample size to provide sufficient number of minority respondents. BRFSS

Need up-to-date analysis for all age groups. Had to use the different

databases because one did not provide all age groups.

NFCS

Coverage of geographic areas More specific regional coverage would be highly useful for assessing

the diffusion of dietary and other health behaviors.

NHANES

Geographic area coverage should be more specific to allow analysis

and interpretation of data for individual states.

CSFII

It would be valuable to be able to provide county-level data for use by

local health departments.

BRFSS

Timeliness and documentation Would like faster turnaround from CDC to states for annual PedNSS tables. PedNSS

If results could be published more frequently, it would help us

see how well interventions are working.

NHANES

Need more documentation, especially of what was done in the

survey, how it was done, and how the statistical analyses were done.

CSFII

Dissemination Data can be made more accessible and more timely using modern

technology and user-friendly systems.

NHANES

Put on CD-ROM, include software that facilitates use, establish bulletin

board with updates as new data become available.

DHKS

Conclusions The NNMRRP data systems provide an important resource, serving a wide

set of purposes in a variety of settings. Moreover, the data users are mostly

satisfied with the quality of the data and the degree to which their data

needs are met. Despite this evidence of satisfaction with the data systems,

those who responded to our survey had numerous suggestions for

improving the data collection activities of the NNMRRP. These suggestions

are consistent with many of the past criticisms of the nutrition monitoring

system. Understanding who uses the data and for what purposes is

essential to developing and implementing an effective nutrition monitoring

Page 8 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

system. Our study begins this process and provides a useful framework of

purposes for nutrition monitoring data.

Agency Comments We provided the Board and responsible agencies with summary survey

data so they could begin revising their data collection activities as we

continued with our analyses and prepared this report. A draft of this report

was then sent to USDA, HHS, and members of the National Nutrition

Monitoring Advisory Council for review and comment. USDA and HHS

provided written comments, which are included in appendixes VI and VII.

In general, officials from these Departments agreed with our principal

findings and conclusions. USDA noted that our survey results will be useful

as they plan future monitoring activities, and HHS indicated that our report

provides a good overview of the user survey. HHS officials, however,

thought that our report did not sufficiently describe all of the major

federal uses of nutrition monitoring data, and they provided further detail

about these uses. Both HHS and USDA also presented additional information

about actions taken that respond to concerns raised by survey

respondents regarding information and data quality needs. Technical

comments made by HHS and USDA officials and the members of the

Advisory Council that reviewed the report have been incorporated where

appropriate.

As arranged with your office, we will be sending copies of this report to

the Director of the Office of Management and Budget, the Interagency

Board on Nutrition Monitoring and Related Research, the agencies

Page 9 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
B-260632 

responsible for data collection, and to other interested parties. We will

also make copies available to others upon request. If you have any

questions or would like additional information, please call me at

(202) 512-3092. Major contributors to this report are listed in appendix

VIII.

Sincerely yours,

Kwai-Cheung Chan

Director of Program Evaluation

 in Physical Systems Areas

Page 10 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Page 11 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Contents

Letter 1

Appendix I 

Questionnaire Design,

Sampling Plan, and

Analysis Decisions

16

Appendix II 

Uses of Nutrition

Monitoring Data

23

Appendix III 

Summaries of the

Suggestions Made by

Users

29

Appendix IV 

Characteristics of the

Uses of Data

Collection Activities

39

Appendix V 

List of Experts

43

Appendix VI 

Comments From the

Department of

Agriculture

45

Appendix VII 

Comments From the

Public Health Service

57

Page 12 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Contents

Appendix VIII 

Major Contributors to

This Report

62

Tables Table 1: Data Collection Systems in Our Survey 3

Table 2: Respondents’ Examples of Decisions Made Based on the

Data

6

Table 3: End Users of Most Frequently Used Data Systems 7

Table 4: Comments Illustrating Users’ Suggestions for Change 8

Table I.1: Data Requesters Included in Our Sample 19

Table I.2: Additional Sources of Nutrition Data Users 20

Table I.3: Additional State and Local Nutrition Data Users 21

Table II.1: Two Most Frequently Used Data Collection Activities 23

Table II.2: Respondents’ Indication of Use of Data for Problem

Identification

24

Table II.3: Respondents’ Indication of Use of Data for

Policy-making or Program Planning

25

Table II.4: Respondents’ Indication of Use of Data for Program

Evaluation or Management

26

Table II.5: Respondents’ Indication of Use of Data for Research 27

Table II.6: Respondents’ Indication of Use of Data for Activities

by States and Localities

28

Table III.1: CDC’s NCCDPHP Surveillance Systems 29

Table III.2: User Suggestions for Improving PedNSS and PNSS 30

Table III.3: User Suggestions for Improving BRFSS 31

Table III.4: CDC’s NCHS Data Collection Activities 32

Table III.5: User Suggestions for Improving NCHS Data Collection

Systems

33

Table III.6: USDA’s Data Collection Activities 35

Table III.7: User Suggestions for Improving USDA Data Collection

Systems

37

Table IV.1: Main Occupation of Respondents 39

Table IV.2: Respondents’ Use of Data Systems Within the Past 5

Years

40

Table IV.3: Users’ Response to How Well the Current Data Meet

Their Information Needs

41

Table IV.4: Users’ Response to How Well the Current Data Meet

Quality Needs

41

Table IV.5: Users’ Response to Whether Changes Would Increase

Their Confidence in or Use of the Data

42

Page 13 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Contents

Abbreviations

ARS Agricultural Research Service

BRFSS Behavioral Rish Factor Surveillance Survey

CDC Centers for Disease Control and Prevention

CSFII Continuing Survey of Food Intake by Individuals

DHKS Diet and Health Knowledge Survey

FDA Food and Drug Administration

HHANES Hispanic Health and Nutrition Examination Survey

HHS Department of Health and Human Services

HNS Health and Nutrition Survey

NCCDPHP National Center for Chronic Disease Prevention and Health

Promotion

NCHS National Center for Health Statistics

NFCS Nationwide Food Consumption Survey

NHANES National Health and Nutrition Examination Survey II and

III

NHEFS National Health and and Nutrition Examination Survey I

NHIS National Health Interview Survey

NNMRR National Nutrition Monitoring and Related Research

NNMRRP National Nutrition Monitoring and Related Research

Program

NTIS National Technical Information Service

PedNSS Pediatric Nutrition Surveillance System

PHS Public Health Service

PNSS Pregnancy Nutrition Surveillance System

USDA U.S. Department of Agriculture

WIC Special Supplemental Food Program for Women, Infants,

and Children

Page 14 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Page 15 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

Questionnaire Design This section describes how we developed our questionnaire and provides

an overview of the questionnaire content.

Selecting Data Collection

Systems

Our survey queried respondents about only 14 of the approximately 70

data collection activities listed in the Directory of Federal and State

Nutrition Monitoring Activities. (See table 1 on p. 3.) All 14 systems met

our criteria of focusing on (1) dietary, nutritional, and health status;

(2) food consumption; or (3) dietary knowledge, attitudes, and behavior.

Information about the food composition databases or activities for

monitoring the food supply was not gathered. This allowed us to

concentrate on survey-based data collection activities.

An additional criterion was that the data collection system be an ongoing

program. For example, periodic surveys like the Nationwide Food

Consumption Survey were included, while one-shot evaluations of food

assistance programs were excluded. We made three exceptions to this

criterion. We chose the NHIS-Vitamin and Mineral Supplements and

NHIS-Cancer Epidemiology and Cancer Control because of their large size

(nearly 11,800 and 45,000 interviews, respectively). We also collected

information on the Navajo Health and Nutrition Survey because of the

need for data on subpopulation groups expressed in public comments to a

draft of the 10-year comprehensive plan.

Defining Users Most of the questions in the survey were directed only to primary users of

the data from the 14 selected activities. We defined a primary data user as

one who directly accesses these data. This includes those who request

analyses from others as well as those who access the data systems

themselves. In contrast, secondary users are those who use nutrition

monitoring information that has already been processed and interpreted

by others in reports, articles, publications, or other documents. We chose

this definition to target the questionnaire to respondents with firsthand

experience with the design and content of the data collection activities

and the strengths and limitations of the data.

Structuring the

Questionnaire

We sent our survey to both known and potential users. Primary users of

the data could not be identified in advance, so in the first section, we

screened out secondary users and nonusers of the 14 data collection

systems. Then we asked the remaining respondents—the primary data

Page 16 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

users—to identify the data collection system they used most frequently

and the next most frequently in the past 5 years.

A major portion of the survey was dedicated to determining how the

respondents used the data. Through literature reviews and a series of

expert panel meetings, we developed an inventory of the uses of federal

nutrition monitoring data. As shown in appendix II, specific uses were

categorized in five main areas: (1) problem identification,

(2) policy-making and program planning, (3) policy and program

evaluation and management, (4) research related to nutrition, and

(5) support of state and local nutrition monitoring activities. Respondents

also had the opportunity to record up to five additional purposes for which

they used the data. To ascertain the validity of the uses they identified, we

asked them to list at least one report, article, or other document produced

with the data.

We also obtained information on the extent to which the data collection

systems met the respondent’s information and data quality needs. We

asked whether changes are needed to better meet their needs for the data.

Of those indicating a need for changes, we asked for their suggestions on

improving the (1) data elements collected, (2) data collection methods,

(3) units of analysis, (4) time of data collection, (5) population group

coverage, (6) geographic area coverage, and (7) ease of use.

Sampling Plan This section details the sampling approach and provides information on

the sources from which we obtained names of people to survey.

Overview We had no way of identifying all the users of the federal nutrition

monitoring data, so we chose a nonrandom sampling approach to

maximize the heterogeneity of the individuals surveyed. From a variety of

sources, we obtained lists of known and potential users of the data and

also of contacts in organizations likely to contain data users.

We mailed out a total of 1,614 surveys. Addresses were incorrect for 29, so

the sample size was reduced to 1,585. We received 1,180 responses, or

74.5 percent. An additional 10 responses came from a confidential industry

mailing list, increasing our total responses to 1,190. Of those, 344 were

nonusers, 190 were secondary users, and 440 were primary users. In

addition, 123 indicated that their responses were included in with other

respondents, and 93 did not provide useful information because they were

Page 17 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

ill, retired, or failed to complete the questionnaire. Primary users tended to

respond early, and nonusers tended to respond only after one or two

follow-ups.

Identifying Users and

Potential Users

We selected users both as individuals likely to use nutrition monitoring

data and as members of organizations likely to contain one or more data

users. We asked the former to answer only for their own uses and the

latter to direct the survey to the most appropriate user within their

organization, who would also answer only for his or her own use. We did

not distinguish between individual and organizational respondents in our

analyses.

Individuals We identified actual and potential individual users of the 14 nutrition

monitoring activities from a variety of sources, including lists maintained

by federal agencies of people who had requested data, referrals of likely

users from the Interagency Board and other federal contacts, lists of

people attending workshops and conferences, and professional

association membership lists. Tables I.1-I.3 identify the sources of our lists

and the number of people surveyed.

Table I.1 provides the number individuals we surveyed, by data collection

activity, who requested data from the National Technical Information

Service (NTIS) or directly from the federal agencies administering any of

the 14 systems. The two largest groups were the NHANES III and Cancer

Risk Survey data requesters.

Page 18 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

Table I.1: Data Requesters Included in

Our Sample

Source agency Data system

Number

surveyed

USDA/HNIS CSFII 1989 7

CSFII 1990 1

NFCS 1987-88 7

NFCS (household) 5

NFCS (low income) 9

NFCS (household and individual intakes) 6

HHS/NCHS NHANES I 16

NHANES II 24

NHANES III 58

NHANES I Epidemiological Follow-up 13

NHANES I Follow-up Group Members 12

Hispanic HANES 7

NHANES (unspecified) 9

Cancer Risk Survey 42

Vitamin and Mineral Survey 5

HHS/FDA Health and Diet Survey 6

NTIS CSFII 10

Nutrient Data Base 8

NHANES I 2

NHANES II 9

NHANES I Epidemiological Follow-up 7

NHIS 10

NFCS 1977-78 6

NFCS 1987-88 10

To ensure that we had full coverage of federal government users, we asked

the Interagency Board for the names of directors of agency divisions

mentioned in the 10-year comprehensive plan. Through other referrals, we

added the names of 26 potential users within those agencies. From lists of

attendees at three federally-sponsored, nutrition-related workshops, we

identified another set of known or likely users that we surveyed. Members

of associations for nutrition professionals were another source of

potential users we surveyed. Finally, we obtained lists of local government

officials working in nutrition. (Table I.2 provides the source and

application of these additional potential users that we surveyed.)

Page 19 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

Table I.2: Additional Sources of

Nutrition Data Users

Source Affiliation

Number

surveyed

Likely federal users Division directors (list provided by the

Interagency Board)

23

Potential federal users identified through

referrals

26

Nutrition-related workshop

attendees

Dietary Consensus Conference 58

Food Insecurity Conference 61

USDA Household User Group 14

Association members American Dietetic Associationa 203

American Institute of Nutritionb 268

Society for Nutrition Educationc 107

Lists of local government

officials

CityMatCH members (Urban Maternal and

Child Health directors)

144

National Association of County Health

Officialsd

27

aFrom more than 65,000 members, we identified a subgroup of 2,030 employed in education and

research, and then we drew a 1-in-10 sample.

bFrom nearly 3,000 members, we drew a 1-in-10 sample.

cWe drew a 1-in-3 sample of members working in higher education, industry, public health, and

county extension education.

dFrom their National Directory of Local Health Departments, we chose the directors from 30

counties containing 1 million or more inhabitants, or 23.6 percent of the 1990 U.S. population

(according to the Census Bureau).

Organizations To capture any state and local officials we may have missed, we targeted

organizations that were likely to have at least one or more state and local

nutrition monitoring data users. We asked organizations to direct the

survey to the most appropriate or experienced officials, who would

respond only for their own use of the data, not for the organization as a

whole. (See table I.3.)

To cover the fragmented groups in nutrition research and policy analysis,

we built in some redundancy within the sampling plan. Respondents who

received more than one survey, however, were counted only once in our

analyses.

Page 20 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

Table I.3: Additional State and Local

Nutrition Data Users

Affiliation Respondents

Number

surveyed

Association of State and Territorial

Public Health Officials

Child health and nutrition officials 45

Chronic disease officials 42

Health education officials 53

Maternal and Child Health

Association

Maternal and child health directors 52

Center on Budget and Policy

Priorities

WIC directors 44

State Government Yellow Book Health department directors 52

Surveillance system contacts BRFSS 47

PedNSS 39

PNSS 18

Limitations of the

Sampling Plan

While our approach allowed us to cast a wide net and contact as many

data users as possible, it also has some limitations. The major one is our

inability to determine the degree to which the survey respondents are

representative of primary data users in general. With a nonprobability

sample, we cannot generalize beyond our respondents to the universe of

all users of the 14 data collection activities the questionnaire addresses. In

addition, we cannot make any inferences about the extent of use across

groups. Our ability to identify primary users within groups varied, so

differences in reported use may be a function of our sample design rather

than of actual differences in use. For example, we were able to target

federal users of the data more closely than users in other sectors, but it

would be inappropriate to compare the extent of their use to that by other

groups of respondents.

A further limitation of our survey design is that we asked respondents to

comment on individual data collection activities (such as DHKS) and not on

the specific survey components within each activity (such as the 1989,

1990, 1991, or 1992 DHKS). Some users who are familiar with only a specific

survey component and not all the survey components may have made

suggestions to us for changes that have already been addressed by the

agencies in later versions of the data collection activity.

Page 21 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix I 

Questionnaire Design, Sampling Plan, and

Analysis Decisions

Analysis Decisions

Construction of Analysis

Groups

Our six groups of primary users—federal, state, local, academic, business

(for-profit), and other—were constructed according to respondents’

self-reports. Healthcare (hospital, nursing home) was the dominant group

within the “other” category, which also included nonprofit businesses and

charitable organizations. Overall, the groups were diverse, and no single

subgroup dominated any group.

The federal group in our sample consisted of at least 23 different agencies.

Those with the largest number of respondents were the National Center

for Health Statistics (17) and the Human Nutrition Information Service

(11). They are responsible for the main data systems, and they provided an

insider’s view to the strengths and weaknesses of these systems. The state

group included 45 states that responded and the District of Columbia and

Guam. The largest number of respondents from one state was eight, or

6 percent, so no one state had a large influence on the whole group. The

local group had one user each from 28 different counties.

The academic group came from 67 different cities, and some cities, such as

Boston and Chicago, were represented by more than one school. The

largest number of respondents from one school was five, or 5 percent. Five

of the 37 in the business group did not give their addresses; the others

came from 26 different cities, and the largest number from one city was

three. The 40 respondents in the “other” group came from 33 cities, and

the largest number from one city was three.

Page 22 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix II 

Uses of Nutrition Monitoring Data

The data collected by the nutrition monitoring systems are not only used

across a variety of organizational settings, but they also support a range of

uses. With the assistance of our expert consultants, we developed an

inventory of the purposes that federal nutrition monitoring data serve.

Specific purposes were categorized in five overarching areas: (1) problem

identification, (2) policy-making and program planning, (3) policy and

program evaluation and management, (4) research related to nutrition, and

(5) support of state and local nutrition monitoring activities.

For the two data collection activities used most frequently (see table II.1),

respondents to our survey were asked to indicate the purposes the data

had served. The data users were also asked to write in specific decisions

that the data had supported. Although our respondents presented a variety

of purposes and decisions, one noted that the data were not timely

enough, or sufficiently on target, to truly inform decisions. We classified

the written comments into the five major categories of purposes.

Table II.1: Two Most Frequently Used

Data Collection Activities

Most Second most Total

Frequency of use

Data collection activity Number Percent Number Percent Number Percent

NHEFS 34 8 20 6 54 12

NHANES II 46 10 55 16 101 23

NHANES III 54 12 36 11 90 20

HHANES 11 2 17 5 28 6

NIHS-Vitamin 2 1 1 0 3 1

NHIS-Cancer 9 2 4 1 13 3

BRFSS 64 15 21 6 85 19

PNSS 29 7 44 13 73 17

PedNSS 68 15 28 8 96 22

Health and Diet 8 2 5 1 13 3

NFCS 69 16 41 12 110 25

CSFII 43 10 46 14 89 20

DHKS 3 1 20 6 23 5

Total 440 101a 338b 99a 440c

aBecause of rounding, total percentages do not add up to 100.

bSome respondents had used only one data system; thus, the number of the second most

frequently used system was less than 440.

cTotal sample; not column total.

Page 23 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix II 

Uses of Nutrition Monitoring Data

Problem Identification The expert panels distinguished seven kinds of problems that the data

might be used to identify. (See table II.2.) As shown in the table, problem

identification was a commonly indicated purpose supported by the data.

However, there is some variation in the kinds of problems examined by

the different data sets. For example, NHANES data—with its emphasis on

health—is used to examine chronic degenerative diseases, as well as

deviations in nutritional status.

Table II.2: Respondents’ Indication of

Use of Data for Problem Identification Purposea

Data collection activity 1234567

NHEFS 70% 48% 6% 52% 6% 4% 14%

NHANES II 49 45 11 55 9 2 10

NHANES III 55 47 21 57 6 5 14

HHANES 65 42 4 54 4 0 8

NHIS-Cancer 31 39 0 39 8 0 8

BRFSS 55 41 1 74 1 0 13

PNSS 14 86 15 80 0 1 6

PedNSS 17 88 16 80 0 1 5

Health and Diet 39 39 8 62 8 23 15

NFCS 29 44 22 56 18 11 43

CSFII 30 44 23 61 21 14 42

DHKS 36 27 23 77 32 23 27

Median percent 38 44 13 59 7 3 14

aFor each of the two nutrition monitoring data collection activities that they used most frequently,

respondents were asked to indicate if they had used the data to identify or estimate the risk,

incidence, prevalence, duration, or cost of any of the following problems:

1. Chronic degenerative diseases and their relationship to diet and nutritional status;

2. Nutritional deficiency diseases and health-related issues;

3. Hunger and food insecurity, including its relationship to diet and its periodicity;

4. Deviations in nutritional status (e.g., obesity) and diet quality across the life-cycle and across

population groups;

5. Food safety problems over which consumers have little control (e.g., contaminants);

6. Food safety problems over which consumers have some control (e.g., microbiological

problems resulting from food preparation, handling, or consumption activities); and

7. Other food quality problems (availability, accessibility, and composition).

Page 24 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix II 

Uses of Nutrition Monitoring Data

Policy-making or

Program Planning

Within the overarching objective of policy-making or program planning,

the expert panel identified seven purposes. (See table II.3). As shown in

the table, most of the program purposes reported were fairly common uses

of nutrition monitoring data sets. One exception was “to compare

cost-effectiveness” of different kinds of policy or program interventions,

which is not surprising since the nutrition data do not provide this

information. Many of the uses that respondents wrote in response to the

request for four specific decisions informed by the data appeared to fit

under policy-making or program planning.

Table II.3: Respondents’ Indication of

Use of Data for Policy-making or

Program Planning

Purposea

Data collection activity 1234567

NHEFS 51% 30% 42% 20% 2% 19% 42%

NHANES II 58 36 44 28 4 23 43

NHANES III 64 52 55 31 8 27 61

HHANES 58 46 46 36 4 27 54

NHIS-Cancer 46 31 31 31 0 15 77

BRFSS 72 58 47 53 4 46 78

PNSS 85 70 65 85 7 60 89

PedNSS 86 69 66 80 11 60 89

Health and Diet 77 54 54 46 8 39 54

NFCS 66 44 52 38 8 22 48

CSFII 76 51 57 40 6 23 51

DHKS 74 48 73 52 0 23 57

Median percent 69 50 53 39 5 25 56

aFor each of the two nutrition monitoring data collection activities that they used most frequently,

respondents were asked to write in four specific decisions informed by the data, many of which fit

within the following purposes:

1. Define and quantify the extent and distribution of a food- or nutrition-related problem or the

risk of the problem;

2. Assess the importance of a problem or risk of the problem relative to other problems;

3. Identify determinants of a food- or nutrition-related problem or risk of the problem;

4. Identify policy and programmatic responses to the problem or risk of the problem;

5. Compare cost-effectiveness of responses;

6. Justify the selection of a response (prevention, intervention to mitigate, or intervention to deal

with the consequences); and

7. Serve as a basis for targeting prevention or intervention resources or both.

Page 25 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix II 

Uses of Nutrition Monitoring Data

Program Evaluation

and Management

The inventory of purposes served by nutrition monitoring data included

nine program evaluation and management purposes. (See table II.4.) As

shown in the table, two of the more commonly indicated purposes in this

area were measuring changes in deficiency diseases and assessing

achievement of specific dietary objectives. For example, several of the

uses described by the survey respondents focused on monitoring or

modifying Healthy People 2000 objectives. In contrast, measuring changes

in food safety problems was one of the least common purposes indicated

in the survey.

Table II.4: Respondents’ Indication of

Use of Data for Program Evaluation or

Management

Purposea

Data collection activity 123456789

NHEFS 47% 8% 31% 35% 8% 33% 12% 15% 8%

NHANES II 37 8 23 43 7 46 14 22 13

NHANES III 46 13 33 42 7 50 13 28 18

HHANES 39 8 23 27 0 31 8 23 12

NHIS-Cancer 25 8 8 33 0 42 17 33 8

BRFSS 44 1 26 46 4 77 1 41 38

PNSS 16 7 63 66 1 78 3 40 64

PedNSS 20 7 75 73 0 79 3 30 63

Health and Diet 33 0 46 39 8 77 25 69 46

NFCS 19 17 20 50 13 51 28 42 27

CSFII 19 17 19 55 13 57 26 48 25

DHKS 36 18 14 46 10 57 23 57 30

Median percent 35 8 24 45 7 54 14 37 26

aFor each of the two nutrition monitoring data collection activities that they used most frequently,

respondents were asked if they had used the data for the following:

1. Measure changes in chronic degenerative diseases, their relationship to diet and nutritional

status, and the risk of such diseases;

2. Measure changes in hunger and food insecurity;

3. Measure changes in deficiency diseases and health-related issues;

4. Measure changes in deviation in nutritional status and diet quality;

5. Measure changes in food safety problems;

6. Measure achievement of specific dietary objectives;

7. Evaluate food supply and nutrient supplements and fortificants;

8. Measure changes in food- and nutrition-related behaviors and their precursors and

determinants; and

9. Assess targeting and coverage of food- and nutrition-related programs.

Page 26 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix II 

Uses of Nutrition Monitoring Data

Research Related to

Nutrition

The expert panelists included seven research purposes for nutrition

monitoring data in the inventory. (See table II.5.) Across the different data

sets, respondents commonly indicated that the data were used to increase

basic research knowledge of the determinants of problems and options for

intervention.

Table II.5: Respondents’ Indication of

Use of Data for Research Purposea

Data collection activity 1234567

NHEFS 27% 14% 54% 44% 35% 19% 10%

NHANES II 29 26 50 48 33 22 9

NHANES III 40 31 53 45 36 21 7

HHANES 54 44 62 50 42 39 8

NHIS-Cancer 39 23 54 23 23 8 0

BRFSS 33 44 40 33 18 8 5

PNSS 41 62 52 40 33 13 3

PedNSS 35 51 44 35 32 13 4

Health and Diet 15 15 39 39 23 8 0

NFCS 28 23 43 44 27 36 23

CSFII 26 30 51 49 40 41 22

DHKS 27 18 64 68 36 46 19

Median percent 31 28 52 44 33 20 8

aFor each of the two nutrition monitoring data collection activities that they used most frequently,

respondents were asked to indicate if they had used the data for the following:

1. Improve sampling and statistical methods for gathering data from people at different

life-cycle stages or for minority or other subpopulations, especially those at risk of food- or

nutrition-related problems;

2. Improve methods for informing decisionmakers so analysis results are timely, pertinent, and

understandable;

3. Increase basic research knowledge of the determinants of problems and options for

intervention;

4. Increase basic research knowledge of the relationships between food, nutrition, and health;

5. Identify and stimulate needed research and development on monitoring methods;

6. Conduct other kinds of basic research (e.g., on the distribution of human nutrient

requirements, databases on food cost and food preparation, individual variability); and

7. Conduct food composition research and improve food composition databases.

Page 27 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix II 

Uses of Nutrition Monitoring Data

A few data users conduct food composition research; however, the food

composition databases were not included as a focus of the survey. Specific

research uses supported by the data included identifying foods for

food-frequency questionnaires, establishing cut-points for defining

research subjects, and developing survey instruments.

Support of Monitoring

Activities by States

and Localities

Under the general goal of supporting the monitoring activities of states

and localities, the expert panelists identified two specific purposes for the

nutrition monitoring data: supporting state and local surveillance activities

and supporting technical assistance. At least some portion of the users of

each of the data collection activities identified one of these purposes as a

way in which they use the data. (See table II.6.) Of the two, the latter was

more commonly indicated. This pattern also appears in the respondents’

comments, many of which focus on identifying the need for technical

assistance and improving the quality of data collection. (See appendix III.)

Table II.6: Respondents’ Indication of

Use of Data for Activities by States and

Localities

Purposea

Data collection activity 1 2

NHEFS 10% 10%

NHANES II 5 13

NHANES III 12 26

HHANES 8 19

NHIS-Cancer 8 15

BRFSS 12 54

PNSS 22 63

PedNSS 19 62

Health and Diet 15 15

NFCS 11 16

CSFII 8 16

DHKS 14 23

Median percent 12 18

aFor each of the two nutrition monitoring data collection activities that they used most frequently,

respondents were asked to indicate if they had used the data for the following:

1. Support state and local surveillance of and responses to food- and nutrition-related crises;

and

2. Support development and provision to states and localities of technical assistance in data

collection, analysis, and interpretation.

Page 28 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Systems Under

NCCDPHP

The National Center for Chronic Disease Prevention and Health Promotion

administers three surveillance systems that collect information on health

and nutritional status: the Pediatric Nutrition Surveillance System, the

Pregnancy Nutrition Surveillance System, and the Behavioral Risk Factor

Surveillance System. The three systems are overseen by the National

Center, but are actually implemented by the states that participate in the

surveillance programs. Table III.1 describes each system’s target

population and data collection methods.

Table III.1: CDC’s NCCDPHP Surveillance Systems

System Target population Type and source of sample Data collection method

PedNSS Low-income, high-risk children Participants in publicly-funded prenatal

nutrition and food assistance programs

Clinic staff record data at checkups

(body measurements, blood test

results, and demographic data)

PNSS Low-income, high-risk pregnant women Participants in publicly-funded prenatal

nutrition and food assistance programs

Clinic staff record health status, blood

test results, risk behaviors, and

demographic data

BRFSS Adults, age 18 and over Random telephoning of households Telephone interviews (body

measurements, risk behaviors, food

choices)

These surveillance systems vary in their purposes, methods of data

collection, and types of respondents, yet we found common themes in the

recommendations made by primary users of the systems. For all three

systems, users suggested providing

• more data on dietary intake,

• better controls on the quality of the data collected,

• more detail on subpopulation groups in the reporting of the data,

• increased ability to look at substate geographic divisions,

• improved timeliness of CDC’s return of the data,

• simplified reports that are more readily used at the local level, and

• additional technical and financial assistance in data collection and

interpretation.

User recommendations specific to each of the systems are presented in

the tables below.

Comments on PedNSS and

PNSS

Because PedNSS and PNSS collect data on several similar issues, they are

listed together in table III.2. In addition to these comments, some

respondents complimented CDC on the quality of PedNSS, specifically for the

Page 29 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

automated system and for its coordination of the data collection with the

WIC Program.

Table III.2: User Suggestions for Improving PedNSS and PNSS

Type of change Comment

Data elements Collect more data on

 Infant feeding practices, particularly breast-feeding or type of formula (and provide

 better analysis)

 Pregnancy risk information (PNSS)

 Dietary intake (food frequency, 7-day records, “usual” intake)

 Food security and hunger (PedNSS)

 Demographics

 Other indicators— blood lead levels, serum cholesterol, immunizations, height and

 weight at 2 and 3 years, household smoking (PedNSS) and physical activity and

 risk behaviors (PNSS)

Data collection methods Improve data collection quality control (training, uniform reporting, better software, and

standardization of measurements)

Use other sources of information (vital records, private physicians)

Maintain cultural sensitivity

Streamline and simplify questions

Stop changing the data requirements (PNSS only)

Use more biochemical measures

Develop methods to obtain data from more sources than public clinics (e.g., scannable

forms that private physicians could complete)

Units of analysis Maintain records by individual child, not by clinic visit (PedNSS only) to avoid

duplication of records

Time of data collection Facilitate analyses of changes over time by linking all records to the individual child

(PedNSS only)

Population group coverage Expand beyond participants in WIC and other publicly-funded programs to include

non-low-income women and children

Collect and report more data by subgroup (race, ethnicity, age, sex, income)

Geographic area coverage Enable reporting by substate divisions

Improve national estimates by including all states (currently, states choose whether

they participate)

Ease of use Provide

 Improved timeliness of reporting

 Simplified report format and content

 Reports that are more accessible for local users

 Improved flexibility of the PedNSS automated system and exportability of the data

 An automated system for PNSS that is similar to PedNSS

 Technical training and funding assistance to states to implement systems

Page 30 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Comments on BRFSS Data User recommendations for BRFSS are listed below in table III.3.

Table III.3: User Suggestions for Improving BRFSS

Type of change Comment

Data elements Collect more data on

 Dietary intake (in general and to link to behavior)

 Specific dietary elements (fat, food groups, fiber, nutritional supplements, alcohol, ethnic

 foods)

 Households (number of adult smokers)

 Ethnicity (state-specific)

Improve

 Questions on dietary fat to obtain a better measure

 Correspondence of health indicators with health objectives (percent of fat, salt intake, label

 reading)

Add questions on cholesterol, diabetes, and disease risk

Make questions more culturally sensitive and relevant

Data collection methods Address data validity and other quality control issues (translation for non-English-speakers,

applicability to adults in households without telephones, nonuniformity across states)

Develop a method to gather more complete dietary data

Units of analysis Maintain records by individual, with more data on the individual’s household

Time of data collection Collect nutrition data

 Continually or at least every 2 years

 At times most representative of year-round habits (make seasonal adjustments)

Population group coverage Provide more detail on

 Subpopulations in general (increase sample)

 Racial and ethnic groups (and those specific to a state)

 Specific age groups

 High-risk populations

Include populations without telephones

Geographic area coverage Improve national estimates by including all states

Increase sample sizes within states for better estimates

Provide information for substate divisions (counties, cities, rural areas), which will assist in

planning and evaluating community interventions

Ease of use Provide

 Improved timeliness of data (not only for state and local users, but also for researchers,

 who must obtain permission from each state)

 An automated system for state analyses of data

 Improved documentation

 Facilitated access to the data for nonstate users

 Technical assistance in data interpretation (especially dietary fat data)

Page 31 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Systems Under NCHS Users of four of the data collection activities run by the National Center

for Health Statistics commented on these systems in our survey: the

National Health and Nutrition Examination Survey, the NHANES I

Epidemiological Follow-up Study, the Hispanic Health and Nutrition

Examination Survey, and the National Health Interview Survey on Cancer

Epidemiology and Cancer Control. Table III.4 provides some summary

information about these surveys.1

Table III.4: CDC’s NCHS Data Collection Activities

Activity Target population Type and source of sample Data collection method

NHANES Civilian, noninstitutionalized population

age 2 months and older

Stratified, multistage, probability cluster

sample of households; oversampling of

children, elderly, African-Americans,

and Mexican-Americans

In-person interviews, including a single

24-hour recall and physical

examinations

NHEFS All persons between 25 and 74 years

old who completed a medical

examination at NHANES I in 1971-75

Same as for NHANES, with tracing of

age group of interest

In-person interviews, physical

measurements, review of hospital and

other records

HHANES Civilian, noninstitutionalized Hispanics

(Mexican- Americans, Cubans, Puerto

Ricans) age 6 months-74 years residing

in households in three defined U.S.

geographic areas

Stratified, multistage, probability cluster

sample of the target populations

In-person interviews and physical

examinations

NHISCancerCivilian, noninstitutionalized U.S.

population age 18 years and older

Stratified, multistage, cluster sample,

including one randomly selected

person 18 years or older in each NHIS

household; oversampling of Hispanics

and African-Americans in the last

implementation (1990)

In-person interviews

Our survey asked respondents to identify themselves as users of NHANES II,

which was conducted from 1976 to 1980, or NHANES III, which started in

1988 and was completed in 1994. (NHANES I was conducted between 1971

and 1975.) Because the surveys are very similar in their design, no

distinction is made between suggestions made by users of NHANES II and

those made by users of NHANES III in the discussion below. HHANES differs

from NHANES in its focus on three Hispanic subpopulations, but is

otherwise similar in methodology. The data collected by NHEFS, unlike that

for NHANES and HHANES, allow for the study of changes over time through

follow-up surveys (in 1982-84, 1986, 1987, and 1992) with all persons

between 25 and 74 years of age who had completed a medical examination

for NHANES I. All three use both food-frequency questions and 24-hour

recall to collect dietary intake data.

1

Our survey also asked respondents if they had used and had comments on a fifth NCHS survey—the

National Health Interview Survey on Vitamin and Mineral Supplements. Only three respondents

identified themselves as primary users of NHIS-Vitamin, and none of these had comments.

Page 32 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

In contrast, NHIS-Cancer relies solely on food-frequency questions. Even

though it is not one of the major nutritional data collection systems,

NHIS-Cancer was included in our survey because it measured nutrition

variables such as frequency of eating selected food items, vitamin and

mineral supplement intake, and knowledge of the relationship between

diet and cancer.

Although the target populations and methods used for the NCHS data

collection activities vary, some common themes emerged in the analysis of

the comments from the users of the different systems. Users suggested

providing

• more information on health habits and outcomes;

• more detailed data on food consumption;

• improved dietary intake methods, whether food-frequency questions or

24-hour recalls;

• a focus on the individual unit of analysis, with information linking the

individual to the family or household unit;

• continuous or more frequent data collection;

• more detailed information on racial, ethnic, and age groups;

• data that can support estimates for smaller geographic areas;

• improved timeliness and documentation of the data; and

• increased dissemination of the data in general and in formats that facilitate

access and analysis.

The specific comments made under these general themes and on other

subjects are detailed in table III.5.

Table III.5: User Suggestions for Improving NCHS Data Collection Systems

Type of change Comment

Data elements Collect more information on

 Health-related habits (physical activity, smoking, alcohol use)

 Medical history

 Health outcomes in general (arthritis, skin diseases, food allergies, cancer, and for elderly,

 hearing loss)

 Cause of death (NHIS-Cancer)

 Dietary intake

 Demographics (occupation as a source of nondietary exposure to cancer)

 Environmental risk factors

 Nonrespondents

(continued)

Page 33 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Type of change Comment

Data collection methods Conduct more research on data collection methods, in particular

 Measuring for race, ethnicity, and age (minorities, adolescents)

 Validating portion size (absolute amounts and percent of calories

 Using biochemical analyses (larger samples)

Improve automation and processing

Standardize techniques

Use multiple measures (telephone and in-person interviews and mailed questionnaires)

Expand the use of food-frequency questions

Include Hispanic foods and newer versions of common foods

Translate questionnaires for non-English-speaking persons and use fully bilingual interviewers

Obtain dietary data on more than a single day (multiple 24-hour data or 3-day records)

Units of analysis Retain individual as most important unit for nutrition issues (NHANES, HHANES, NHEFS)

Link individual data to family or household unit (NHIS-Cancer)

Account for non-Hispanics in household (HHANES)

Time of data collection Continuously collect nationally representative NHANES data while collecting subpopulation dataa

Shorten NHANES to conduct more frequently if not continuously (NHIS is a model for continuous

collection.)

Shorten cycles of surveys to produce more frequent updates

Conduct more methods research and data analysis between surveys

Increase frequency for nutritionally vulnerable groups

Conduct longitudinal follow-up on chronic diseases

Account for seasonality

Population group coverage Provide

 Better and more coverage of racial, ethnic, and age groups

 Clearer criteria on definition of race

 Comparable age-sex groups for racial and ethnic groups

 Generalizability (HHANES)

Geographic area coverage Provide

 More specific regional coverage

 More refinement of geographic detail (rural; urban; standard metropolitan statistical areas;

 Alaska, Hawaii, Puerto Rico, and Indian reservations)

Use small-area estimation models

Provide state-level estimates

(continued)

Page 34 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Type of change Comment

Ease of use Provide

 More timely release of data

 More and better documentation of complex sample designs

 On-line documentation Improved advertising of availability of different data

 Training in using complex sample designs

 Data in common statistical package format

 Occupation data coded for risk categories

 Anthropometric data using 15th and 85th percentiles as well as 25th and 75th

aOne of the criticisms of HHANES is that the data were collected at a different time from the

NHANES data, and thus, the health and nutritional status of the Hispanic groups cannot be

compared to that of the nation as a whole.

Systems Under USDA Our survey asked users of three USDA data collection activities to comment

on changes to the surveys that would increase their use of the data. The

three USDA surveys addressed are the Nationwide Food Consumption

Survey (now called the Household Food Consumption Survey), the

Continuing Survey of Food Intakes by Individuals, and the Diet and Health

Knowledge Survey. Table III.6 describes each activity’s target population

and data collection methods.

Table III.6: USDA’s Data Collection Activities

Activity Target population Type and source of sample Data collection method

NFCS Households in the 48 contiguous states

and individuals residing in those

households

Stratified, multistage, area probability

sample with oversampling for

low-income households

Personal interview with the

household food manager,

including a 7-day record of

household food use; personal

interview with household

members on dietary intake,

including 3 consecutive days of

dietary intake data collected

with one 24-hour recall and a

2-day record

CSFII Individuals in the 48 contiguous states Stratified, multistage, area probability

sample with oversampling for

individuals in low-income households

Personal interviews with

household members on dietary

intake, including 3 consecutive

days of dietary intake data

collected with one 24-hour recall

and a 2-day record

DHKS Main meal planner or preparer in

households that participated in CSFII

Same as CSFII Computer-assisted telephone

interviews (supplemented with

in-person interviews for

respondents without telephones)

Page 35 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

The focus of NFCS is on household use of food, including food costs, food

preparation, and food consumption. NFCS data are intended to inform

policies related to food production and marketing, food safety, food

assistance, and nutrition education. CSFII is intended to complement NFCS

in two ways. First, it provides a more frequent source of information than

the decennial NFCS, and second, it focuses on individual, rather than

household, food consumption. DHKS, a follow-up to CSFII, is intended to

support analyses of the relationship between dietary intake and

knowledge and attitudes about dietary guidance and food safety.

Although NFCS and CSFII vary in their target populations and purposes, they

are similar in their sampling approach (national with oversampling for

low-income population) and the focus on food consumption. Their

similarities are reflected in the common themes in the recommendations

made by the primary users of the two data collection systems. DHKS users

had somewhat different concerns about data elements and data collection

methods, but their comments were otherwise consistent with remarks

made about the other two systems.

The major themes in the comments about the data systems were to

provide

• more specificity and detail about foods and better data on food

composition;

• improved questions on dietary behavior;

• more information about health and demographic variables;

• reduced respondent burden and improved response rates;

• higher quality dietary recall data in general and, specifically, more

nonconsecutive days of 24-hour recall;

• individual data and information on the individual’s household;

• continuous or more frequent collection;

• longitudinal component;

• increased sample size and broadened coverage;

• more detail on racial, ethnic, age, and income groups;

• refined geographic area coverage, specifically state and substate data;

• more rapid release of the data;

• improved documentation; and

• dissemination of the data in alternative forms (for example, CD-ROM,

formatted for use with statistical packages).

Table III.7 provides more user suggestions and other issues from our

survey on USDA systems.

Page 36 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Table III.7: User Suggestions for Improving USDA Data Collection Systems

Type of change Comment

Data elements Collect more data on

 Food eaten away from home

 Food shopping access, prices, and behavior; food preparation methods and facilities; food

 storage; and safety

 Use of salt, condiments, nutritional supplements, specific foods (dairy, caffeine, water, fruits,

 processed, seafood)

Improve

 Quality and completeness of food composition data (newer products, brand names, reliability

 of data)

 Questions to assist linking diet and behavior (nutrition knowledge and opinion, exercise,

 barriers and motivation to change, participation in food programs)

 Health data (by measuring rather than self-reporting height, weight, health status)

Bring questions in line with current theory (DHKS); standardize questions from year to year

Data collection methods Streamline the instrument to reduce burden on respondents

Use automation to improve response rates (also telephones, home bar scanners)

Collect more days of recall data and more nonconsecutive days

Use two periods of household records (shorten the 7 days) to measure better the usual food use

Ensure questionnaires are answered completely

Use multiple measures (telephone and in-person interviews and mailed questionnaires)

Units of analysis Focus on individual data (NFCS)

Focus on household data and individuals (CSFII)

Time of data collection Need continuous survey or at least collect data more frequently

Collect NFCS data every 5 years and CSFII data in the interim

Collect longitudinal data to track changes in individual consumption (NFCS and CSFII)

Population group coverage Increase coverage of subpopulations and racial, ethnic, and age groups

Increase sample size

Need clearer criteria for definition of race

Focus on high-risk groups

Integrate CSFII with NHANES sample

Geographic area coverage Need

 More refinement of geographic detail (regions, localities, areas of low density)

 State-level estimates (allow states to collect their own data and feed into national survey)

Use small-area estimation models

Provide specific estimates for major population centers

(continued)

Page 37 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix III 

Summaries of the Suggestions Made by

Users

Type of change Comment

Ease of use Collect data more frequently and allow more rapid access to both published reports and raw

data

Provide

 User-friendly documentation

 More detailed data on sampling design variables

 Clear documentation on data tape and file format to facilitate combining record types

 Documentation on changes in format in food composition database, codebook, and recipe file

 Survey protocol and operations manual

 More technical assistance (to nonnutritional researchers)

 On-line documentation

 Data in common statistical package format

 Lists of surveys and sources for both data and technical assistance in professional journals

Page 38 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix IV 

Characteristics of the Uses of Data

Collection Activities

The tables in this appendix are based on the 440 responses from those

who have used at least one of the 14 data systems in the past 5 years. Since

some respondents pooled their answers, each of these 440 responses may

represent one or more than one user. The first two tables describe some

characteristics of the users in our sample. Table IV.1 shows the

occupations that users identified themselves with, by organizational

setting. To construct table IV.2, we asked them what data collection

activites they have used at all in the past 5 years.

Table IV.1: Main Occupation of Respondentsa

Organizational setting

Federal State Local Academic Business Otherb Average

Sample size 112 125 28 98 37 40

Occupation

Service delivery 6% 14% 32% 16% 14% 33% 15%

Basic research 47 6 0 57 35 28 32

Applied research 29 10 7 24 49 13 21

Program management

and planning

13 66 71 1 5 15 29

Other 7 6 0 2 16 3 5

aColumn percentage totals exceed 100 percent because some users identified more than one

main occupation.

bOther settings include hospitals, nonprofit organizations, and other charitable organizations.

Page 39 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix IV 

Characteristics of the Uses of Data

Collection Activities

Table IV.2: Respondents’ Use of Data Systems Within the Past 5 Yearsa

Organizational setting

Federal State Local Academic Business Otherb Average

Sample size 112 125 28 98 37 40

Data system

NHEFS 31% 14% 36% 39% 30% 45% 30%

NHANES II 50 17 36 41 59 58 39

NHANES III 57 17 39 29 30 48 35

HHANES 38 10 11 21 24 23 22

NHIS-Vitamin 9 8 7 10 3 23 10

NHIS-Cancer 11 5 14 16 14 20 12

BRFSS 13 71 39 15 11 33 33

PNSS 13 47 50 6 3 33 25

PedNSS 14 58 68 9 5 30 30

Navajo HNS 2 0 0 0 0 5 1

Health and Diet 13 7 11 10 19 23 12

NFCS 44 20 36 67 65 53 44

CSFII 46 10 7 55 46 40 35

DHKS 28 10 7 23 27 30 20

aColumn percentage totals exceed 100 percent because most users checked two data systems.

bOther includes hospitals, nonprofit organizations, and other charitable organizations.

Tables IV.3-IV.5 show three aspects of respondents’ satisfaction with each

data collection activity: first, whether it provided for their information

needs; second, whether it met their data quality needs; and third, whether

they thought changes were needed to either increase their confidence in or

substantially increase their use of the data system.

Page 40 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix IV 

Characteristics of the Uses of Data

Collection Activities

Table IV.3: Users’ Response to How Well the Current Data Meet Their Information Needs

Degree of satisfaction

Data collection activity Little or none Some Moderate Great Very great Sample size

NHEFS 2% 10% 45% 33% 10% 54

NHANES II 3 14 37 31 15 101

NHANES III 2 16 34 28 19 90

HHANES 0 7 37 33 22 28

NHIS-Cancer 0 23 46 15 15 13

BRFSS 4 30 42 19 6 85

PNSS 4 23 27 41 6 73

PedNSS 6 21 31 35 6 96

Health and Diet 8 8 33 33 17 13

NFCS 3 15 34 38 10 110

CSFII 6 13 37 36 8 89

DHKS 13 4 52 22 9 23

Median 4 15 37 33 10

Table IV.4: Users’ Response to How Well the Current Data Meet Quality Needs

Degree of satisfaction

Data collection activity Little or none Some Moderate Great Very great Sample size

NHEFS 4% 18% 36% 34% 8% 54

NHANES II 2 8 32 36 22 101

NHANES III 4 11 33 34 19 90

HHANES 0 7 22 37 33 28

NHIS-Cancer 0 8 23 62 8 13

BRFSS 6 22 42 25 5 85

PNSS 7 14 38 35 6 73

PedNSS 9 17 35 33 6 96

Health and Diet 0 17 42 25 17 13

NFCS 8 13 43 32 4 110

CSFII 5 15 42 31 7 89

DHKS 4 9 52 26 9 23

Median 4 14 37 34 8

Page 41 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix IV 

Characteristics of the Uses of Data

Collection Activities

Table IV.5: Users’ Response to Whether Changes Would Increase Their Confidence in or Use of the Data

Response Data collection

activity No Probably not Uncertain Probably yes Yes No basis to judge

Sample

size

NHEFS 8% 16% 24% 25% 27% 0 54

NHANES II 14 17 14 28 26 1% 101

NHANES III 13 17 16 34 16 5 90

HHANES 19 15 4 15 44 4 28

NHIS-Cancer 0 23 0 46 31 0 13

BRFSS 5 20 16 30 28 1 85

PNSS 4 26 16 26 24 3 73

PedNSS 6 26 11 27 29 2 96

Health and Diet 25 17 17 8 25 8 13

NFCS 3 12 13 30 38 4 110

CSFII 8 9 10 36 33 3 89

DHKS 9 9 22 26 35 0 23

Median 8 17 15 28 29 3

Page 42 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix V 

List of Experts

This appendix lists the expert advisers who assisted on this project. The

advisers were organized into three panels: core policy panel, methodology

panel, and data users panel.

Core Policy Panel Johanna Dwyer, D.Sc., R.D., Francis Stern Nutrition Center, New England

Medical Center and Tufts University Schools of Medicine and Nutrition

Jean-Pierre Habicht, M.D., Ph.D., Division of Nutritional Sciences, Cornell

University

Catherine Woteki, Ph.D.1

Methodology Panel Norman Bradburn, Ph.D., Director, National Opinion Research Center

Marilyn Buzzard, Ph.D., Director, Nutrition Coordinating Center,

University of Minnesota

Ricardo O. Castillo, M.D., M.P.H., Co-Director, Pediatric Gastroenterology,

Stanford University Medical Center

Alan R. Kristal, Dr. P.H., Fred Hutchinson Cancer Research Center and

Department of Epidemiology, University of Washington

James Lepkowski, Ph.D., Institute for Social Research, University of

Michigan

Cheryl Ritenbaugh, Ph.D., Department of Family and Community

Medicine, University of Arizona

Laura Sims, Ph.D., Department of Nutrition and Food Science, University

of Maryland

Data Users Panel Elizabeth Barnett, Ph.D., North Carolina Department of Environment,

Health, and Natural Resources

Doris Disbrow, Dr. P.H., R.D., Center for Health Education

1

Dr. Woteki withdrew from the panel when she was appointed to the Office of Science and Technology

Policy in the White House. During her participation in our work, she was the Director of the Food and

Nutrition Board of the National Academy of Sciences.

Page 43 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix V 

List of Experts

Pamela Haines, Dr. P.H., R.D., Department of Nutrition, University of

North Carolina

Jay Hirschman, M.P.H., Senior Analyst, Food and Consumer Service, U.S.

Department of Agriculture

Karen J. Morgan, Ph.D., Senior Director, Nutrition and Consumer Affairs,

Nabisco Brands

Barbara Petersen, Ph.D., Technical Assessment Systems

Page 44 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 45 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 46 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 47 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 48 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 49 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 50 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 51 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 52 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 53 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 54 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 55 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VI 

Comments From the Department of

Agriculture

Page 56 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VII 

Comments From the Public Health Service

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 57 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VII 

Comments From the Public Health Service

Page 58 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VII 

Comments From the Public Health Service

Page 59 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VII 

Comments From the Public Health Service

See comment 1.

See comment 2.

Page 60 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VII 

Comments From the Public Health Service

The following are GAO’s comments on the letter from the Public Health

Service dated May 1, 1995.

GAO Comments 1. Our analysis of the responses to the open-ended questions involved,

first, sorting responses by data collection activity and focus of the

comment (such as data element, population coverage, ease of use). These

responses were then aggregated to identify major themes. The comments

made by FDA users were not identified as a major theme across the many

users of the different systems and, thus, were not reported separately.

However, detailed summaries of the responses were provided to the

responsible agencies for their use.

2. We have included a reference to the Directory of Federal and State

Nutrition Monitoring Activities for those readers who are interested in

more information on the data collection systems. (See p. 3.)

Page 61 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Appendix VIII 

Major Contributors to This Report

Program Evaluation

and Methodology

Division

John Oppenheim, Assistant Director

Leslie Riggin, Assignment Manager

Lê Xuân Hy, Project Manager

James Joslin, Social Science Analyst

Venkareddy Chennareddy, Referencer

Elizabeth W. Scullin, Communications Analyst

(973403) Page 62 GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be mailed to a

single address are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100
GAO/PEMD-95-15 Nutrition Monitoring Data Serve Many Uses