United States General Accounting Office

GAO Report to the Ranking Minority Member,

Committee on Science, House of

Representatives

July 1995 NUTRITION

MONITORING

Establishing a Model

Program

GAO/PEMD-95-19

GAO United States

General Accounting Office

Washington, D.C. 20548

Program Evaluation and

Methodology Division

B-261325 

July 19, 1995

 The Honorable George E. Brown, Jr.

Ranking Minority Member

Committee on Science

House of Representatives

Dear Mr. Brown:

This report responds to your request that we identify features of a model nutrition monitoring

system and examine approaches to incorporating those features in the National Nutrition

Monitoring and Related Research Program (NNMRRP). It builds on two earlier reports. The first,

Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23), assessed

the NNMRRP’s planning activities. The second, Nutrition Monitoring: Data Serve Many Purposes;

Users Recommend Improvements (GAO/PEMD-95-15), surveyed users of nutrition monitoring data

about how the data are used and what changes are needed to the data collection activities. This

report describes features of a model nutrition monitoring system and presents the strengths and

limitations of various strategies for achieving four of the model features.

As arranged with your office, we will be sending copies of this report to the Director of the

Office of Management and Budget, the Interagency Board for Nutrition Monitoring and Related

Research, the agencies responsible for data collection, and interested congressional

committees. We will also make copies available to others upon request.

The major contributors to this report are listed in appendix V. If you have any questions or

would like additional information, please call me at (202) 512-3092.

Sincerely yours,

Kwai-Cheung Chan

Director of Program Evaluation

 in Physical Systems Areas
Executive Summary

Purpose Nutrition-related data are used to monitor food safety, study the

relationship between diet and health, inform agricultural policies, evaluate

food assistance programs, and help food industries develop new products.

However, past evaluations have criticized federal nutrition monitoring

activities as poorly coordinated and insufficiently responsive to the needs

of data users. At the request of the former House Committee on Science,

Space, and Technology, GAO defined the features of a model monitoring

program and compared current and potential approaches to responding to

the needs of data users.

Background The current National Nutrition Monitoring and Related Research Program

(NNMRRP) consists of a combination of national surveys, federallysupported surveillance systems operated by the states, and other research

and data collection activities. The 1990 National Nutrition Monitoring and

Related Research Act (P.L. 101-445) established an Interagency Board that

gave the Departments of Agriculture (USDA) and Health and Human

Services (HHS) primary responsibility for developing and implementing a

10-year comprehensive plan for federal nutrition monitoring.1

In reviewing the Interagency Board’s progress in coordinating the

program, an earlier GAO report found that coordination between USDA and

HHS had improved, but the 10-year plan was still inadequate. It neither

established priorities nor provided a framework for evaluating existing or

potential monitoring activities.2

 Drawing on a survey of users of nutrition

monitoring data, a second GAO report found that the data collected by the

NNMRRP support an extensive range of purposes, but changes could

increase the utility and credibility of the data.3

 For this report, GAO first

defined the features of a model program by reviewing past evaluations of

federal nutrition monitoring and the 10-year plan, consulting experts, and

surveying users of nutrition data. Then, with the assistance of expert

advisers and studies of illustrative programs, GAO identified and compared

current and potential approaches to incorporating model features in the

NNMRRP. (See chapter 1.)

1

Ten-Year Comprehensive Plan for the National Nutrition Monitoring and Related Research Program,

58 Fed. Reg. 111 (June 11, 1993), pp. 32752-806.

2

See Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23; May 27,

1994).

3

See Nutrition Monitoring: Data Serve Many Purposes; Users Recommend Improvements

(GAO/PEMD-95-15; June 20, 1995).

Page 2 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Executive Summary

Results in Brief GAO identified four features of a model program: It would have a

coordinated set of activities, provide data continuously, generate reliable

inferences about important subpopulations and small geographic areas,

and support state and local monitoring activities. While the NNMRRP already

has some elements of a model program, other strategies may lead to

improved nutrition monitoring capabilities. Alternate approaches to

coordination, such as an independent central office with authority over the

NNMRRP agencies or a single lead agency, may not provide any clear

advantages to the current structure, which relies on the Interagency

Board. However, such approaches do offer useful tools that could be

adopted by the Board, such as the use of formal budget reviews and the

establishment of a central contact for data users. The current source of

continuous data—the state-based surveillance systems—cannot meet the

needs for national-level information or for in-depth dietary intake

information. Alternatives are to attach modules of nutrition-related

questions to other ongoing surveys or to field a core set of questions

continuously, supplemented periodically by questions of intermittent or

emerging interest. Current approaches to providing more information on

subpopulation groups and small geographic areas are to oversample

selected groups as part of the national surveys and to collect data on

specific high-risk groups through the surveillance systems. These

approaches could be complemented by special studies and indirect

estimation. Finally, to support state and local monitoring activities, HHS

provides a combination of technical and financial assistance for the

state-based surveillance systems. One option to provide more relevant

data to localities is community-based data collection.

GAO’s Analysis

Features of a Model

Nutrition Monitoring

Program

Coordination of the data collection activities is needed to ensure that the

diverse needs for the data are met efficiently. For example, NNMRRP’s major

national surveys collect complementary data, with one focusing on health

and nutritional status and the others emphasizing food consumption.

Combining these data would increase their utility. (See chapter 2.)

Continuous data collection, “repeated regularly and frequently” (as defined

in the 10-year plan), is needed to track trends in diet-related health risks,

such as those targeted in HHS’ year 2000 health objectives. In addition,

continuous data are important in evaluating the effects of policy changes,

such as replacing current food assistance programs with block grants.

Page 3 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Executive Summary

The dietary habits and nutritional problems of a subpopulation or small

geographic area may deviate from the national profile. Data on these

groups are needed to appropriately target programs that address their

nutrition-related problems, such as diabetes in some minority groups.

Finally, states and localities need relevant nutrition monitoring data to

plan programs that respond to the specific health and nutrition concerns

of their populations. For example, currently available data have been used

to initiate “5-a-Day” programs that encourage consumption of fruits and

vegetables and to establish anemia as a priority issue for nutrition

education.

Current and Potential

Approaches to Achieving a

Model Program

While the Interagency Board appears to have improved communication

and cooperation among the agencies, its 10-year plan did not establish

clear priorities for nutrition monitoring across agencies. The Board has

conducted a priority-setting exercise, but it has made little progress in

developing a framework to guide NNMRRP decision-making. The two

alternate approaches to coordinating nutrition monitoring that GAO

examined suggest mechanisms for strengthening the work of the

Interagency Board. Appointing an NNMRRP administrator, for example,

could provide a central contact point for users of nutrition monitoring

data. In addition, regular budget reviews could be conducted to ensure

that priority objectives of the programs are funded. (See chapter 3.)

None of the three major national surveys are implemented continuously.

Moreover, the surveillance systems, which are the current source of

continuous data, cannot meet the needs of data users, who depend on

national-level data because of limitations in the amount and kind of data

they gather and in the populations they target. A continuous national

survey could be more efficient in terms of costs per respondent than the

current, intermittent surveys, especially if a smaller, core set of data

elements are collected continuously and supplemented periodically by

topical modules. However, activities that are now implemented

sequentially would have to be funded and staffed to operate concurrently.

Another approach to gathering information more frequently is the

inclusion of nutrition-related questions on other ongoing surveys. (See

chapter 4.)

At present, the state-based surveillance systems and oversampling in the

national surveys provide information on subpopulations and small

geographic areas. Two surveillance systems focus on low-income pregnant

women and children, but not all states participate and data are collected

only from participants in public health and nutrition programs.

Page 4 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Executive Summary

Oversampling of specific subpopulations as part of a national survey is

most appropriate for groups that are geographically clustered and that do

not require different data collection procedures than those used for the

general population. Its advantage is that the data on the group of interest

can be compared to national estimates.

Two alternate ways of gathering information on subpopulations and small

geographic areas are special studies and indirect estimation programs.

Special studies are appropriate for populations that are geographically

dispersed or require different data collection procedures, but they can be

costly relative to other approaches. Indirect estimates, which use data

from other sources to predict the value for the group or area of interest,

are likely to be less costly than collecting new data for direct estimation.

However, gauging the bias introduced by the models that produce the

indirect estimates is difficult; thus, they may not be as useful as direct

estimates. (See chapter 5.)

Through the state surveillance systems, HHS provides assistance in the

collection, analysis, and interpretation of state and local data. Although

the data collection instruments can be modified from state to state, the

systems are organized to collect standardized information across the

participating states. As a result, they may have limited sensitivity to state

or local differences. An alternate strategy is to design monitoring

programs specifically for local needs. However, unlike the data collected

by the surveillance systems, community-based data could probably not be

aggregated to describe statewide needs. (See chapter 6.)

Recommendations GAO is not making recommendations in this report.

Agency Comments In their written comments on a draft of this report, officials from USDA and

HHS (1) found the report to be factual, (2) agreed with the model program

features identified by GAO, (3) agreed with the importance of the model

features discussed in the report, and (4) provided additional detail on the

progress that has been made by the Interagency Board and its working

groups toward improving NNMRRP. Both USDA’s and HHS’ comments as well

as comments received from three members of the National Nutrition

Monitoring Advisory Council noted that limited resources affect the

NNMRRP’s ability to incorporate the model features. (See appendixes III and

IV for comments from USDA and HHS and our response.)

Page 5 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Contents

Executive Summary 2

Chapter 1 

The Current Nutrition

Monitoring Program

10

Core Data Collection Activities of the NNMRRP 10

Structure of the NNMRRP 13

Objectives, Scope, and Methodology 14

Organization of the Report 18

Chapter 2 

A Model Nutrition

Monitoring Program

20

Features of a Model Program 20

The Selected Features 22

Approaches to Achieving the Selected Model Features 29

Chapter 3 

Approaches to

Coordinating the

Nutrition Monitoring

Program

31

Current Approach: the Interagency Board 31

Potential Options 36

Conclusions 41

Chapter 4 

Approaches to the

Continuous Collection

of Data

42

Current Approach: the State-Based Surveillance Systems 42

Potential Options 45

Conclusions 49

Chapter 5 

Approaches to

Supporting Reliable

Inferences About

Subpopulations

50

Current Approaches 51

Potential Options 53

Conclusions 57

Page 6 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Contents

Chapter 6 

Approaches to

Assisting States and

Localities

59

Current Approach: the State-Based Surveillance Systems 59

Potential Option: Community-Based Nutrition Monitoring 61

Conclusions 63

Appendixes Appendix I: List of Experts 64

Appendix II: Sources for the Model Features 66

Appendix III: Comments From the Department of Agriculture 73

Appendix IV: Comments From the Public Health Service 80

Appendix V: Major Contributors to This Report 85

Related GAO Products 88

Tables Table 1.1: Selected Uses of NNMRRP Data by Content Area 10

Table 1.2: Major NNMRRP Activities by USDA and HHS 11

Table 2.1: Features of a Model Nutrition Monitoring Program and

Their Sources

20

Table 2.2: Examples of State and Local Uses of Nutrition Data 28

Table 2.3: Features of a Model Program and Current and Potential

Approaches

30

Table 3.1: Approaches to Coordinating Nutrition Monitoring

Activities

31

Table 3.2: Responsibilities and Coordination Tools of Three

White House Offices

37

Table 4.1: Approaches to Providing Continuous Data 42

Table 5.1: Approaches to Supporting Inferences About

Subpopulations

51

Table 6.1: Approaches to Assisting States and Localities 59

Table II.1: Features of a Model Program Suggested by Past

Evaluations

66

Table II.2: Examples of Planned NNMRRP Activities Associated

With Model Features

68

Table II.3: Advisory Panel Suggestions Associated With Model

Features

69

Table II.4: Major Themes in the Suggestions Made by Data Users 71

Page 7 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Contents

Abbreviations

ARS Agricultural Research Service

BRFSS Behavioral Risk Factor Surveillance Survey

CPS Current Population Survey

CSFII Continuing Survey of Food Intake by Individuals

DHKS Diet and Health Knowledge Survey

HHANES Hispanic Health and Nutrition Examination Survey

HHS Department of Health and Human Services

HPCCP High Performance Computing and Communications

Program

NCST National Council on Science and Technology

NFCS Nationwide Food Consumption Survey

NHANES National Health and Nutrition Examination Survey

NNMRRP National Nutrition Monitoring and Related Research

Program

OMB Office of Management and Budget

ONDCP Office of National Drug Control Policy

OSTP Office of Science and Technology Policy

PATCH Planned Approach to Community Health

PedNSS Pediatric Nutrition Surveillance System

PHS Public Health Service

PNSS Pregnancy Nutrition Surveillance System

USDA U.S. Department of Agriculture

WIC Special Supplemental Food Program for Women, Infants,

and Children

Page 8 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Page 9 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

This chapter first provides background information on the National

Nutrition Monitoring and Related Research Program. Then, the objectives,

scope, and methodology of our review of current and potential approaches

to achieving a model program are described. The chapter concludes with

an overview of the organization of the rest of the report.

Core Data Collection

Activities of the

NNMRRP

The NNMRRP is a complex system of data collection and research activities,

including national surveys, state surveillance activities, and a variety of

research programs. Over time, the NNMRRP has developed activities focused

on five content areas: (1) food and nutrient consumption; (2) nutritional

and health status; (3) dietary knowledge, attitudes, and behavior; (4) food

composition; and (5) food supply. As shown in table 1.1, the information

produced by these activities is used for a variety of purposes, from

supporting basic research on human nutritional needs to informing policy

decisions about health, agriculture, and food programs.

Table 1.1: Selected Uses of NNMRRP Data by Content Area

Content area Selected usesa

Food and nutrient consumption Assess potential exposure of children to pesticide residues

Develop standards for food stamp eligibility

Evaluate impact of farm commodity programs on demand and pricing

Monitor trends in diet-related risks of chronic diseases

Nutritional and health status Evaluate food assistance programs

Research the relationship between fat consumption and cancer

Target food fortification to foods eaten by people with nutrient deficiencies

Update infant growth charts

Inform policies on cholesterol screening and treatment

Dietary knowledge, attitudes, and behavior Develop food labeling policies

Inform nutrition education programs

Food composition Support estimation of the consumption of nutrients and food components related to

chronic disease

Plan meals for military personnel and other groups

Monitor trends in nutrient availability and food safety

Food supply Manage federal marketing and agricultural policies

aThese uses were selected to illustrate the variety of purposes supported by NNMRRP data. They

are not an exhaustive list.

Table 1.2 lists specific activities managed by the Departments of

Agriculture and Health and Human Services, which have major

Page 10 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

responsibilities for the five areas. Other agencies, including Commerce,

Defense, and the Environmental Protection Agency, also participate in the

NNMRRP.

Table 1.2: Major NNMRRP Activities by

USDA and HHS Department

Content area USDA HHS

Food and nutrient

consumption

Nationwide Food

Consumption Survey

Continuing Survey of Food

Intake by Individuals

National Health and

Nutrition Examination Survey

Total Diet Study

Nutritional and health status No major role National Health and

Nutrition Examination Survey

Pregnancy Nutrition

Surveillance System

Pediatric Nutrition

Surveillance System

Dietary knowledge, attitudes,

and behavior

Diet and Health Knowledge

Survey

Health and Diet Survey

Behavioral Risk Factor

Surveillance System

Food composition National Nutrient Data Bank Food Label and Package

Survey

Food supply Food and Nutrition Supply

Series

No major role

USDA Activities USDA has major responsibilities for collecting information in all of the

content areas except nutritional and health status. USDA gathers data on

food and nutrient consumption through two national surveys—the

Nationwide Food Consumption Survey (NFCS) and the Continuing Survey

of Food Intake by Individuals (CSFII). In the past, NFCS has gathered

nationally-representative information on the food consumption behavior

of households and individuals. It provides detailed data on household

costs for food. One of the major uses for these data is the development of

the Thrifty Food Plan, which is the basis for calculating food stamp

benefits. Implemented decennially, NFCS suffered from severe response

rate problems (less than 40 percent) in 1987-88. As a result, the individual

food consumption portion of the NFCS is expected to be dropped in the

future.

Page 11 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

Since the mid-1980s, CSFII has supplemented NFCS by providing regular

information on individual dietary intake. The data collected in its most

recent implementation (1994-96) will be used to describe both general and

low-income populations. The Diet and Health Knowledge Survey (DHKS),

which collects data on dietary knowledge, attitudes, and behavior, is a

follow-up to CSFII. Together, CSFII and DHKS are intended to inform policies

relating to food production and marketing, food safety, food assistance,

and nutrition education.

The USDA activities focused on food composition and food supply are not

surveys. For food composition, USDA gathers data from food industries and

other sources on the nutrient content of foods. These data support the

dietary surveys by translating the foods consumed into their nutrient

components. Food supply is estimated by deducting data on exports,

year-end inventories, and nonfood use from data on production, imports,

and beginning inventories.

HHS Activities HHS is responsible for the National Health and Nutrition Examination

Survey (NHANES) and the state-based surveillance systems. These data

collection activities provide information on the content areas (1) food and

nutrient consumption, (2) nutritional and health status, and (3) dietary

knowledge, attitudes, and behavior. Like NFCS and CSFII, NHANES collects

data from a nationally-representative sample. However, NHANES’ unique

contribution is its use of physical examinations and clinical and laboratory

tests as well as traditional survey methods to gather information. NHANES’

data support research on the relationship between diet and health and

inform health policy decisions, such as the promotion of cholesterol

screening. After two earlier implementations in the 1970s, NHANES has just

completed its third administration (1988-94). NHANES has been

supplemented by the Hispanic Health and Nutrition Examination Survey

(HHANES) in the early 1980s, follow-up surveys of respondents to NHANES I,

and follow-up matches of the records from NHANES II to the National Death

Index and other vital statistics records.

The state-based surveillance systems were set up to provide quick

information to states to use in planning and managing nutrition and health

programs. They include the Pediatric Nutrition Surveillance System

(PedNSS), Pregnancy Nutrition Surveillance System (PNSS), and Behavioral

Risk Factor Surveillance System (BRFSS). Participating states collect the

data for these systems with technical and other kinds of assistance from

HHS. Both PedNSS and PNSS rely on data from clinic records from publicly

Page 12 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

funded health, nutrition, and food assistance programs, primarily the

Special Supplemental Food Program for Women, Infants, and Children

(WIC). PedNSS monitors nutritional status among low-income, high-risk

children, while PNSS focuses on low-income, high-risk pregnant women,

measuring nutrition-related problems and behavioral risk factors

associated with low birthweight.

In contrast to PedNSS and PNSS, data for BRFSS are gathered through

telephone interviews, with respondents (adults 18 years and over)

sampled through random digit dialing. In addition to a core set of

questions on various health risk factors, BRFSS includes optional modules

for the assessment of dietary fat and fruit and vegetable consumption.

Other major HHS monitoring activities include the Total Diet Study, which

analyzes nutrient and contaminant levels in the food supply; the Food

Label and Package Survey, which monitors nutrition labeling practices;

and the Health and Diet Survey, which assesses dietary knowledge and

practices as they relate to health problems.

Structure of the

NNMRRP

Although the United States has one of the most comprehensive monitoring

programs in the world, several problems with nutrition monitoring

activities have been identified over the past two decades. Of key concern

has been the lack of coordination and compatibility of different data

collection activities. This encompasses differences across surveys in

methods for assessing dietary intake and nutritional status, sampling

designs, population descriptors and other measures, and the timing and

reporting of results.

To improve the coordination of federal nutrition monitoring activities and

the quality of the data collected, the Congress passed the National

Nutrition Monitoring and Related Research Act of 1990 (P.L. 101-445). The

act established an Interagency Board, jointly chaired by USDA and HHS, to

coordinate activities across the various agencies involved in nutrition

monitoring. The Interagency Board was charged with developing a

strategic plan that would establish a comprehensive nutrition monitoring

and related research program. This plan—known as the 10-year

comprehensive plan—was published in the Federal Register on June 11,

1993. It outlines a set of planning activities, including a general time frame

and lead agencies for each activity. The activities are organized around six

objectives, which are to

Page 13 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

• provide for a comprehensive NNMRRP through continuous and coordinated

data collection;

• improve the comparability and quality of data across the NNMRRP;

• improve the research base for nutrition monitoring;

• develop and strengthen state and local capacity for continuous and

coordinated nutrition monitoring data collection that complements

national nutrition surveys;

• improve methodologies to enhance comparability of NNMRRP data across

federal, state, and local levels; and

• improve the quality of state and local nutrition monitoring data.

In addition, an Advisory Council of experts from outside the federal

government was created to guide the Interagency Board on scientific and

technical matters. (Chapter 3 contains more information on the

Interagency Board and its activities.)

Objectives, Scope,

and Methodology

Objectives This is the third and final report in a series responding to a request from

the former House Committee on Science, Space, and Technology. The first

report reviewed past evaluations of federal nutrition monitoring and

examined the progress of the NNMRRP since the passage of the 1990 act.1

 It

concluded that (1) a coherent program for nutrition monitoring was not

yet in place and (2) although there has been progress in coordinating the

program, the 10-year plan is incomplete because it does not include a

framework for evaluating current and potential activities or detailed plans

for achieving the objectives.

Based on a survey of users of nutrition monitoring data, the second report

described the purposes for which nutrition data are used and summarized

respondents’ suggestions for improving NNMRRP activities.2

 These

suggestions, which addressed such issues as the timing of the surveys,

their coverage of subpopulations, and the ease with which the data could

be used, were consistent with the concerns raised by past evaluations of

1

Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23).

2

Nutrition Monitoring: Data Serve Many Purposes; Users Recommend Improvements

(GAO/PEMD-95-15).

Page 14 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

federal nutrition monitoring and indicated a continued need to address the

long-standing problems of federal nutrition monitoring.

Completing our response to the Committee’s request, this report builds on

our earlier work to meet two objectives: (1) define a model nutrition

monitoring program and (2) compare the current system with potential

options for implementing the components of a model program in the

NNMRRP.

Scope and Methodology

Defining the Features of a

Model Program

Before defining the features of a model program, we first limited our

review to three of the five content areas covered by the NNMRRP: (1) food

consumption and dietary intake; (2) health and nutritional status; and

(3) knowledge, attitudes, and behavior. These three variables were

selected for both substantive and methodological reasons. Substantively,

the selected elements provide the data that support the planning and

evaluation of interventions that directly affect health, such as nutrition

education and food assistance programs. A substantive argument could

also be made for the inclusion of food composition data, which are used to

translate the information on what foods are eaten into estimates of

nutrient intake. However, because an earlier GAO project focused on the

NNMRRP’s food composition activities, we did not include this in our

review.3

Methodologically, the three selected variables are linked because they rely

on data obtained from individuals through surveys and physical

examinations. In contrast, food composition information is based on

chemical analyses of foods, and food supply is determined from

macroeconomic data. Our focus on the components of the NNMRRP that rely

on surveys facilitates the comparison of current and potential approaches

to achieving a model program.

To identify features of a model nutrition monitoring program, we used

four sources: reviews of previous evaluations of federal nutrition

monitoring activities, review of the objectives and related activities

outlined in the 10-year comprehensive plan developed by the Interagency

Board, consultation with expert advisers, and our survey of data users.

3

See Food Nutrition: Better Guidance Needed to Improve Reliability of USDA’s Food Composition

Data (GAO/RCED-94-30; Oct. 25, 1993).

Page 15 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

These sources are detailed below, and supporting material is provided in

appendix II. (See chapter 2 for a description of the features.)

Past Evaluations of Federal Nutrition Monitoring. Past evaluations by such

groups as the National Academy of Public Administrators, the Joint

Nutrition Monitoring Evaluation Committee, and the National Research

Council identified several concerns about the federal nutrition monitoring

program. Because these evaluations also informed the Interagency Board’s

development of its 10-year plan, we used them as the starting point for our

identification of features of a model nutrition monitoring program. Our

report, Nutrition Monitoring (GAO/PEMD-94-23), discusses these past

evaluations and NNMRRP progress in addressing their recommendations.

(Table II.1 in appendix II lists the criticisms identified in these evaluations

and categorizes them by the features of a model program that they

suggest.)

The 10-Year Comprehensive Plan. As described above, the NNMRR Act

required the Interagency Board to develop a plan for the program. The

plan outlined six objectives, three with a federal focus and three

emphasizing state and local monitoring, and listed 68 planned activities.

These were reviewed for responsiveness to the model features suggested

by the past evaluations. (See table II.2 in appendix II for examples of the

68 activities listed in the plan.)

Consultation With Expert Advisers. To assist us at critical decision points

in this project, we organized three panels. (The members of each panel are

listed in appendix I.) The Core Policy Panel consisted of nationally-known

experts in nutrition and nutrition monitoring policy. These panelists were

consulted throughout the project. In addition, they helped us develop a

framework of purposes for nutrition monitoring data that guided our

survey of data users.

The Methodology Panel included renowned experts in such fields as

sampling, survey design, dietary assessment, and nutritional epidemiology.

This panel met to help us identify promising approaches to critical

elements of a nutrition monitoring system. In addition, the panelists

assisted us on issues related to their areas of specialization.

The Data Users Panel consisted of users of the nutrition monitoring data,

chosen to reflect the broad range of purposes that the data must serve,

including the support of state and local nutrition programs, academic

research, food industry research, and the development and evaluation of

Page 16 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

federal food assistance programs. As with the Methodology Panel, this

panel was convened once to help us identify promising approaches to

nutrition monitoring. Individual panelists were consulted later about

specific issues related to their expertise.

Through the process of reviewing materials and participating in panel

meetings, the expert advisers generated several suggestions for possible

changes to the NNMRRP, examples of which are given in table II.3 in

appendix II. In addition, they reviewed a draft of this report.

Suggestions From the Survey of Data Users. Our survey of users of

nutrition monitoring data focused on primary users of 14 of the NNMRRP

surveys and surveillance systems. Primary users were defined as those

who directly access the data rather than use information that has already

been processed and interpreted by others in reports and other documents.

Since there is no single list of primary users of NNMRRP data, we obtained

lists of known and potential data users such as people who had requested

the data from NNMRRP agencies, attendees at nutrition-related workshops,

state and local government officials working in nutrition, and members of

associations for nutrition professionals.4

A major portion of the survey was dedicated to determining how the

respondents used the data. We also asked whether changes are needed to

better meet the respondent’s information and data quality needs. If the

respondent indicated a need for change, we asked for suggestions in the

following categories: (1) data elements collected, (2) data collection

methods, (3) units of analysis, (4) time of data collection, (5) population

group coverage, (6) geographic area coverage, and (7) ease of use. These

comments were analyzed to identify major themes for each of three

groups of data collection activities—USDA surveys, HHS surveys, and HHS

state-based surveillance systems. In appendix II, table II.4 identifies the

themes associated with the features of a model program.

Comparing Approaches to

Attaining the Model Features

From the features that were identified, we selected four as the focus of the

second objective—the comparison of current and potential approaches to

achieving the model program. We focused on features that reflect

long-standing concerns about federal nutrition monitoring, that

encompass other desired characteristics, and that generate debate about

how they should be addressed.

4

See Nutrition Monitoring (GAO/PEMD-95-15) for details of the survey.

Page 17 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

For each of the selected features, we identified current activities of the

NNMRRP through interviews with staff in NNMRRP agencies, attendance at

meetings of the Interagency Board and the Advisory Council, and reviews

of program documents. Potential approaches were identified through

literature review, analysis of our survey results, and consultation with the

expert advisers. We did not identify the universe of potential approaches.

Instead, our search was focused on those approaches deemed promising

and feasible by our expert advisers.

To assess each of the potential approaches, we first identified programs

from the same set of sources that helped us define the features of a model

program—that is, the literature, our survey of data users, and expert

advisers. Where possible, we limited our consideration to programs that

had some linkage to nutrition monitoring. For example, for separate

studies of subpopulation groups, we looked at the experience with the

Hispanic Health and Nutrition Examination Survey in the early 1980s. To

describe the strengths and weaknesses of the programs illustrating the

potential options, we reviewed program documents and related literature

and interviewed managers and staff.

This review was conducted between October 1993 and December 1994 in

accordance with generally accepted government auditing standards.

Strengths and Limitations

of Our Methodology

This report describes the results of a systematic examination of current

and potential approaches to selected features of a model nutrition

monitoring program. The strength of the review is its reliance on multiple

sources of information. In addition to surveying users of nutrition

monitoring data, consulting with experts, and reviewing both technical

literature and program documents, we also interviewed officials and

program staff in nutrition monitoring programs and in programs

illustrating the alternatives. The major limitation of our work is its

prospective nature. Because we were examining potential changes to the

NNMRRP, hard evidence of the costs or effectiveness of the options was not

available. Instead, the strengths and weaknesses of the options relative to

the current nutrition monitoring system are supported primarily by logic

and stated in tentative terms. Given this limitation, the report makes no

recommendations for specific changes to the NNMRRP.

Organization of the

Report

In response to the first objective of the project, chapter 2 describes the

model program and provides information on the selection of four model

Page 18 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 1 

The Current Nutrition Monitoring Program

features as the focus of the report. (The sources used to develop the model

are described above.) The second objective—comparing current and

potential approaches to each model feature—is addressed in chapters 3-6,

organized by the four model features. Specifically, chapter 3 examines

coordination options; chapter 4 compares alternate approaches to

providing continuous data; chapter 5 discusses different methods of

supporting inferences about subpopulation groups and small geographic

areas; and chapter 6 reviews approaches to assisting state and local

monitoring activities. Appendix I provides additional detail on the expert

advisers to the project, and appendix II describes the sources for the

model features. Agency comments on a draft of the report are in

appendixes III and IV.

Page 19 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

This chapter responds to the first objective of our review—the definition

of a model nutrition monitoring program. First, the major features

identified from the sources described in chapter 1 are outlined. From this

set of features, we selected four as the focus for our response to the

second objective of the review—the comparison of current and other

approaches to achieving the model characteristics. The chapter describes

our selection process and details the importance of the four features that

are the subject of the rest of the report.

Features of a Model

Program

Depending on the purposes that the data serve, the specific elements of a

model nutrition monitoring program change. For example, researchers

and program managers interested in food safety need detailed information

on dietary intake, including specific brand names of the foods consumed.

In contrast, a nutrition educator may place a higher priority on

information about dietary knowledge, attitudes, and behaviors. However,

at a more general level, some common ideal characteristics can be

identified. Focusing on this general level, we used the sources described in

chapter 1 to identify a number of features of a model program. Table 2.1

lists these features and the sources that support them.

Table 2.1: Features of a Model

Nutrition Monitoring Program and

Their Sources

Feature Sourcea

Coordinated system that responds efficiently

to the diverse needs of the data users

Past evaluations

10-year plan

Expert advisers

Continuous or more frequent collection of data Past evaluations

10-year plan

Expert advisers

Survey of data users

Support for reliable inferences about subpopulation

groups and small geographic areas

Past evaluations

10-year plan

Expert advisers

Survey of data users

(continued)

Page 20 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

Feature Sourcea

Assistance to states and localities Past evaluations

10-year plan

Expert advisers

Survey of data users

Strong research base, including improved methods 

for assessing dietary intake and nutritional status

Past evaluations

10-year plan

Expert advisers

Survey of data users

High response rates and low respondent burden Past evaluations

Survey of data users

Timely processing and dissemination of survey

information

Past evaluations

10-year plan

Survey of data users

On-going evaluation of the system’s content and 

methods, including a review of the information needs

Expert advisers

Collection of both household and individual data Expert advisers

Survey of data users

Maintenance of data comparability over time Expert advisers

Longitudinal data (collection of information on the

same individuals or households over time)

Expert advisers

Survey of data users

aFor more information on these sources, see the discussion in chapter 1 and the tables in

appendix II.

All of the features identified are clearly important elements of a

comprehensive nutrition monitoring program. However, we selected four

characteristics as the focus of our review: (1) a coordinated set of

activities that (2) provides data on a continuous basis, (3) supports

inferences about important population groups, and (4) assists state and

local monitoring activities. These features encompass other desired

characteristics, respond to long-standing concerns about federal nutrition

monitoring, and generate debate about how they can be achieved.

The first criterion for selection was to focus on the most general concerns.

With these features, other desirable characteristics can be considered even

Page 21 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

though they are not emphasized. Specifically, a mechanism for evaluating

the NNMRRP’s content and methods, including a review of the information

needs of the data users, is described in chapter 3 as an element of a

coordinated program. The theme of evaluating options in relation to the

needs for data also underlies the discussion of the alternatives for the

other features. Three other features—the comparability of data over time,

the collection of longitudinal data, and the timeliness of data release—are

related to the continuous collection of data and, as such, are considered

briefly in chapter 4.

The four features were also selected because they respond to

long-standing criticisms of federal nutrition monitoring activities.

Concerns about coordination, the continuity and timeliness of the data,

the availability of information on subpopulations, and the role of states

and localities were raised as early as 1977 by witnesses before a House

subcommittee.1

 In contrast, the concerns with response rates and the level

of the data (individual or household) can be traced to the problems with

the last NFCS (described in chapter 1), which USDA has taken steps to avoid

in the future.

The selected features are also not easily addressed; that is, there is debate

about the best approach to achieving each feature. For example, some

expert advisers stated that assistance to states and localities should focus

on data analysis and interpretation, while others argued for a larger state

and local role in data collection. Similarly, to provide information on

subpopulation groups, national surveys could be supplemented by such

means as the surveillance systems and oversampling or the national

surveys could be abandoned and their resources dedicated to special

studies of specific groups. In contrast to the debates about how best to

achieve these and the other selected features, there has been consensus

about the kind of dietary intake methodology that will be used by the

NNMRRP surveys and ongoing research to improve these methods through

automation and other means.

The Selected Features An ideal federal nutrition monitoring program would have a coordinated

set of activities that provides data on a continuous basis, covers important

population groups, and supports state and local monitoring activities.

Coordination is the key both to the efficiency of the system and its

responsiveness to the needs of data users. A continuous flow of data

1

Hearings before the House Subcommittee on Domestic and International Scientific Planning, Analysis

and Cooperation, Committee on Science and Technology, July 26, 27, and 28, 1977.

Page 22 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

would ensure that the information on the nation’s nutritional status was

up-to-date and would also enable the tracking of dietary behavior and

nutritional status over time. Information on population groups that are

vulnerable or growing rapidly is needed to plan, manage, and evaluate

programs intended to prevent or ameliorate nutritional problems. Because

state and local governments are often the location of such programs, they

need assistance in either interpreting available data or collecting their own

information. The importance of these features is further explained in the

following sections.

A Coordinated Nutrition

Monitoring Program

The coordination of nutrition monitoring activities has implications for

both the utility of the information produced and the costs of the program

as a whole. The utility of the information is constrained when data from

different data collection activities cannot be easily combined. For

example, research on the relationship between diet and health could be

strengthened if CSFII’s data on dietary intake could be combined with

NHANES’ data on health and nutritional status. However, because of

differences in the sampling designs and nutrition measures, combining the

data from the two surveys is difficult and controversial. Similarly, poor

integration of the data collected by the state-based surveillance systems

with the national surveys presents a barrier to meaningful comparisons of

state and national populations.

To the extent that the lack of coordination results in unnecessary

duplication, a fragmented system can also increase the costs of nutrition

monitoring. For example, the current NNMRRP includes two surveys

focused on dietary knowledge, attitudes, and behaviors—one operated by

USDA and the other by HHS. The Interagency Board plans to review these

surveys for duplication. While some overlap can be useful as a quality

check on data from different sources, unnecessary redundancy in the

system uses resources that could be used for currently unmet data

needs—such as the need for information on specific subpopulations at

risk for nutrition-related problems.

The need for a coordinated system was supported by three of the four

sources used to identify features of a model program. In addition to a

general concern about the lack of coordination, past evaluations of federal

nutrition monitoring have criticized the incompatibility of the data

gathered by different enterprises. Specifically, these evaluations have

called for compatible methods of assessing dietary intake, a core set of

standardized measures for the major surveys, compatible sampling

Page 23 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

techniques in the national surveys, and integrated reporting. Another

criticism related to coordination focused on the absence of a systematic

process for determining the needs for nutrition monitoring data across the

different data collection activities. Although needs are assessed for

individual activities, no comprehensive assessment of needs in relation to

the total system of activities has taken place.

Coordination is also a major theme of the 10-year plan. Four of the six

objectives discussed in the plan focus on coordinating data collection and

improving the comparability of the data. Of the specific activities listed for

each objective, some of those focused on coordination are: coordinating

the planning for coverage, tracking, and reporting of findings from surveys

and surveillance systems; identifying ways to increase comparability

within a dietary method to improve the quality and usefulness of data; and

establishing a mechanism for improved coordination among federal

agencies that collect and use survey information about knowledge,

attitudes, and behavior to assess gaps and duplications in existing surveys.

The third source of support for the need for coordination comes from

meetings of the advisory panels. The panelists noted that the different

agencies involved in nutrition monitoring have different missions and

priorities and, hence, coordination is difficult. Their major suggestions for

improving coordination were to give coordination responsibility to a single

lead agency, to coordinate from an interagency body with permanent staff

and enforcement authority, to locate nutrition monitoring in statistical

agencies within the user departments or within a central statistical agency,

to centralize the congressional appropriations process for nutrition

monitoring activities, and to ensure informed review of data collection

plans by qualified staff at the Office of Management and Budget.

Our survey queried respondents about changes to specific data collection

activities, rather than the NNMRRP as a whole, so coordination was not a

major theme of the comments provided by the data users.

The Continuous Collection

of Data

The Interagency Board defines continuous data as data collection that is

“repeated regularly and frequently.”2

 Two consequences follow when data

are not regularly available. First, because the kinds of foods that are

available and the eating patterns of the American people change rapidly,

the data become outdated quickly. Compounded by delays between data

collection and data release, long intervals between administrations of the

2

Ten-Year Comprehensive Plan, 58 Fed. Reg. 32806 (1993).

Page 24 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

surveys diminish the relevance of the data to the current situation and,

hence, their utility in program planning and management. For example, if

any of the policy changes currently being considered are implemented—

such as the consolidation of food assistance programs—up-to-date

information at regular intervals before and after the change will be needed

to monitor any positive or negative effects of such policy changes on the

population.

A second consequence of pauses in data collection is that potential

efficiencies of a continuous survey operation are lost. Each

implementation of a national survey requires extensive planning, including

reviews of the needs of data users, development and testing of data

collection procedures, and all the steps involved in approving a contract.

An ongoing data collection operation could streamline some of these

processes. In addition, when surveys are not in the field continuously or

even at dependable intervals, they may attempt to meet as many of the

needs of data users as possible when they are administered. For example,

the low response rates of the 1987-88 NFCS have been partially attributed to

the burden on survey respondents resulting from its attempt to obtain both

household and individual data with one interview. In contrast, an ongoing

survey could consist of a core set of questions and rotating modules of

questions that address the needs of specific users.

The continuous collection of data was a theme in all four of our sources

for the features of a model program. Past evaluations of nutrition

monitoring have called for the continuous collection and timely release of

nutrition-related data. Although the 10-year plan does not list specific

activities focused on the continuous collection of data, two of the plan’s

objectives indicate the Interagency Board’s concern with the timeliness of

the data collected at the federal and state and local levels. The expert

advisers also emphasized the need for regularly available data and

suggested the following mechanisms for the ongoing collection of the

information:

• continuous national nutrition surveys,

• addition of nutrition-related modules to existing surveys,

• reliance on program data that are already collected, and

• collection of longitudinal data (that is, data collected from the same

sample over time).

Finally, responses to our survey of the users of nutrition data not only

stated a desire for continuous data, but also indicated that data that are

Page 25 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

collected at regular and frequent intervals serve some important purposes.

For example, data that are currently available on a regular basis have been

used to measure progress toward the Healthy People 2000 objectives and

to evaluate policies such as the fortification of infant formula.3

Support for Reliable

Inferences About

Subpopulation Groups and

Small Geographic Areas

The need for continued and improved information on subpopulation

groups and small geographic areas is supported by several arguments.

First, information is needed on specific populations known to be at risk

for nutrition-related problems, such as Native Americans or homeless

persons, in order to identify their needs and develop and target assistance

programs. Second, some subpopulations, including Hispanics and the

elderly, are growing rapidly. Their dietary patterns or nutritional needs

may be different from the population as a whole; thus, information about

these groups is needed to monitor their needs and to understand their

effect on estimates of the prevalence of various nutritional problems in the

overall population. Finally, the samples for the three major NNMRRP surveys

are designed to yield national estimates. However, much of the planning

for health and nutrition programs is conducted at the state and local

levels. Hence, states and localities also need information on

nutrition-related indicators for their populations.

For the reasons given above, past evaluations of federal nutrition

monitoring have criticized the program for not covering specific

population groups and geographic areas. Although none of the overall

objectives of the 10-year comprehensive plan focus on the need for

information on subpopulation groups, several of the activities do. For

example, one planned action is to develop and implement a plan for

improved coverage of groups at nutritional risk. Our advisory panels also

discussed the importance of information on subpopulations, noting

differences in the kinds of foods consumed in different regions of the

country and by different ethnic and racial groups as well as differences in

nutritional needs at different ages. Their suggestions for improving the

availability of data on subpopulation groups and small geographic areas

included different sampling strategies for different populations, contracts

with states and localities to gather information on geographically-based

populations, and indirect estimation to support inferences about

subpopulation groups and small geographic areas.

3

The Healthy People 2000 objectives, published in 1990, are targets for improving the health of the

population by the year 2000. The targets are phrased in specific terms and with reference to baseline

information so that progress toward the objectives is measurable. Nutrition-related objectives address

such issues as overweight, breast-feeding, consumption of fruits and vegetables, iron deficiency, and

others.

Page 26 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

The availability of data on important subpopulations and states and

localities was one of the changes to the current data collection activities

requested by respondents to our survey of data users. The users also

emphasized the importance of information on subpopulations by

describing the uses supported by currently available data on

subpopulation groups such as determining dietary needs of the elderly,

assessing differences between blacks and whites in the effect of obesity on

diabetes, informing policies on the fortification of infant and toddler

foods, and targeting a blood pressure screening program to the

Mexican-American population.

Assistance to States and

Localities for Nutrition

Monitoring

State and local governments are interested not only in the applicability of

federal nutrition monitoring data to their jurisdictions, but also in having

federal assistance in collecting and interpreting their own data. The major

justification for the emphasis on state and local monitoring is the wide

range of uses that states and localities have for nutrition data. The

examples in table 2.2, drawn from responses to our survey of data users,

illustrate the utility of existing NNMRRP data for state and local

governments. While NNMRRP data collection systems meet some of the state

and local needs for nutrition monitoring data, state and local officials have

called for additional technical assistance in analyzing and interpreting

existing sources of data and for federal support in collecting their own

data.

Page 27 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

Table 2.2: Examples of State and Local

Uses of Nutrition Dataa Purpose Specific use

Problem identification Determine nutritional habits of school-age children

Identify which nutrition problems are most prevalent

among WIC participants

Assess fruit and vegetable consumption and need for

intervention

Program planning or

policy-making

Plan cardiovascular interventions for state residents with

diabetes

Establish anemia as a priority issue for nutrition education

Develop a program targeted toward the prevention of

obesity in children

Seek iron fortification of flour used in ethnic and large

county bakeries

Launch a “5-a-Day” program to encourage consumption

of fruits and vegetables

Select ethnic target groups for Preventive Health Block

Grants

Map anemia rates to identify highest risk areas for

targeting resources

Program evaluation or

program management

Monitor progress toward Healthy People 2000 objectives,

such as improvement in rates of breast-feeding

Assess access to food assistance

Justify procurement of hematological equipment

Expand nutrition technical assistance to county health

departments

aProvided by state and local officials who responded to our survey.

In addition to the support provided by the survey of data users, our other

sources indicated the importance of building state and local capacity for

nutrition monitoring. For example, past evaluations recommended

assisting state and local nutrition monitoring activities. These

recommendations are mirrored in the Interagency Board’s 10-year

comprehensive plan, which clearly signals the importance of states and

localities in the NNMRRP by devoting three of its six objectives to

strengthening state and local monitoring activities.

The expert advisers also noted the role of states and localities in nutrition

monitoring, but disagreed about the responsibility states should have for

data collection. Some argued for state-based data collection that feeds into

a federal system, and others argued for less state responsibility for data

collection, but for increased consideration of state needs in federal data

Page 28 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

collection activities. Their suggestions for assisting states and localities

include

• providing financial assistance to states to determine their own data needs,

• creating a federal-state partnership in which states can provide funds for

some extra sampling or extra questions on federal surveys,

• developing standardized modules of interest for state data collection

activities,

• assisting state collection of data on subpopulations, and

• providing technical assistance in data interpretation.

Approaches to

Achieving the

Selected Model

Features

The rest of the report describes the strengths and limitations of current

and potential approaches to achieving the selected model features. These

approaches are listed in table 2.3. The alternate strategies were selected

from the suggestions generated by the expert advisers using the criteria of

responsiveness to criticisms of the current approach and feasibility. For

example, the current approach to coordination—the Interagency

Board—is criticized for its lack of authority over the member agencies. In

contrast, an independent central authority could have influence over the

NNMRRP agencies. The options were considered feasible if they were

already used in other programs with similar issues (such as the lead

agency approach for other cross-agency programs), past activities of the

NNMRRP (such as the special study approach for information on

subpopulations), or related current activities by NNMRRP agencies (such as

indirect estimation).

Page 29 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 2 

A Model Nutrition Monitoring Program

Table 2.3: Features of a Model

Program and Current and Potential

Approaches

Model feature Current approach Potential option

Coordinated system of

activities

Interagency Board and

Working Groups

Coordination through a

central authority

Coordination from a single

lead agency

Continuous collection of data State-based surveillance

systems

Continuous

nationally-representative

survey

Addition of nutrition

questions to existing

ongoing surveys

Support for reliable

inferences about

subpopulation groups and

small geographic areas

Oversampling of selected

groups in the national

surveys

Regular, separate studies of

subpopulations

State-based surveillance

systems

Development of indirect

estimation programs

Assistance to states and

localities

State-based surveillance

systems

Community-based nutrition

monitoring

Page 30 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

As described in chapter 2, a model program would have a mechanism for

coordinating the various nutrition monitoring activities to maximize the

utility of the data and minimize the costs of its collection. This chapter

first reviews the status of current NNMRRP activities to improve

coordination. Then, two other possible coordination mechanisms—

coordination by a central authority and coordination by a single lead

agency—are examined. (Table 3.1 provides an overview of the strengths

and limitations of the various approaches to coordination.)

Table 3.1: Approaches to Coordinating Nutrition Monitoring Activities

Approach Strength Limitation

Current: Interagency Board and Working Groups Improved communication and

cooperation among the agencies

Compilation of a coordinated budget

report

Difficulties setting priorities across

agencies

Inadequate assessment of the needs

for data by different kinds of users

Difficulties in disaggregating the costs

of the nutrition component of

multipurpose programs

Limited authority, staff, and financial

resources for coordination

Potential

Coordination through a central authority Increased ability to establish priorities 

Alignment of agency priorities with

overarching goals through centralized

budget reviews

A central contact point for state and

local governments, interest groups,

Congress, and other federal agencies

Conflict between research concerns

and policy concerns

Difficulties in disaggregating the costs

of multipurpose programs

Limited staff and financial resources for

coordination

Coordination from a single lead agency Improved communication and

cooperation among the agencies

Compilation of a coordinated budget

report

A central contact point for state and

local governments, interest groups,

Congress, and other federal agencies

Difficulties setting priorities across

agencies

Difficulties in disaggregating the costs

of multipurpose programs

Limited authority, staff, and financial

resources for coordination

Decreased attention to the data needs

of other agencies

Current Approach: the

Interagency Board

As described in chapter 1, the NNMRRP meets multiple needs for

nutrition-related data. Yet, historically, these needs have not been met by

Page 31 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

an integrated program with the capacity for evaluating data needs and

making adjustments as those needs change. Instead, a fragmented system

of activities developed over the decades as new needs for nutrition data

were identified. For example, in the early 1930s, USDA developed its first

national survey of household food consumption because data on the food

supply provided no information about the distribution of food at the

household and individual levels. Similarly, the nutrition component was

added to the National Health Examination Survey in the early 1970s in

response to a need for more information about hunger, and state-based

surveillance systems were established in recognition of the primary role of

states in providing services to populations at risk of nutritional problems.

To address concerns about the lack of coordination across the agencies

involved in nutrition monitoring, the National Nutrition Monitoring and

Related Research Act of 1990 required the Secretaries of Agriculture and

Health and Human Services to implement a coordinated program of

nutrition monitoring. The act specified several tools: an Interagency

Board, the development of a comprehensive plan for the program, a

council of outside advisers, and an integrated budget.

The Interagency Board created by the act has the difficult task of

coordinating numerous data collection and analysis activities across

several agencies that have traditionally had separate and distinct missions

and operations. The Board has two chairpersons, one selected by the

Secretary of Agriculture and one by the Secretary of Health and Human

Services. For USDA, the chair is the Under Secretary for Research,

Education, and Economics. The HHS chair is the Assistant Secretary for

Health. Membership on the Board includes representatives of various

agencies in USDA and HHS, as well as the Bureau of the Census, Agency for

International Development, Bureau of Labor Statistics, Department of

Defense, Department of Veterans Affairs, and Environmental Protection

Agency, among others. The Executive Secretary for the Board rotates

between USDA and HHS every 2 years. To facilitate coordination, the

Interagency Board established working groups focused on survey

comparability, food composition, and federal-state linkages and

information dissemination.

The Secretaries and the Interagency Board were charged with developing

a 10-year strategic plan, which was published in the Federal Register on

June 11, 1993. The plan outlines a set of planning activities, including a

general time frame and the lead agencies for each activity. The activities

are organized around six objectives, listed in chapter 1 (see p. 14). The

Page 32 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

Interagency Board clearly recognizes the need for improved coordination

since four of these objectives focus on either the coordination of the data

collection activities or the comparability of the data.

To advise the Board on the development and implementation of the

NNMRRP, the act established the National Nutrition Monitoring Advisory

Council. The members of the Council represent academic institutions and

other interested parties drawn from outside the federal government. The

act also required the Interagency Board to submit annually a coordinated

budget for nutrition monitoring.

Strengths Both the concern that preceded passage of the act and the structure it

created appear to have improved communication and cooperation among

the agencies. The Board and its working groups provide mechanisms for

communication and joint decision-making. Specific actions that

demonstrate the increased coordination include the development of

• common population descriptors for use in conducting and reporting the

1994-96 CSFII and the next NHANES,

• a marketing and distribution plan for NNMRRP reports,

• an automated dietary intake interview that would facilitate timely data

release and linkage across CSFII and NHANES, and

• a common set of questions on food security (a concept that addresses the

certainty about having enough to eat) to be used in the Current Population

Survey.

In addition, a jointly funded research project explored the possible linkage

of CSFII and NHANES sampling plans. Alternate sampling designs were

evaluated using the criteria of (1) ability to satisfy the separate objectives

of the two surveys, (2) benefits in overall costs or analytic power, and

(3) feasibility, especially in terms of burden on the survey respondents.

The draft report from the contractor on the project emphasized the

compromises one or both surveys would have to make to link their sample

designs.1

 For example, while the combination of the NHANES and CSFII into a

single survey could yield a rich database of information on diet and health,

the likely increase in respondent burden could reduce response rates and

response quality. Another alternative—using linked samples for NHANES

and CSFII—could decrease CSFII’s precision because NHANES’ sampling

design is determined by the survey’s reliance on mobile examination

1

An Evaluation of Linked Survey Designs for NHANES and CSFII, prepared for NCHS by Westat,

July 29, 1994.

Page 33 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

centers to conduct physical exams of the respondents.2 The Interagency

Board concluded that the two surveys should remain independent,

although work on improving the comparability of the data should

continue.

As required by the act, the Interagency Board submits an annual budget

for the NNMRRP to the Congress. The budget report includes costs allocated

for data collection, related research, information dissemination and

exchange, and technical assistance; however, these different types of costs

are not distinguished. Instead, the funds dedicated to nutrition monitoring

and related activities are reported only by agency, not by type of activity.

The budget report for fiscal years 1994-96 indicated that a total of

$157.7 million was dedicated to nutrition monitoring or related research in

1994. Of that, $30 million was reported by the Centers for Disease Control

and Prevention (which has responsibility for the HHS surveys and

state-based surveillance systems) and $9.3 million was reported by the

Human Nutrition Information Service (which had responsibility for NFCS

and CSFII).3 The remainder was accounted for by agencies whose primary

involvement in nutrition monitoring is related research. For example, HHS’

National Institutes of Health reported $25.9 million dedicated to NNMRRP

activities and USDA’s Agricultural Research Service accounted for

$50.7 million.

The budget report is useful in communicating to the Congress a general

sense of the cost of the NNMRRP across agencies. However, funds for

nutrition monitoring cannot always be disaggregated from other purposes

of the data collection and research programs. As a result, the budget

report contains only approximate amounts dedicated to nutrition

monitoring and related research. Moreover, with the recent incorporation

of the office responsible for NFCS and CSFII into ARS, determining which

funds are dedicated to the monitoring activities and which are used for

related research will be more difficult.

Limitations The literature on the development of objectives to increase accountability

for program results indicates that (1) objectives should be written in terms

that can be used to judge progress toward achieving them and

(2) implementation plans and specific measures of progress should be

2

HHS maintains only three of these expensive mobile units and, as a result, has a sampling design that

minimizes the geographic dispersion of respondents. However, HHS is reviewing alternatives to the

mobile examination centers.

3

The Human Nutrition Information Service and its functions have been merged with the Agricultural

Research Service (ARS).

Page 34 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

developed for the goals and objectives.4 In the 10-year plan, the objectives

are stated in general, global terms so that it is not clear when an objective

can be considered achieved.

For example, the first objective is to “provide for a comprehensive NNMRRP

through continuous and coordinated data collection.” Neither the

objective itself, nor the text following it clearly defines what the terms

“comprehensive” or “coordinated” mean. Similarly, the activities are too

vague to be considered implementation plans for the objectives; for

example, no activity directly relates to the development of continuous data

collection.

As another illustration, the activity, “identify ways to increase

comparability within a dietary method to improve the quality and

usefulness of data,” specifies neither the degree of quality required nor the

uses that will be facilitated. Without more concrete, measurable

objectives, there is little accountability for the program because progress

toward the objectives cannot be assessed. (Other examples of activities

listed in the 10-year plan can be found in table II.2 in appendix II.)

In addition, the plan did not describe how activities would be ranked by

importance or addressed within current fiscal constraints. However, since

the plan was published, the Interagency Board drafted an approach for

ranking the 68 activities listed in the plan into three categories of

priorities: (1) essential (mandatory, legislatively required), (2) necessary

(critical but not mandatory), and (3) beneficial. An interagency

implementation group, involving around 60 agency representatives,

applied the approach. Twenty-five of the activities were ranked in the

high-priority category, 33 were ranked as next most important, and only 10

were ranked as beneficial but not critical. While this is an important first

step in setting priorities, the Interagency Board has not yet linked the

top-ranked tasks to the costs and benefits of completing them. Such a

framework could be used to understand the trade-offs in selecting one or

another approach to each task.

Since an assessment of the effectiveness of the different approaches to the

objectives requires information on the uses of the data, another obstacle to

the Interagency Board’s implementation of a coordinated system is the

absence of a comprehensive assessment of how nutrition monitoring data

are used across the federal government and by data users in other settings.

4

Performance Measurement: An Important Tool in Managing for Results (GAO/T-GGD-92-35; May 5,

1992).

Page 35 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

For example, although USDA and HHS held a joint workshop to assess the

needs of users of dietary intake data in August 1994, the workshop

included only representatives of federal agencies.

Finally, the ability of the Interagency Board to coordinate the NNMRRP is

limited by the lack of resources dedicated to coordination. The

Interagency Board has no staff, although two people—one from USDA and

one from HHS—have been given primary responsibility for organizing

NNMRRP activities. The NNMRR Act gives the Secretaries the option of

appointing an administrator for the program, but so far they have chosen

not to exercise that option.

Potential Options While recognizing the progress made by the Interagency Board, we also

considered other coordination mechanisms. Two options for improving

coordination—through a central authority or by a lead agency—are

reviewed in detail in this section. In addition, other options discussed by

our expert advisers are briefly presented.

Coordination Through a

Central Authority

The suggestion of coordination through a central authority came in

response to the lack of enforcement power held by the Interagency Board

over the member agencies. The kind of central, coordinating agency

envisioned by the expert advisers is most clearly exemplified by the

executive offices in the White House. Therefore, to examine the

advantages and disadvantages of having a central authority provide

coordination, we reviewed the literature (including prior GAO reports and

congressional hearings) and interviewed agency officials in three White

House offices that have coordination responsibilities: Office of

Management and Budget (OMB), Office of Science and Technology Policy

(OSTP), and Office of National Drug Control Policy (ONDCP). As indicated by

the brief descriptions of each office provided in table 3.2, the coordination

tools used by the White House offices are similar to those used by

NNMRRP’s Interagency Board, including interagency committees and

working groups, development of plans, and review of budgets.

Page 36 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

Table 3.2: Responsibilities and

Coordination Tools of Three White

House Offices

Office Responsibility Tool

OMB Ensure coordination and

cooperation where federal agency

lines of authority overlap

Assist in the development of

efficient coordinating mechanisms

Expand interagency cooperation

Review and approve agency

budgets

Review and approve data

collection instruments

OSTP Coordinate research and

development activities in other

agencies

Review and participate in the

annual formulation of the budgets

related to science, research, and

development

National Council on Science and

Technology (NCST), which is

chaired by the President and

includes Cabinet Secretaries and

selected agency heads

NCST subcommittees and working

groups

Review of agency budgets for

interagency programs

Commissioning studies of

controversial issues

ONDCP Through the National Drug Control

Strategy, establish policies,

objectives, and priorities for federal

drug control programs

Coordinate federal budget for drug

activities

Interagency working groups

Review of agency plans for

implementing the National Drug

Control Strategy

Review and certification of drug

control budgets

Potential Strengths One potential advantage of elevating the coordination of nutrition

monitoring to a high-level central authority is the increased participation

of high-level officials in coordination activities. This participation could, in

turn, increase the ability of the coordinating body to establish priorities.

Currently, some of the agencies participating in the Interagency Board are

represented by administrators or directors, while other agencies are

represented by staff members who have no authority to change agency

activities or establish priorities. In contrast, the political visibility of a

program under White House management could encourage agencies to

send representatives in positions of authority, capable of establishing

priorities and committing resources to support them.

The political visibility that comes with the participation of a high-level

central authority may also contribute to the effectiveness of the various

coordination tools. For example, the Interagency Board compiles budgets

Page 37 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

obtained from each of the agencies involved in nutrition monitoring

without reviewing them for consistency with the activities identified in the

10-year comprehensive plan. In contrast, the White House offices use the

budget review process to bring activities of the agencies into line with

overarching policy goals. For example, ONDCP can threaten to decertify an

agency’s drug budget if it is not consistent with the National Drug Control

Strategy. Decertification has no practical ramifications, but it sends a

politically important message about the priority given to drug control

activities by the White House.

An additional potential advantage of having a central authority provide

coordination is that it can be a central location for assistance to all data

users. For example, ONDCP has a Bureau of State and Local Affairs that

works with state and local government agencies involved in drug control

activities. The Bureau serves as a clearinghouse for information about

state and local activities and uses conferences to increase communication

with and among state and local officials. In addition, the Bureau can

communicate the concerns of state and local governments and community

groups to the federal agencies involved in drug control programs. For

nutrition monitoring, such an office could provide a central contact point

for users in a variety of settings, including federal, state, and local

governments, food industry, and health care organizations.

Potential Limitations Although the increased political visibility of a high-level central

coordination office may facilitate the development of priorities, it may also

increase the potential for political pressure on the data collection and

research. A conflict between ONDCP and HHS over the data collection and

reporting of drug data illustrates this issue. HHS was concerned about the

degree of ONDCP’s involvement in how the data were collected and

reported, while ONDCP expected HHS to meet its data needs. Political

influence on the scientific agenda is also a concern for OSTP, where

priorities may change with changing administrations even if scientific

issues remain the same.

In addition, although White House offices have more influence over the

budgets reported by the different agencies, the budgets developed for

other programs that cut across agency jurisdictions share the limitation of

the coordinated NNMRRP budget. Specifically, agency activities may serve

multiple purposes, thus making it difficult to determine how much of the

overall costs are dedicated to the interagency program. For example,

Coast Guard patrol boats serve drug interdiction purposes, but are also

used in search and rescue missions that are not related to drug control.

Page 38 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

Thus, the Coast Guard can only estimate the portion of its resources

dedicated to supporting national drug control efforts. Ambiguity about the

portion of a multipurpose program that serves the interagency purpose

makes it difficult to monitor the costs of the program.

Although central authorities like the White House offices are financed

separately from the agencies they oversee, resource limitations persist.

For example, HHS staff attributed part of their conflict with ONDCP to a lack

of technical and substantive expertise among ONDCP staff. Similarly, a past

director of OSTP identified limited staff resources as a reason why

long-term planning received less attention than short-term problems.

Coordination Through a

Lead Agency

Another possible approach to coordinating the NNMRRP is locating the

responsibility within a single agency. To investigate this approach, we

discussed the option with our expert panels and used program documents

and evaluations conducted by GAO and the Congressional Budget Office to

examine the experience of the High Performance Computing and

Communications Program (HPCCP).5

 Like the NNMRRP, the HPCCP involves

multiple agencies with different strengths and missions. Unlike the

NNMRRP, the oversight of the HPCCP is located in OSTP, which delegated the

responsibility for coordinating the activities across the agencies to a single

agency, the National Library of Medicine. While not a major player in

high-performance computing, the National Library of Medicine was seen

as an independent, unbiased participant with interest in and knowledge

about the technology. The Library of Medicine’s role has been to pull

together materials for program reports, convene meetings, and provide a

clearinghouse.

Potential Strengths HPCCP’s National Coordination Office shares two of the strengths of the

Interagency Board: It appears to have facilitated communication among

the agencies and coordination of individual activities. In addition, it has

provided the Congress with a budget that looks at the costs of highperformance computing activities across agencies. Moreover, the National

Coordination Office has the added advantage of providing the Congress

and the public with a central contact point for information on

high-performance computing.

Potential Limitations Just as the HPCCP shares the NNMRRP’s strengths, it also has some of its

limitations. Specifically, no mechanism is in place to set priorities across

5

The purpose of HPCCP is to further the development and dissemination of U.S. supercomputer and

high-speed computer network technologies.

Page 39 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

different agency activities. In a recent review, GAO suggested that the HPCCP

needs

“an explicit technical agenda, identifying and prioritizing specific technology challenges

and establishing a framework of expected costs and results . . . [to] clarify the program’s

goals and objectives, focus efforts on critical areas, and serve as a baseline for measuring

program progress and results.”6

In addition, HPCCP did not have uniform guidelines for which research

activities should be included in the budgets submitted by the different

agencies, mirroring the difficulty the other coordination mechanisms have

in tracking funds used for multipurpose programs. The National

Coordination Office also shares the NNMRRP’s lack of budget and staff

resources for coordination.

An additional concern about the lead agency model was raised by our

expert advisers: If responsibilities for nutrition monitoring were located in

one agency, nutrition monitoring might become a monopoly, serving the

needs of only one agency. Its current dispersion across agencies allows the

different components to serve different purposes. However,

safeguards—such as that used for the HPCCP when it was located in an

agency that did not have a large investment in high-performance

computing, relative to some of the other agencies—could address this

concern.

Other Coordination

Suggestions

In addition to coordination by an interagency body with permanent staff

and enforcement authority or a single lead agency, other suggestions for

improving coordination of the NNMRRP were to locate nutrition monitoring

in statistical agencies within the user departments, centralize the

congressional appropriations process for nutrition monitoring activities,

and ensure informed review of data collection plans by qualified staff at

the Office of Management and Budget.

The first idea—locating nutrition monitoring in statistical agencies within

the departments that use the data—was intended to focus the program on

the quality of the data collected. However, a concern about this suggestion

was that it could result in decreased responsiveness to the needs of the

data users. We did not pursue it because the major surveys are already

located in statistical or research branches of HHS and USDA. (NHANES is

6

High Performance Computing and Communications: New Program Direction Would Benefit From a

More Focused Effort (GAO/AIMD-95-6; Nov. 4, 1994), p. 22.

Page 40 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 3 

Approaches to Coordinating the Nutrition

Monitoring Program

operated by the National Center for Health Statistics, and NFCS and CSFII

were recently relocated to ARS.)

Similarly, OMB already has responsibility for reviewing the data collection

activities of the agencies, although OMB staff reported that they rely on the

agencies to describe coordination efforts. Finally, we did not review the

advantages and disadvantages of centralizing the congressional

appropriations process for nutrition monitoring activities because its

relevance to the utility and efficiency of the data collection activities was

not clear.

Conclusions Lack of coordination across program activities has implications for both

the effectiveness and the efficiency of the NNMRRP as a whole. The

Interagency Board has made progress toward coordinating activities

across the different agencies. However, the other approaches to

coordination suggest mechanisms that could further strengthen the work

of the Interagency Board.

First, because its responsibilities are shared by USDA and HHS, the

Interagency Board does not provide a central contact point for users of

nutrition monitoring data. A central contact could be established if the

Secretaries of USDA and HHS used their option to appoint an administrator

of the program or if responsibility for responding to requests for

information about the program was assigned to a single agency, as it was

for high-performance computing.

Second, the Interagency Board does not review the agency budgets it

compiles for consistency with the overarching priorities of the program.

Before such a review could occur, specific objectives and priority

activities would need to be identified. Then, the Interagency Board could

work with OMB to secure funding of NNMRRP priorities.

Page 41 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

Evaluating programs intended to reduce diet-related chronic disease,

tracking progress toward health objectives, and monitoring changes in our

diets are the kinds of activities that require continuous data. The

advantages and disadvantages of the current and alternate approaches to

collecting continuous data are summarized in table 4.1 and detailed below.

Table 4.1: Approaches to Providing Continuous Data

Approach Strength Limitation

Current: state-based surveillance systems Quick availability of data on specific issues

Connection between data elements and

program decisions

Efficient relative to national surveys

because of the reliance on program

records (PedNSS and PNSS) or telephone

interviews (BRFSS)

No in-depth food intake or health data

Uneven quality and completeness of data

across states

Limited to populations participating in

publicly funded programs (PedNSS and

PNSS) or living in households with

telephones (BRFSS)

Incomplete participation of states in

PedNSS and PNSS

Potential

A continuous, nationally-representative

survey

Up-to-date information on the population

Increased flexibility to respond to new

issues by adding modules to a core set of

data elements

Increased efficiency as start-up costs

diminish

Costs of conducting all survey processes

concurrently

Difficulties in identifying a core set of data

elements

Inflexibility of core data elements

Addition of nutrition questions to existing

ongoing surveys

Efficient means of gathering data at

frequent intervals

Dependent on flexibility of surveys

Current Approach: the

State-Based

Surveillance Systems

Although the provision of continuous data is one of the objectives

identified in the Interagency Board’s 10-year plan, not one of the three

major NNMRRP surveys—NFCS, CSFII, or NHANES—is implemented

continuously. In fact, planned future administrations of two of the surveys

have been postponed, potentially compromising the ability to monitor

trends in diet-related health risks over time and evaluate the effect of any

changes in food assistance policy.

NFCS has been administered at approximately 10-year intervals since the

1930s. Its next implementation was planned to begin in 1996, but is now

tentatively scheduled for 1998, depending on funding. CSFII, originally

Page 42 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

intended to provide continuous data on dietary intake, has had three

separate administrations: 1985-86, 1989-91, and 1994-96. After a 1-year

pause in 1997, it is expected to resume for another 3-year period in 1998.

Since the National Health Examination Survey gained a nutrition

component in 1971, NHANES has been fielded three times: 1971-75, 1976-80,

and 1988-94. Like NFCS, its future implementation is uncertain because of

budget constraints. Its planned implementation in 1997 is now expected to

be postponed.

Because of the lack of certainty about the implementation of the national

nutrition surveys, the state-based surveillance systems are currently the

primary source of continuous data in the NNMRRP. As described in chapter

1, PedNSS and PNSS rely on data from clinic records from publicly funded

health, nutrition, and food assistance programs and BRFSS collects

information through telephone interviews, with respondents (adults 18

years and over) identified through random digit dialing. While a valuable

source of quick information for state and local program managers, the

surveillance systems do not meet the needs of researchers or program

decisionmakers who require either national data or in-depth food intake

data.

Strengths Compared to the national surveys, one of the strengths of the state-based

surveillance systems is that they not only provide data continuously, but

they are also able to process and report the data relatively quickly. For

PedNSS and PNSS, information is collected as part of the process of receiving

services from WIC and other publicly funded health, nutrition, and food

assistance programs. Because they depend on program records, PedNSS and

PNSS do not burden respondents the way surveys dependent on interviews

do. The information is transmitted from the records of local health and

nutrition programs to the state, which then forwards the records to HHS for

analysis. Similarly, the data collected by the states for BRFSS are sent to HHS

for processing. According to HHS officials, all three systems report data

back to the states within a year and generally in less than 9 months. HHS

has also helped states conduct their own analyses by distributing a

standardized software package.

Another strength of the state-based surveillance systems is that the data

they collect are directly linked to program decisions. Although PedNSS and

PNSS include only a few indicators of nutritional deficiencies and

behaviors, they are selected to support state data needs for program

planning and management. For example, the data are used to target

Page 43 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

resources for the WIC program. Similarly, BRFSS data have been used to

inform decisions about nutrition education programs, such as campaigns

to encourage the consumption of five servings of fruits and vegetables a

day.

Limitations In contrast to the national surveys, the surveillance systems do not permit

examination of diverse diet-health associations across the entire

population. Instead of collecting extensive biochemical, anthropometric,

and interview data, they focus on a narrow range of variables relevant to

specific programs or nutritional risks. For example, PedNSS collects clinical

data on weight and height, monitors infant feeding practices, and assesses

anemia. PNSS collects information on anemia and behaviors associated

with low-birthweight babies. BRFSS asks respondents to report on their

consumption of fat, fruits, and vegetables. This focus limits the breadth of

uses that can be supported by the data; however, as noted above, it also

limits the burden placed on respondents.

While the systems are currently limited in the amount of dietary data they

collect, HHS is exploring other methods of gathering these data. For

example, with HHS support, the University of Texas examined the use of

bar code data to look at dietary patterns. They concluded that the

technology is not yet ready for use, but through a partnership with food

manufacturers could be a promising method for the future. In addition,

with USDA, HHS is evaluating the feasibility of collecting additional dietary

data in the clinics that provide the PedNSS and PNSS records.

Another concern about surveillance systems is the quality and

completeness of the data across the different states. For example, for

PedNSS, error can be introduced by variations in practice in weighing

infants, such as with or without the baby’s winter clothes, or by clerical

errors in entering the data in states that do not have automated data

systems. PNSS, which attempts to collect a wider range of information than

PedNSS, suffers from missing data on several variables, such as the

pregnancy risk factors of smoking and alcohol consumption. However, HHS

provides technical assistance to help states standardize their data

collection procedures. In fact, by flagging biologically implausible values

for the physical measures, HHS analyses of the surveillance data help

identify clinics that may have poor procedures.

The surveillance systems are also limited because, within the participating

states, only certain groups of the population are covered. As described

Page 44 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

earlier, PedNSS and PNSS primarily provide information on mothers and

children participating in the WIC program. As a result, data may not be

available on other populations that are potentially at risk, such as

homeless people or older children not eligible for WIC. In contrast, BRFSS

has a wider target population, collecting data from randomly selected

adults 18 years and over. However, neither adults in households without

telephones nor children are covered by BRFSS. Since there is evidence that

some health risk factors, such as smoking, are associated with living in a

household without a telephone, this could affect estimates of the extent of

diet-related risk factors as well.

Finally, not all states participate in the PedNSS and PNSS surveillance

systems and, therefore, they are not good sources of national-level data on

their populations of interest. In 1993, 38 states participated in PedNSS and

only 20 states participated in PNSS. (In contrast, BRFSS has good state

coverage; in 1993, only Wyoming did not participate.)

Potential Options In addition to the state-based surveillance systems, we reviewed two other

approaches to providing continuous data. The first option is a national

nutrition-related survey that is operated continuously. To describe the

strengths and weaknesses of this approach, we consulted with our expert

advisers and interviewed managers of the current national nutrition

monitoring surveys about the continuous operation of the surveys. The

second option is the inclusion of nutrition-related questions on continuous

non-NNMRRP surveys. The strengths and weaknesses of this alternative

were explored by examining the NNMRRP’s recent experience developing

food insecurity questions for the Current Population Survey (CPS). In

addition, our expert advisers and the data users who responded to the

survey made other suggestions, which are also briefly presented.

A Continuous NNMRRP

Survey

An ongoing national survey is a possible approach to providing

continuously updated information that addresses some of the limitations

of the state-based surveillance systems. One of the current national

surveys, the CSFII, was developed in response to calls for the continuous

collection of individual data on food intake. It has not been implemented

regularly in the past; however, current plans are for the ongoing

implementation of CSFII as a 3-year survey followed by a 1-year pause for

planning and development before the next 3-year period.

Page 45 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

Potential Strengths An ongoing survey has the potential to yield continuously updated, timely

data for monitoring the nutritional status of the nation’s population. The

increased timeliness of the data could decrease the current pressure from

collaborating federal agencies to include components on each

implementation of the periodic surveys. For example, in contrast to the

recent implementation of NHANES, which attempted to meet as many data

needs as possible within a single survey, the ongoing implementation of

NHANES could contain a core set of data items that would be collected

continuously, supplemented at intervals by rotating modules. Such a

streamlining would also reduce the burden on respondents.

Moreover, the surveys could become more flexible by distinguishing

between variables that change rapidly, variables that need regular but not

continuous monitoring, and variables of emerging policy importance. The

core set of items in continuous implementation could gather information

on rapidly changing variables. Topical modules on issues that do not

change as rapidly could be included on a regular schedule. Finally, as new

issues arise, additional questions could be added.

In addition to increased timeliness and flexibility, a continuous survey

operation could be more efficient than periodic surveys because current

costly start-up activities, such as planning, designing sampling strategies,

and training interviewers, would be diminished. Thus, the data could be

collected for less cost per respondent. However, as described below,

without concomitant streamlining of the survey, overall costs could

increase.

Potential Limitations While an ongoing survey could save money on start-up costs, it could

increase costs overall because activities that are now funded and staffed

sequentially would require a continuous flow of resources to be conducted

concurrently. For example, as NHANES is currently implemented, staff

change their activities as the survey moves through the phases of planning,

implementation, analysis, and dissemination. If NHANES was implemented

continuously, all of these activities would be going on at the same time.

According to both USDA and HHS officials, the primary constraints on the

continuous operation of their surveys are the need for dependable funding

and sufficient staff resources. As described above, the absence of

dependable funding has affected the frequency with which the national

surveys are now implemented.

The development of a survey that continuously collects data on a core set

of items and intermittently collects data on other issues raises two difficult

Page 46 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

issues. First, the definition of the core items is complex. An expert panel

convened by the Federation of American Societies for Experimental

Biology was charged with identifying a set of core indicators to assess the

nutritional status of difficult-to-sample populations. The report

summarizing the work of the panel noted that (1) the suitability of an

indicator changes with the purposes for the data and (2) information on

the determinants and the consequences of each indicator is also needed.1

The panel ended by identifying three sets of indicators—minimal,

intermediate, and comprehensive—without recommending specific

measures for the indicators.

The second issue with a continuous survey is the potential inflexibility of

the core items once they are selected and implemented. While

opportunities to test new methods are enhanced because the survey is in

the field continually, making changes to the data collection procedures

can be difficult because of the pressure to ensure that the measures are

consistent over time. Without such assurance, changes in an indicator

such as obesity may be the result of changes in how obesity is measured

rather than changes in the prevalence of the condition itself. Our expert

advisers suggested a survey with built-in periods of transition to allow the

survey to incorporate new methods and new data elements as they

emerge.

Adding Nutrition

Questions to Existing

Continuous Surveys

A third approach to obtaining frequently up-dated information is the

addition of nutrition-related questions to existing continuous surveys. The

potential of existing surveys to provide data regularly on some variables is

demonstrated by plans that included NNMRRP questions on food security in

the April 1995 Current Population Survey conducted by the Census

Bureau.2

 Food security is a concept intended to go beyond the idea of

hunger to measure the availability of food for a family or individual.

The food security questions were developed by an interagency working

group cochaired by HHS’ National Center for Health Statistics and USDA’s

Food and Consumer Service. While the working group developed

questions, USDA reserved space on the CPS for 1995. The question

development process included determining how the data would be used

and soliciting input from both federal and other data users. The food

security module contains both core questions and supplemental ones. The

1

S.A. Anderson, ed., “Core Indicators of Nutritional State for Difficult-to-Sample Populations,” Journal

of Nutrition, 120:11S (Nov. 1990), 1559-99.

2

The CPS is a monthly survey of about 60,000 households across the country.

Page 47 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

Census Bureau will include the complete module. NNMRRP surveys could

include the smaller core set of questions. If the initial implementation of

the questions yields useful data, USDA plans to continue to support the

inclusion of the food security questions annually in the CPS.

The same strategy of piggy-backing nutrition-related questions on

continuous surveys could be used with other surveys, such as the Survey

of Income and Program Participation or the National Health Interview

Survey. The latter has the added advantage of collecting health data that

could be linked to nutritional indicators.

Potential Strengths The major advantage of this approach is the efficiency with which data on

specific issues can be gathered. USDA will pay for the cost that the food

security questions add to the CPS without having the responsibility or the

cost of fielding and managing the survey itself.

Potential Limitations A potential disadvantage of this approach is that, just as core questions on

nutritional status could be inflexible, existing continuous surveys can be

hard to change because of their momentum. Moreover, existing surveys

have their own set of constraints and limitations. For example, while CPS

could accommodate the addition of questions about food security, it

probably could not accommodate a module obtaining data on an

individual’s dietary intake over the last 24 hours. A 24-hour recall

instrument requires considerable training to administer and adds

substantially to the burden on the respondent. For these reasons,

opportunities to piggy-back nutrition-related questions on other surveys

may be limited.

Other Options for

Continuous Data

In addition to the approaches reviewed, other actions were suggested by

our expert advisers and data users who responded to our survey.

Specifically, longitudinal surveys were suggested as an efficient way to

collect data over time because a new sample does not have to be selected

every time a survey is fielded. Since original respondents are followed up

in subsequent administrations of the survey, longitudinal surveys can also

be useful for tracking individual-level changes in food consumption

behavior. However, longitudinal surveys have their own costs, including

the need to collect additional data so that respondents can be found for

later surveys and the likelihood of attrition as respondents either drop out

or cannot be found.

Page 48 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 4 

Approaches to the Continuous Collection of

Data

Another suggestion was the collection of survey data using automated

survey technology. Because direct entry of the data into an automated

system can speed the processing and aggregation of the survey, it can also

accelerate the release of the data for analysis. However, it does not affect

the regularity with which the data are collected in the first place. HHS

already uses automated data collection for NHANES. USDA is planning to

automate the next administration of the CSFII.

Conclusions The funding constraints that have caused the postponement of two of the

national surveys jeopardize the availability of periodic data on the

population as a whole. Although the state-based surveillance systems

provide continuous information, they are inadequate to meet the need for

data on the population at large or on in-depth nutrition and health status.

Moreover, if approved, proposals to collapse funds for the WIC program

with other food assistance programs into a block grant for the states could

affect the major source of data for two of the state-based surveillance

systems, PedNSS and PNSS.

Although the NNMRRP has pursued such creative solutions as including food

security questions on the CPS, the availability of up-to-date information

could worsen. To decide how best to meet the needs for continuous data

in the future, the NNMRRP would first need to analyze the purposes that

require frequently collected data and the current mechanisms for

supporting those purposes. Within this framework, the strengths and

limitations of the different approaches to increasing the frequency with

which important indicators are measured could be weighed according to

which purposes are supported and which are diminished.

Page 49 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

Subpopulations can be defined by geographic location as well as by age

and sex (such as infants or elderly women), physiological characteristics

(such as pregnancy), ethnicity or race (such as Hispanic or Native

American), income, and the intersection of any of these groups (such as

low-income children). As described in chapter 2, information on

subpopulations is needed to appropriately target and evaluate programs

that address nutrition-related programs. As summarized in table 5.1, this

chapter describes current approaches and some potential options to

responding to the calls for better information on subpopulation groups

and small geographic areas in the NNMRRP. Other issues of federal

assistance to states and localities for nutrition monitoring are discussed in

chapter 6.

Page 50 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

Table 5.1: Approaches to Supporting Inferences About Subpopulations

Approach Strength Limitation

Current

Oversampling of selected groups in the

national surveys

Less costly than a separate survey

Ability to compare the subpopulation to the

general population

Appropriate only for special populations

that are covered by the national sampling

frame (such as children in low-income

households)

Primarily appropriate for populations that

are geographically clustered or large

Costs of screening for members of the

group

State-based surveillance systems Collection of data on high-risk populations:

PedNSS (low-income children) and PNSS

(low-income pregnant women)

Collection of data representative of the

state population of adults, 18 years or older

(BRFSS)

Potential expansion to include programs

that serve other at-risk populations

No in-depth food intake or health data

Uneven quality and completeness of data

across states

Incomplete participation of states in

PedNSS and PNSS

Potential

Special studies Ability to tailor data collection instruments

and content to the subpopulation

Collection of data on subpopulations that

are not adequately covered by the

sampling frame used for national surveys

Inability to compare to the rest of the

population unless the special study is

conducted in tandem with a national survey

using comparable instruments

Costs of tailoring data collection

instruments, developing the sampling

frame, and screening

Difficulties in developing a sampling frame

that allows for generalization

Development of indirect estimation

programs

Efficient means of generating estimates for

subpopulations

Historical use in other areas (estimates of

population, crop yields)

Response to limitations of other sources of

information, such as cost and relevance

Difficult to assess the quality of the estimate

Diminished confidence in the estimates

because they are based on models rather

than direct observation

Current Approaches Subpopulation groups are covered in two ways by current NNMRRP

activities. First, the three national surveys use oversampling of certain

groups to ensure the selection of enough respondents to support

subpopulation estimates. NHANES focuses on racial and ethnic

Page 51 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

subpopulations, while CSFII and NFCS have included subpopulations defined

by income, reflecting USDA’s focus on food assistance to low-income

populations. The second way in which data on subpopulations are

gathered is through the state-based surveillance systems—PedNSS, PNSS, and

BRFSS.

Oversampling Oversampling includes members of subpopulation groups in a sample at a

rate greater than their proportion in the population. The purpose of

oversampling is to ensure that data will be collected on enough group

members to support inferences about the group as a whole. Oversampling

is already used in the major NNMRRP surveys. NHANES III (1988-94)

oversampled non-Hispanic blacks and Mexican-Americans, as well as

persons 60 years or older and children 1-5 years old. NHANES staff indicated

that two groups (Hispanics and persons 75 years and older) are likely to be

important groups in the next implementation of the survey because both

are growing and have significant health-related issues to study. In addition

to estimates of the general population, the current administration of CSFII

(1994-96) is expected to produce estimates for low-income populations

through oversampling. Also, staff of both USDA and HHS surveys stated that

the national surveys could oversample a state’s population, but that the

state would have to finance the added costs.

Strengths Oversampling has two major strengths. First, because it can be used in

conjunction with a national survey, it has efficiencies of scale. Specifically,

the planning and implementation costs are diminished because they are

part of a larger survey. Second, data on both the group and the rest of the

population are collected at the same time and with the same survey

procedures, facilitating the comparison of the two population groups.

Limitations Oversampling implies that the subpopulation group of interest is included

in the sampling frames used to identify participants in the national

surveys.1

 However, not all special populations are well covered by a

national sampling frame. For example, homeless individuals, persons who

live in institutions, and American Indians and Alaska Natives living on

reservations would not be included in the national household sampling

frame.

Even for those individuals who are included in the sampling frame,

oversampling may not be appropriate because of the costs incurred in

1

The sampling frame is the group of units from which a sample is actually drawn. So, for example, both

NHANES and CSFII sample from a sampling frame of households in the 50 states and the District of

Columbia.

Page 52 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

screening for members of the group. Screening is the process of asking

questions at sampled households to identify whether they represent (or

include representatives of) the subpopulation of interest. Screening adds

to the costs of the survey because enough households have to be sampled

and screened to identify the smaller number of households or individuals

that meet the definition of the subpopulation. To reduce screening costs,

oversampling is most effective for subpopulations that are geographically

clustered or fairly well represented in the general population, such as

persons with low income who are often clustered by neighborhood. In

contrast, oversampling is not appropriate for groups that are few in

number or geographically dispersed, such as pregnant women.

A possible response to some of the limitations of oversampling is the use

of multiple sampling frames. For some subpopulations, alternative frames

or lists may be available that can be used in conjunction with the national

sampling frame. Samples can be selected from both the subpopulation

frame and the general population frame, and weighted estimates can then

compensate for the fact that some group members could be selected from

two different sources. For example, to oversample for the frail elderly, the

elderly individuals identified in the sample drawn from the national

sampling frame could be supplemented by samples drawn from lists of

elderly who participate in congregate meals programs. This approach is

useful because, relative to screening in the general population, it is an

inexpensive way to identify members of the subpopulation.

State-Based Surveillance

Systems

Although identified as a limitation in the previous chapter, the focus of the

state-based surveillance systems on particular subpopulations can also be

seen as a strength. As described in chapters 1 and 4, states use BRFSS to

collect data on the health behaviors of their adult population. PedNSS and

PNSS are sources of information on the nutritional status of low-income

mothers and children. In addition, HHS is exploring opportunities to

expand the program to other populations, such as schoolchildren.

However, the other disadvantages of the systems, such as the limited

amount of nutritional data they collect and the incomplete participation of

the states, diminish their utility as a source of information on

subpopulations.

Potential Options The two major alternate approaches to providing subpopulation data are

special studies and indirect estimation. Special studies are those that use a

separate sampling frame from a national survey and are not necessarily

Page 53 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

conducted at the same time as a national survey. To describe the strengths

and weaknesses of special studies as an approach to collecting

information on subpopulations, we examined HHS’ past experience with

the Hispanic Health and Nutrition Examination Survey (HHANES) and

discussed the option with our expert advisers.

Indirect estimation uses data that are not direct observations of the group

of interest to develop inferences about the subpopulation. Our review of

this approach relied on reviews of technical literature, interviews with

USDA and Bureau of Census staff responsible for indirect estimation

programs, and consultation with methodological experts on our panel of

advisers and in HHS. These and the current approaches address the

suggestions made by the expert advisers and data users who responded to

our survey.

Special Studies: the

HHANES Model

An alternate approach to covering subpopulations is conducting a special

study. A model of this approach is the Hispanic Health and Nutrition

Examination Survey conducted in 1982 by HHS. HHS developed HHANES in

response to recommendations made by the National Academy for Public

Administration, which identified the Hispanic population as growing,

likely to have low income, and potentially at risk for health and nutrition

problems. HHANES was conducted as a separate study rather than

integrated into the national survey because NHANES II was already

completed and funding was not available to conduct HHANES as part of a

national survey.

Potential Strengths One of the advantages of conducting HHANES as a separate study was the

opportunity to change the content of the survey instrument to address

issues of special relevance to the Hispanic population. For example, unlike

NHANES II, HHANES gathered information on health services use and

gallstone disease. In addition, HHS took steps to address the cross-cultural

issues of applying HHANES to different Hispanic populations. The survey

instrument was translated into the idiomatic Spanish of each of the three

Hispanic groups surveyed (Cuban-Americans, Mexican-Americans, and

Puerto Ricans) and an appropriate plan for reaching out to the

respondents was developed. For example, unlike the regular NHANES,

which relies primarily on press releases and the formal leaders (such as

mayors) of the places they have sampled to communicate the importance

of participation in the survey, HHANES used informal leaders (such as

church leaders) and Spanish-language media. The opportunity to tailor a

special survey to the population of interest is an advantage of the

Page 54 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

approach, but it can be complicated for a population whose members

speak multiple distinct languages and have varying degrees of assimilation

into the U.S. population.

Although not demonstrated by HHANES, another advantage of a special

study is the ability to study populations that are not appropriately studied

through oversampling. As described above, this includes groups that are

not included in the national sampling frame, such as homeless persons,

and groups that are geographically dispersed or not well represented in

the general population, such as pregnant and lactating women.

Potential Limitations Unlike oversampling, in which the subpopulation is surveyed at the same

time and usually with the same procedures as the population as a whole,

the data collected by special studies may not be comparable to the

population as a whole. For example, differences found between HHANES

data on Hispanic groups and NHANES data on the nation could be the result

of national changes in health and nutrition status during the gap between

the two surveys rather than actual differences between the groups. Of

course, conducting special studies in tandem with a national survey is

possible. In fact, HHS is considering conducting special subpopulation

surveys at the same time as the next NHANES.

Special studies that are conducted in addition to the national surveys

clearly add to the overall cost of data collection. Specific costs for surveys

of ethnic populations include bilingual interviewers, the translation of the

survey instrument, and outreach to the group. Other costs are the

development of a separate sampling frame and screening potential

respondents to identify those that belong to the group of interest. Finally,

if a special study is conducted at the same time as the national survey, the

burden on the survey support facilities, such as the laboratories that

analyze blood and urine specimens, is increased, which may slow down

the processing and release of the data.

If a special study is conducted because the group of interest is not well

covered by the national sampling frame, it needs a sampling frame that

allows for generalization to the subpopulation group, which may be

difficult to construct. For example, to use survey results to draw

conclusions about the population of people who are without homes, one

needs to sample from a complete list of the members of that group. For

the homeless population, such a list would be very costly to construct. As

a result, other means—such as sampling shelters—would have to be used.

The survey results, though, would probably not be generalizable to the

Page 55 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

overall population of homeless people because these other means are

likely to be incomplete.

Indirect Estimation Indirect estimation (also known as small area or synthetic estimation)

refers to procedures that use values of the variable of interest from an area

or time other than the area and time of interest. For example, to develop

an indirect estimate of the prevalence of iron-deficiency anemia in a

county, the national estimate of iron-deficiency anemia can be adjusted

based on the county’s demographic profile. Both USDA and HHS have

experience with indirect estimation. Since the early part of the century,

USDA has had a program to develop indirect estimates of crop yields.

Although some information is available from state surveys of

nonprobability samples of farmers, the USDA program adjusts these less

dependable estimates so that they aggregate to the more reliable regional

and national estimates that are based on a survey of a national probability

sample of farmers. HHS has no regular program to produce indirect state

estimates, but since 1968, it has supported the occasional development

and evaluation of indirect estimates from National Health Interview

Survey data. In addition, HHS produced state estimates from the National

Natality and National Fetal Mortality Surveys conducted in 1980 using

demographic data from the states to adjust the national estimates. Indirect

estimation models range from the relatively straightforward adjustments

of national or regional estimates to match local demographic profiles to

more complex models.

Potential Strengths The major strength of indirect estimation is that, compared to increasing

survey sample sizes to obtain data for direct estimates at state and local

levels, it is far less costly. In addition, it is a means of extending the

usefulness of costly national survey data to inform decisions made at state

or local levels of government. Moreover, it is an established approach.

Indirect estimates of such variables as state and local populations,

employment and unemployment, and crop yields are already used by the

federal government in formulas for determining eligibility and benefit

amounts for federal programs. Some state governments have also used

indirect estimation to conduct analyses for economic and other types of

programs.

Indirect estimation also responds to limitations of the data on which direct

estimates might be based. Program records, such as those used by PedNSS

and PNSS, have the advantage of timeliness since they are usually collected

continuously, but their relevance may be limited because the data are

Page 56 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

collected for specific administrative purposes, not just for nutrition

monitoring. Sample survey data, however, have the advantage of

relevance, but the data can be costly to obtain at a level of detail that will

support estimates for states, localities, or other subpopulation groups. In

contrast, indirect estimation is an approach to producing timely, relevant,

and detailed information without a major increase in cost.

Potential Limitations The major limitation of indirect estimation is the difficulty of determining

the quality of the estimate. The best way is to compare it to the true value

for the population. For example, comparing an indirect estimate of a

county’s population to census data on the county’s population would

enable an assessment of the bias of the estimate. However, since indirect

estimation is used when data are not available for direct estimation, such a

comparison is usually not feasible. Moreover, the quality of the estimates

yielded by a model changes for different populations and for different

times. In other words, even if a model yields estimates that prove to be

unbiased in comparison to direct observations for the same year,

estimates from the same model for subsequent years may be biased if the

relationships between the variables change over time.

This limitation has an implication for the use of indirect estimates.

Specifically, indirect estimates may be difficult to defend in the political

arena because they are based on models rather than direct observations.

However, in the absence of direct data from other, more expensive

approaches, indirect estimation is preferable to no information at all.

Although indirect estimation has been used successfully with other federal

surveys, there are constraints on the development of indirect estimation

programs for the major nutrition monitoring surveys. According to USDA

and HHS staff, major obstacles include the lack of staff resources to

support the program and the complexity of the surveys. Despite these

concerns, both agencies have long experience with indirect estimation in

other arenas, which could be applied to nutrition monitoring.

Conclusions Information on subpopulation groups and small geographic areas is used

to identify nutrition-related problems that are associated with specific

populations and to target programs to those most at risk. Different

strengths and weaknesses are associated with the four approaches to

meeting this need that we reviewed. Surveillance systems and indirect

estimation programs are likely to be less expensive than oversampling or

special surveys. However, oversampling and special surveys can yield

Page 57 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 5 

Approaches to Supporting Reliable

Inferences About Subpopulations

more detailed information than the surveillance systems that rely on

program records and more dependable estimates than indirect estimation

models. Of the four approaches, only indirect estimation has not been a

part of NNMRRP activities at one time or another. From programs in other

areas, this approach appears to be a potentially efficient means of

expanding the information available on subpopulations and small

geographic areas. However, before the Interagency Board can determine

what priority to give to this promising approach, a complete picture of

what needs would be met by each option is required.

Page 58 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 6 

Approaches to Assisting States and

Localities

State and local governments are interested not only in the applicability of

federal nutrition monitoring data to their jurisdictions, but also in having

federal assistance in collecting and interpreting their own data. This

chapter follows on the previous discussion about NNMRRP support for

reliable inferences about state and local populations to describe current

and potential options for assisting states and localities in their own

monitoring activities.

The NNMRRP currently assists states through the state-based surveillance

systems, which provide technical and other assistance to states. The

alternative that we examined, community-based nutrition monitoring, is a

response to states and localities that are interested in building their

capacity to collect their own data. The strengths and weaknesses of these

two approaches to assisting states and localities are summarized in table

6.1 and detailed below.

Table 6.1: Approaches to Assisting States and Localities

Approach Strength Limitation

Current: state-based surveillance systems Balance between federal need for

standardization and state need for tailoring

Development of states’ capacity for data

collection and interpretation

Potential for improvements on current

foundation

Limited flexibility for state and local

governments

Limited support for local use of the data

Potential: community-based nutrition

monitoring

Development of local capacity for

collecting and using data tailored to

community concerns

Limited federal investment

Technical and financial resources required

to assess needs or implement responses to

needs

Tension between federal interest in

standardization and local interest in

flexibility

Other suggestions by our expert panelists—such as providing financial and

technical assistance, developing standardized modules of interest for state

data collection activities, and assisting state collection of data on

subpopulations—can be implemented as part of either of the two

approaches.

Current Approach: the

State-Based

Surveillance Systems

The strengths and limitations of the surveillance systems as sources of

continuous data and of information on state and other population groups

have been discussed in chapters 4 and 5. However, they have additional

Page 59 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 6 

Approaches to Assisting States and

Localities

strengths and limitations related to their ability to meet state and local

needs for assistance.

Strengths Surveillance systems currently balance the federal interest in information

that is collected in a standard format across states and the states’ needs

for flexibility. For example, with BRFSS, HHS supplies states with

standardized modules of questions, training in collecting the data, and

support in analysis and reporting. In addition, changes to the survey

content are made in consultation with state participants in the surveillance

system, and states have the opportunity to add their own questions to the

survey.

The surveillance systems have also played a role in building state capacity

for data collection and analysis. Specifically, some states have used their

experience with BRFSS to implement their own telephone surveys using the

random digit dialing procedure. In addition, standardized software

developed by HHS for PNSS has enabled states to generate their own reports.

Yet another strength of the surveillance systems is the foundation of

federal-state partnership it provides for future improvements of federal

technical assistance to state and local nutrition monitoring activities. For

example, as mentioned in previous chapters, HHS is exploring ways to use

the surveillance systems to gather additional nutritional data, such as

more in-depth information on dietary intake, and to cover new

populations, such as schoolchildren. Additional technical assistance in

data analysis and interpretation could also be provided through the

surveillance system structure.

Limitations While the surveillance systems could be expanded and improved to further

respond to the interest of states and localities in receiving more technical

assistance, they are limited in their flexibility because of the federal

interest in standardization across states. In addition, users of surveillance

system data who responded to our survey identified some concerns about

the surveillance systems. Specific issues included the availability of the

data to localities, timeliness of HHS’ processing of the data, and the formats

of the reports that HHS provided. Recommendations for improving the

surveillance systems included increasing local access to data, reducing the

time it takes HHS to process the data, simplifying reports for local users,

and providing additional technical and financial assistance in data

collection and interpretation.

Page 60 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 6 

Approaches to Assisting States and

Localities

Potential Option:

Community-Based

Nutrition Monitoring

The federal-state linkage forged by the surveillance systems could be

further extended to support community-based nutrition monitoring. To

explore the strengths and limitations of this approach, we reviewed the

literature on two models of community-based programs. HHS has funded

Planned Approach to Community Health (PATCH) projects, which used the

BRFSS survey instrument as the basis for a community needs assessment.

The survey data, in combination with interviews with knowledgeable

informants in the community, were used to plan health promotion

programs. HHS provided technical assistance in the needs assessment

portion of the projects and small awards of a few thousand dollars for

project activities.

Researchers at Cornell University developed a similar model specifically

for nutrition monitoring that was pilot-tested in three New York counties

with funding from the State Department of Health and technical assistance

from the university. In their approach, a coalition of interested community

members first articulates potential information needs. Then, the group

selects specific needs on the basis of the likelihood that the data will be

used by the community. To meet the selected information needs, feasible

sources of routinely available data are identified. According to the model,

data collection should depend on procedures that are already in place,

from such sources as program participation records, patient charts, and

school health screenings. In its reliance on program records, this approach

is similar to PedNSS and PNSS. However, there is no expectation that the

same issues will be targeted or the same sources of data used in each

community. To facilitate the final component of the system—the

communication of monitoring results—a network of users must be

cultivated and the appropriate vehicle for communicating the results must

be used. Drawing on the experience of these projects and HHS’ PATCH

program, the following strengths and limitations of community-based

nutrition monitoring were identified.

Potential Strengths Evaluations of both types of community-based programs found evidence

that local capacity for collecting and using data was developed. By

involving community members in assessing needs, PATCH built skills in

identifying health risks and cultivated community and organizational

supports for health promotion programs. In the New York counties that

pilot-tested the Cornell model of community-based nutrition monitoring,

data were collected and compiled to describe access to food and nutrition

services and nutritional health of county residents. To disseminate the

information and guide decisions based on the data, interagency coalitions

Page 61 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 6 

Approaches to Assisting States and

Localities

were formed. These coalitions brought together local nutrition-related

professionals, so that nutrition interventions were better coordinated and

information was shared.

The limited federal investment in the community-based programs is

another strength. HHS supported PATCH with small grants and technical

assistance, while the community-based nutrition monitoring projects were

sponsored by the state and received technical assistance from the state’s

land grant institution.

Moreover, there is some evidence that the activities put in place by these

projects will continue. For example, Cornell reports that

“At the end of the [3-year] pilot stage, one county obtained local funding to continue their

monitoring activities, including the issue-based coalitions; in the other counties, local

nutrition councils are coordinating continued monitoring efforts.”1

The program has also spread to other counties, some of which have

initiated monitoring activities without outside funding.

Potential Limitations The experience with community-based programs indicates that

communities require a set of resources—specifically, technical skills and

dedicated personnel—to fully benefit from the projects. One evaluation of

PATCH found that the projects were most effective in communities that

already had human services and health professionals who were involved in

community health promotion efforts. In addition, the projects that had

directors seemed to have the greatest implementation successes. Both

kinds of projects required considerable time and effort to collect and

interpret data. Moreover, while HHS provided technical assistance in using

the BRFSS survey in the local needs assessments, additional assistance was

needed to help communities set priorities based on the data.

The reliance on BRFSS was a limitation for PATCH because BRFSS did not

necessarily include the issues of primary concern for the community. For

example, one PATCH project was specifically interested in water quality,

which is not addressed in BRFSS. In contrast, the Cornell model emphasized

local sources of data. Based on the initial experience with PATCH, HHS no

longer expects communities to use BRFSS for its needs assessment.

1

“Community-Based Nutrition Monitoring,” Division of Nutritional Sciences, Cornell University, and

Division of Nutrition, N.Y. State Department of Health, Mar. 1993.

Page 62 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Chapter 6 

Approaches to Assisting States and

Localities

Conclusions Data from the HHS surveillance systems are already extensively used in

state and local program planning. In addition, the systems are flexible and

are exploring ways to increase the information they collect on dietary

intake. However, they seem less responsive to the needs of localities than

to the needs of states. Recent efforts to build local capacity for data

collection and interpretation indicate that community-based programs are

a promising approach to responding to local needs for nutrition

information. Before the Interagency Board can decide what priority to

place on community-based nutrition monitoring, however, it must first

identify the objectives of the NNMRRP that would be furthered and the

importance of these objectives relative to others that are competing for

program resources.

Page 63 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix I 

List of Experts

This appendix lists the expert advisers who assisted with this project. As

described in chapter 1, the advisers were organized into three

panels—core policy panel, methodology panel, and data users panel.

Core Policy Panel Johanna Dwyer, D.Sc., R.D., Francis Stern Nutrition Center, New England

Medical Center, and Tufts University Schools of Medicine and Nutrition

Jean-Pierre Habicht, M.D., Ph.D., Division of Nutritional Sciences, Cornell

University

Catherine Woteki, Ph.D.1

Methodology Panel Norman Bradburn, Ph.D., Director, National Opinion Research Center

Marilyn Buzzard, Ph.D., Director, Nutrition Coordinating Center,

University of Minnesota

Ricardo O. Castillo, M.D., M.P.H., Co-Director, Pediatric Gastroenterology,

Stanford University Medical Center

Alan R. Kristal, Dr.P.H., Fred Hutchinson Cancer Research Center and

Department of Epidemiology, University of Washington

James Lepkowski, Ph.D., Institute for Social Research, University of

Michigan

Cheryl Ritenbaugh, Ph.D., Department of Family and Community

Medicine, University of Arizona

Laura Sims, Ph.D., Department of Nutrition and Food Science, University

of Maryland

Data Users Panel Elizabeth Barnett, Ph.D., North Carolina Department of Environment,

Health, and Natural Resources

Doris Disbrow, Dr.P.H., R.D., Center for Health Education

1

Dr. Woteki withdrew from the panel when she was appointed to the Office of Science and Technology

Policy in the White House. During her participation in our work, she was the Director of the Food and

Nutrition Board of the National Academy of Sciences.

Page 64 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix I 

List of Experts

Pamela Haines, Dr.P.H., R.D., Department of Nutrition, University of North

Carolina

Jay Hirschman, M.P.H., Senior Analyst, Food and Consumer Service, U.S.

Department of Agriculture

Karen J. Morgan, Ph.D., Senior Director, Nutrition and Consumer Affairs,

Nabisco Brands

Barbara Petersen, Ph.D., Technical Assessment Systems

Page 65 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

The model features that are the focus of this report were derived from four

sources. First, we reviewed previous evaluations, from which 13 historical

criticisms of federal nutrition monitoring were identified. Second, we

considered the objectives and related activities outlined in the 10-year

comprehensive plan developed by the Interagency Board. Third, through

an iterative process of panel meetings and review of written materials, a

number of suggestions for improvements to the NNMRRP were generated by

the expert advisers. Fourth, the data users who responded to our survey

provided written comments describing what changes to specific data

collection activities would increase their use of the data.

Past Evaluations of

Federal Nutrition

Monitoring

Table II.1 lists the criticisms identified in past evaluations and categorizes

them by the features of a model program that they suggest. (See Nutrition

Monitoring (GAO/PEMD-94-23), pp. 5-6.)

Table II.1: Features of a Model Program Suggested by Past Evaluations

Feature Past criticism

Coordinated system that responds efficiently to the diverse needs

of the data users

Lack of coordination among nutrition monitoring activities

Information needs of users not systematically determined

Lack of compatibility in methods for assessing dietary intake

Core set of standardized measures not yet developed for major

surveys

Compatible sampling techniques not used for national surveys

Reporting by national surveys not integrated

Continuous or more frequent collection of data Data not collected continuously

Support for reliable inferences about subpopulation groups and

small geographic areas

Specific population groups not covered by major surveys

Specific geographic areas not represented by major surveys

Assistance to states and localities State and local data needs not fully addressed

Improved methodology for assessing dietary intake and nutritional

status

Improvement needed in methodology for assessing dietary intake

and nutritional status

Low respondent burden and high response rates Sampling problems with NFCS

Timely processing and dissemination of data Need for more timely dissemination of survey information

Page 66 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

The 10-Year

Comprehensive Plan

Table II.2 provides examples from the 68 activities listed in the 10-year

comprehensive plan that correspond to the desirable features identified

from the review of previous evaluations of federal nutrition monitoring.

(Other desirable features suggested by the 10-year plan activities but not

addressed in this report focus on strengthening the food composition and

food supply data systems.)

Page 67 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

Table II.2: Examples of Planned NNMRRP Activities Associated With Model Features

Feature Planned activitya

Coordinated system that responds efficiently to the diverse needs

of the data users

Coordinate the planning for coverage, tracking, and reporting of

findings from surveys and surveillance systems

Identify ways to increase comparability within a dietary method to

improve the quality and usefulness of data

Establish a mechanism for improved coordination among federal

agencies that collect and use survey information about

knowledge, attitudes, and behavior to assess gaps and

duplications in existing surveys

Continuous or more frequent collection of data No specific activities listedb

Support for reliable inferences about subpopulation groups and

small geographic areas

Develop and implement a plan for improved coverage of

subgroups of the population at nutritional risk or at risk for

underconsumption or overconsumption of nutrients and food

components

Evaluate the feasibility of alternate statistical methodologies for

creating state and local estimates

Assistance to states and localities Provide assistance for the development and maintenance of state

structure, staff, and programs to support their participation in

NNMRRP

Expand the coverage of current state and local nutrition

monitoring activities in selected population groups through

technical assistance and grant awards

Improved methodology for assessing dietary intake and nutritional

status

Conduct research to develop, improve, and validate laboratory

measures of nutritional status

Develop and evaluate procedures for determining usual intakes of

foods and nutrients from surveys employing 24-hour recall

measures of dietary intake

Low respondent burden and high response rates No specific activities listedc

Timely processing and dissemination of data No specific activities listedd

aSee the Ten-Year Comprehensive Plan, 58 Fed. Reg. 111 (June 11, 1993), pp. 32752-806.

bAlthough no specific activities are planned to ensure the continuous collection of data, two of the

six objectives include continuous data and one of the activities mentions coordinating the timing

of surveys.

cUSDA has separated the household and individual portions of the NFCS survey in order to

reduce respondent burden and improve response rates. In addition, USDA signed interagency

agreements with the Bureau of Census for assistance in designing and conducting the individual

and household surveys, as well as for research on improving methods for collecting household

food use data.

dAlthough none of the activities identified under the objectives specifically address the timeliness

of data processing and dissemination, part of the overarching goal stated for the plan is

“efficiently disseminating and exchanging information with data users.” In addition, the plan

discusses possible mechanisms for increasing awareness and dissemination of the data and

states that the Interagency Board plans to establish a central clearinghouse.

Page 68 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

Consultation With

Expert Advisers

The expert advisers to the project (listed in appendix I) provided guidance

by reviewing materials and participating in panel meetings. Through this

process, several suggestions were generated, and their advantages and

disadvantages were discussed. The suggestions outlined below in table II.3

do not indicate consensus among the panelists.

Table II.3: Advisory Panel Suggestions Associated With Model Features

Feature Suggested approach

Coordinated system that responds efficiently to the diverse needs

of the data users

A single lead agency

An interagency body with permanent staff and enforcement

authority

Location in statistical agencies within the user departments or

within a central statistical agency

Centralization of the congressional appropriations process for

nutrition monitoring activities

Informed review of data collection plans by qualified people at

OMB

Continuous or more frequent collection of data Continuous national nutrition surveys

Addition of modules to existing surveys

Reliance on program data that are already collected

Collection of longitudinal data

Support for reliable inferences about subpopulation groups and

small geographic areas

Surveys of population subgroups instead of national-level surveys

Different sampling strategies for different populations (for

example, list-based sampling for populations that are rare in a

broad-based sample)

Contracts with states and localities to gather information on

geographically-based populations

Indirect estimation to support inferences about population

subgroups and small geographic areas

Assistance to states and localitiesa Financial assistance to states to determine their data needs

A federal-state partnership in which states can provide funds for

some extra sampling or extra questions on federal surveys

Federal development of standardized modules of interest for state

data collection activities

Federal assistance to state collection of data on subpopulations

Technical assistance in data interpretation

(continued)

Page 69 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

Feature Suggested approach

Improved methodology for assessing dietary intake and nutritional

status

Continued or increased research on nutrition monitoring methods

Postmarket surveillance data (such as that gathered through bar

code scanners)

Automated collection of dietary data

Reliance on program records

Low respondent burden and high response rates Provide more feedback to respondents about survey results

Timely processing and dissemination of data Assistance to data users in interpreting and analyzing the data

Development of core analysis data sets that are focused on

specific issues or groups

A data clearinghouse

Automated data collection

aThe panelists disagreed about the responsibility states should have for data collection, with

some arguing for state-based data collection that feeds into a federal system and others arguing

for less state responsibility for data collection, but increased consideration of state needs in

federal data collection activities.

In addition to comments that could be categorized by the model features

suggested by the historical criticisms, the expert advisers also had other

suggestions for improving the system, including

• develop an ongoing evaluation of the system’s content and methods,

including a review of the information needs;

• continue to collect both household and individual level data;

• have a system to maintain data comparability over time; and

• use longitudinal designs to measure change.

Suggestions From the

Survey of Data Users

The comments that users of nutrition monitoring data provided in

response to our survey were analyzed to identify major themes for each of

three groups of data collection activities—USDA surveys, HHS surveys, and

HHS state-based surveillance systems. Table II.4 identifies the themes

associated with the features of a model program. More detailed summaries

can be found in Nutrition Monitoring (GAO/PEMD-95-15).

Page 70 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

Table II.4: Major Themes in the Suggestions Made by Data Users

Major Theme

Model feature USDA surveys HHS surveys HHS surveillance systems

Coordinated system that

responds efficiently to the diverse

needs of the data users

a a a

Continuous or more frequent

collection of data

Continuous or more frequent

data collection

Continuous or more frequent

data collection

b

Support for reliable inferences

about subpopulations

geographic areas

Increased sample sizes and

broadened coverage

More detail on racial, ethnic,

age, and income groups

Refined geographic coverage,

specifically state and substate

data

More detail on racial, ethnic,

and age groups

Data that can support estimates

for small geographic areas

More detail on subpopulation

groups in the reporting of the

data

Increased ability to look at

substate geographic divisions

Assistance to states and localities c c Simplified reports that are more

readily used at the local level

Additional technical and

financial assistance in data

collection and interpretation

Better controls on the quality of

the data collected

Improved methodology for

assessing dietary intake and

nutritional status

More specificity and detail

about foods and better data on

food composition

Improved questions about

dietary behavior

More information about health

and demographic variables

Higher quality dietary recall

data

More information on health

habits and outcomes

More detailed data on food

consumption

Improved dietary intake

methods

More data on dietary intake

Reduced respondent burden and

improved response rates

Reduced respondent burden

and improved response rates

d d

Timely processing and

dissemination of data

Improved timeliness and

documentation of the data

Dissemination of the data in

formats that facilitate access

and analysis

Improved timeliness and

documentation of the data

Dissemination of the data in

formats that facilitate access

and analysis

Improved timeliness of HHS’

return of the data to states

(Table notes on next page)

Page 71 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix II 

Sources for the Model Features

aCoordination was not a major theme of the comments provided in response to our survey of data

users probably because the survey queried respondents about changes to specific data

collection activities rather than NNMRRP as a whole.

bUsers of surveillance system data were unlikely to identify continuous data as a major theme

since these systems collect data continuously.

cBecause our survey asked for comments about the two data collection systems most frequently

used by the respondent, state and local users were more likely to comment on the surveillance

systems. As a result, their interest in changing the national surveys to better meet their needs was

not gauged.

dThe HHS systems have not suffered the same response rate problems that USDA’s 1987-88

NFCS experienced.

The themes reflect the strengths and weaknesses of the different data

collection activities. For example, response rates were probably a major

issue to users of USDA surveys because of the response rate problems of

NFCS. Similarly, assistance to states and localities was a theme from the

users of the surveillance systems, many of whom manage state and local

programs. In addition to the themes that correspond to the features of a

model program, other major issues in the suggestions from the data users

were the collection of longitudinal data (from users of USDA survey data)

and the collection of both individual and household data (from users of

data from USDA and HHS surveys).

Page 72 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 73 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

Page 74 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

See comment 1.

See comment 2.

See comment 3.

See comment 4.

Page 75 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

See comment 5.

See comment 6.

See comment 7.

Page 76 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

See comment 8.

See comment 9.

See comment 10.

Page 77 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

The following are GAO’s comments in response to the June 13, 1995, letter

from the Department of Agriculture.

1. What USDA finds inconsistent with the report reflects the inconsistency

of NNMRRP. The 10-year plan, a natural location for a statement of priorities

within expected funding levels, specified neither priorities nor expected

funding levels. As noted in our report, the later priority-setting exercise

was an improvement but was also limited in its utility because it did not

specify whether data needs and feasibility were considerations in ranking

the objectives.

2. The central contact for data users that is suggested by the alternate

approaches to coordination entails much more than a clearinghouse for

data products. For example, the Office of National Drug Control Policy

acts not only as a source of information for state and local drug

enforcement interests, but also as a conduit of information about state and

local concerns to the federal agencies.

3. USDA makes several recommendations on how the Board can be

improved. These recommendations are consistent with the strengths that

we identified in the other approaches to coordination.

4. The Consumer Expenditure Survey also provides information about the

cost of food used at the household level. However, because of the detail

about the expenditures on specific foods gathered by the NFCS, we have

amended the description of the NFCS.

5. Following the classification provided by the NNMRRP document, The

Directory of Federal and State Nutrition Monitoring Activities, CSFII and

DHKS focus on food and nutrient consumption. Table 1.1 indicates that data

on food and nutrient consumption are used for health- as well as

food-related purposes.

6. The sentence has been changed to reflect this information.

7. Because the coordinated NNMRRP budget reports amounts dedicated to

nutrition monitoring by agency rather than by purpose, the accounting

system used by ARS is not relevant. All NNMRRP expenditures by ARS will be

reported together, whether they are dedicated to surveys or related

research.

Page 78 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix III 

Comments From the Department of

Agriculture

8. The fact that the plan was intentionally vague does not diminish the

degree to which the lack of detail limits its utility as a statement about the

specific objectives of the program, the activities planned to meet the

objectives, and the resources needed to support the plans.

9. While technical expertise can be supplied by staff, only members with

organizational authority can make decisions about agency priorities and

resources. USDA’s comment regarding p. ES-7 (responded to in comment 3

above) suggests the importance of having members with budget

decision-making authority.

10. The statement has been changed to reflect this information.

Page 79 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix IV 

Comments From the Public Health Service

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 80 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix IV 

Comments From the Public Health Service

See comment 1.

Page 81 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix IV 

Comments From the Public Health Service

See comment 2.

See comment 3.

Page 82 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix IV 

Comments From the Public Health Service

See comment 4.

See comment 5.

Page 83 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix IV 

Comments From the Public Health Service

The following are GAO’s comments on the June 16, 1995, letter from the

Public Health Service.

1. As PHS states, the 10-year plan does plan to conduct evaluations in 1997

and 2002. However, neither these evaluations nor the Board’s annual

summaries of progress are a strategy for weighing the costs against the

benefits of current or new activities. A framework for comparing existing

and potential monitoring activities in such terms as their feasibility, ability

to meet data needs, and other considerations is still needed.

2. The paragraph referred to (now on pp. 34-35) is not about the progress

of the working groups, but about the guidance provided by the 10-year

plan for the program as a whole. While the progress reports indicate

specific examples of improvements within working groups, the 10-year

plan does not constitute a comprehensive proposal. Such a proposal

would include specific objectives, detailed plans for how to meet the

objectives, and cost estimates and funding requests to support the plans.

The availability of resources affects the ability to meet an objective, but is

not a reason for not developing specific plans in the first place.

3. In contrast to such coordinating bodies as the Interagency Board and

the National Library of Medicine, the central authorities that we reviewed

have personnel who are not detailed from the agencies. While the potential

disadvantage that PHS suggests (that is, unfamiliarity with NNMRRP) can be

easily overcome, the advantage of independence from the interests of any

single agency is difficult to achieve in the other approaches.

4. The sentence has been changed to clarify that, with current levels of

state participation, the PedNSS and PNSS can not be used to provide

nationally representive data of the low-income mothers and children

receiving assistance through the WIC program.

5. This section has been changed to reflect the lack of coverage of

American Indians and Alaska Natives living on reservations by the national

household sampling frame.

Page 84 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix V 

Major Contributors to This Report

Program Evaluation

and Methodology

Division

John Oppenheim, Assistant Director

Leslie Riggin, Project Manager

Lê Xuân Hy, Adviser

Venkareddy Chennareddy, Referencer

Elizabeth W. Scullin, Communications Analyst

Page 85 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix V 

Major Contributors to This Report

Page 86 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Appendix V 

Major Contributors to This Report

Page 87 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Related GAO Products

Nutrition Monitoring: Data Serve Many Purposes; Users Recommend

Improvements (GAO/PEMD-95-15; June 20, 1995).

High Performance Computing and Communications: New Program

Direction Would Benefit From a More Focused Effort (GAO/AIMD-95-6;

Nov. 4, 1994).

Nutrition Monitoring: Progress in Developing a Coordinated Program

(GAO/PEMD-94-23; May 27, 1994).

Food Nutrition: Better Guidance Needed to Improve Reliability of USDA’s

Food Consumption Data (GAO/RCED-94-30; Oct. 25, 1993).

(973397) Page 88 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be mailed to a

single address are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100
GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program