Nutrition Monitoring: Establishing a Model Program
Released: 1995-07-19T00:00:00

Letter

The Current Nutrition Monitoring Program

This chapter first provides background information on the National Nutrition Monitoring and Related Research Program. Then, the objectives, scope, and methodology of our review of current and potential approaches to achieving a model program are described. The chapter concludes with an overview of the organization of the rest of the report.

Core Data Collection Activities of the NNMRRP

The NNMRRP is a complex system of data collection and research activities, including national surveys, state surveillance activities, and a variety of research programs. Over time, the NNMRRP has developed activities focused on five content areas: (1) food and nutrient consumption; (2) nutritional and health status; (3) dietary knowledge, attitudes, and behavior; (4) food composition; and (5) food supply. As shown in table 1.1, the information produced by these activities is used for a variety of purposes, from supporting basic research on human nutritional needs to informing policy decisions about health, agriculture, and food programs.

Table 1.2 lists specific activities managed by the Departments of Agriculture and Health and Human Services, which have major responsibilities for the five areas. Other agencies, including Commerce, Defense, and the Environmental Protection Agency, also participate in the NNMRRP.

USDA Activities

USDA has major responsibilities for collecting information in all of the content areas except nutritional and health status. USDA gathers data on food and nutrient consumption through two national surveys—the Nationwide Food Consumption Survey (NFCS) and the Continuing Survey of Food Intake by Individuals (CSFII). In the past, NFCS has gathered nationally-representative information on the food consumption behavior of households and individuals. It provides detailed data on household costs for food. One of the major uses for these data is the development of the Thrifty Food Plan, which is the basis for calculating food stamp benefits. Implemented decennially, NFCS suffered from severe response rate problems (less than 40 percent) in 1987-88. As a result, the individual food consumption portion of the NFCS is expected to be dropped in the future.

Since the mid-1980s, CSFII has supplemented NFCS by providing regular information on individual dietary intake. The data collected in its most recent implementation (1994-96) will be used to describe both general and low-income populations. The Diet and Health Knowledge Survey (DHKS), which collects data on dietary knowledge, attitudes, and behavior, is a follow-up to CSFII. Together, CSFII and DHKS are intended to inform policies relating to food production and marketing, food safety, food assistance, and nutrition education.

The USDA activities focused on food composition and food supply are not surveys. For food composition, USDA gathers data from food industries and other sources on the nutrient content of foods. These data support the dietary surveys by translating the foods consumed into their nutrient components. Food supply is estimated by deducting data on exports, year-end inventories, and nonfood use from data on production, imports, and beginning inventories.

HHS Activities

HHS is responsible for the National Health and Nutrition Examination Survey (NHANES) and the state-based surveillance systems. These data collection activities provide information on the content areas (1) food and nutrient consumption, (2) nutritional and health status, and (3) dietary knowledge, attitudes, and behavior. Like NFCS and CSFII, NHANES collects data from a nationally-representative sample. However, NHANES’ unique contribution is its use of physical examinations and clinical and laboratory tests as well as traditional survey methods to gather information. NHANES’ data support research on the relationship between diet and health and inform health policy decisions, such as the promotion of cholesterol screening. After two earlier implementations in the 1970s, NHANES has just completed its third administration (1988-94). NHANES has been supplemented by the Hispanic Health and Nutrition Examination Survey (HHANES) in the early 1980s, follow-up surveys of respondents to NHANES I, and follow-up matches of the records from NHANES II to the National Death Index and other vital statistics records.

The state-based surveillance systems were set up to provide quick information to states to use in planning and managing nutrition and health programs. They include the Pediatric Nutrition Surveillance System (PedNSS), Pregnancy Nutrition Surveillance System (PNSS), and Behavioral Risk Factor Surveillance System (BRFSS). Participating states collect the data for these systems with technical and other kinds of assistance from HHS. Both PedNSS and PNSS rely on data from clinic records from publicly funded health, nutrition, and food assistance programs, primarily the Special Supplemental Food Program for Women, Infants, and Children (WIC). PedNSS monitors nutritional status among low-income, high-risk children, while PNSS focuses on low-income, high-risk pregnant women, measuring nutrition-related problems and behavioral risk factors associated with low birthweight.

In contrast to PedNSS and PNSS, data for BRFSS are gathered through telephone interviews, with respondents (adults 18 years and over) sampled through random digit dialing. In addition to a core set of questions on various health risk factors, BRFSS includes optional modules for the assessment of dietary fat and fruit and vegetable consumption.

Other major HHS monitoring activities include the Total Diet Study, which analyzes nutrient and contaminant levels in the food supply; the Food Label and Package Survey, which monitors nutrition labeling practices; and the Health and Diet Survey, which assesses dietary knowledge and practices as they relate to health problems.

Structure of the NNMRRP

Although the United States has one of the most comprehensive monitoring programs in the world, several problems with nutrition monitoring activities have been identified over the past two decades. Of key concern has been the lack of coordination and compatibility of different data collection activities. This encompasses differences across surveys in methods for assessing dietary intake and nutritional status, sampling designs, population descriptors and other measures, and the timing and reporting of results.

To improve the coordination of federal nutrition monitoring activities and the quality of the data collected, the Congress passed the National Nutrition Monitoring and Related Research Act of 1990 (P.L. 101-445). The act established an Interagency Board, jointly chaired by USDA and HHS, to coordinate activities across the various agencies involved in nutrition monitoring. The Interagency Board was charged with developing a strategic plan that would establish a comprehensive nutrition monitoring and related research program. This plan—known as the 10-year comprehensive plan—was published in the Federal Register on June 11, 1993. It outlines a set of planning activities, including a general time frame and lead agencies for each activity. The activities are organized around six objectives, which are to • provide for a comprehensive NNMRRP through continuous and coordinated data collection; improve the comparability and quality of data across the NNMRRP; improve the research base for nutrition monitoring; • develop and strengthen state and local capacity for continuous and coordinated nutrition monitoring data collection that complements national nutrition surveys; improve methodologies to enhance comparability of NNMRRP data across federal, state, and local levels; and improve the quality of state and local nutrition monitoring data.

In addition, an Advisory Council of experts from outside the federal government was created to guide the Interagency Board on scientific and technical matters. (Chapter 3 contains more information on the Interagency Board and its activities.)

Objectives, Scope, and Methodology

Objectives

This is the third and final report in a series responding to a request from the former House Committee on Science, Space, and Technology. The first report reviewed past evaluations of federal nutrition monitoring and examined the progress of the NNMRRP since the passage of the 1990 act. It concluded that (1) a coherent program for nutrition monitoring was not yet in place and (2) although there has been progress in coordinating the program, the 10-year plan is incomplete because it does not include a framework for evaluating current and potential activities or detailed plans for achieving the objectives.

Based on a survey of users of nutrition monitoring data, the second report described the purposes for which nutrition data are used and summarized respondents’ suggestions for improving NNMRRP activities. These suggestions, which addressed such issues as the timing of the surveys, their coverage of subpopulations, and the ease with which the data could be used, were consistent with the concerns raised by past evaluations of federal nutrition monitoring and indicated a continued need to address the long-standing problems of federal nutrition monitoring.

Completing our response to the Committee’s request, this report builds on our earlier work to meet two objectives: (1) define a model nutrition monitoring program and (2) compare the current system with potential options for implementing the components of a model program in the NNMRRP.

Scope and Methodology

Defining the Features of a Model Program

Before defining the features of a model program, we first limited our review to three of the five content areas covered by the NNMRRP: (1) food consumption and dietary intake; (2) health and nutritional status; and (3) knowledge, attitudes, and behavior. These three variables were selected for both substantive and methodological reasons. Substantively, the selected elements provide the data that support the planning and evaluation of interventions that directly affect health, such as nutrition education and food assistance programs. A substantive argument could also be made for the inclusion of food composition data, which are used to translate the information on what foods are eaten into estimates of nutrient intake. However, because an earlier GAO project focused on the NNMRRP’s food composition activities, we did not include this in our review.

Methodologically, the three selected variables are linked because they rely on data obtained from individuals through surveys and physical examinations. In contrast, food composition information is based on chemical analyses of foods, and food supply is determined from macroeconomic data. Our focus on the components of the NNMRRP that rely on surveys facilitates the comparison of current and potential approaches to achieving a model program.

To identify features of a model nutrition monitoring program, we used four sources: reviews of previous evaluations of federal nutrition monitoring activities, review of the objectives and related activities outlined in the 10-year comprehensive plan developed by the Interagency Board, consultation with expert advisers, and our survey of data users.

These sources are detailed below, and supporting material is provided in appendix II. (See chapter 2 for a description of the features.)

Past Evaluations of Federal Nutrition Monitoring. Past evaluations by such groups as the National Academy of Public Administrators, the Joint Nutrition Monitoring Evaluation Committee, and the National Research Council identified several concerns about the federal nutrition monitoring program. Because these evaluations also informed the Interagency Board’s development of its 10-year plan, we used them as the starting point for our identification of features of a model nutrition monitoring program. Our report, Nutrition Monitoring (GAO/PEMD-94-23), discusses these past evaluations and NNMRRP progress in addressing their recommendations. (Table II.1 in appendix II lists the criticisms identified in these evaluations and categorizes them by the features of a model program that they suggest.)

The 10-Year Comprehensive Plan. As described above, the NNMRR Act required the Interagency Board to develop a plan for the program. The plan outlined six objectives, three with a federal focus and three emphasizing state and local monitoring, and listed 68 planned activities. These were reviewed for responsiveness to the model features suggested by the past evaluations. (See table II.2 in appendix II for examples of the 68 activities listed in the plan.)

Consultation With Expert Advisers. To assist us at critical decision points in this project, we organized three panels. (The members of each panel are listed in appendix I.) The Core Policy Panel consisted of nationally-known experts in nutrition and nutrition monitoring policy. These panelists were consulted throughout the project. In addition, they helped us develop a framework of purposes for nutrition monitoring data that guided our survey of data users.

The Methodology Panel included renowned experts in such fields as sampling, survey design, dietary assessment, and nutritional epidemiology. This panel met to help us identify promising approaches to critical elements of a nutrition monitoring system. In addition, the panelists assisted us on issues related to their areas of specialization.

The Data Users Panel consisted of users of the nutrition monitoring data, chosen to reflect the broad range of purposes that the data must serve, including the support of state and local nutrition programs, academic research, food industry research, and the development and evaluation of federal food assistance programs. As with the Methodology Panel, this panel was convened once to help us identify promising approaches to nutrition monitoring. Individual panelists were consulted later about specific issues related to their expertise.

Through the process of reviewing materials and participating in panel meetings, the expert advisers generated several suggestions for possible changes to the NNMRRP, examples of which are given in table II.3 in appendix II. In addition, they reviewed a draft of this report.

Suggestions From the Survey of Data Users. Our survey of users of nutrition monitoring data focused on primary users of 14 of the NNMRRP surveys and surveillance systems. Primary users were defined as those who directly access the data rather than use information that has already been processed and interpreted by others in reports and other documents. Since there is no single list of primary users of NNMRRP data, we obtained lists of known and potential data users such as people who had requested the data from NNMRRP agencies, attendees at nutrition-related workshops, state and local government officials working in nutrition, and members of associations for nutrition professionals.

A major portion of the survey was dedicated to determining how the respondents used the data. We also asked whether changes are needed to better meet the respondent’s information and data quality needs. If the respondent indicated a need for change, we asked for suggestions in the following categories: (1) data elements collected, (2) data collection methods, (3) units of analysis, (4) time of data collection, (5) population group coverage, (6) geographic area coverage, and (7) ease of use. These comments were analyzed to identify major themes for each of three groups of data collection activities—USDA surveys, HHS surveys, and HHS state-based surveillance systems. In appendix II, table II.4 identifies the themes associated with the features of a model program.

Comparing Approaches to Attaining the Model Features

From the features that were identified, we selected four as the focus of the second objective—the comparison of current and potential approaches to achieving the model program. We focused on features that reflect long-standing concerns about federal nutrition monitoring, that encompass other desired characteristics, and that generate debate about how they should be addressed.

For each of the selected features, we identified current activities of the NNMRRP through interviews with staff in NNMRRP agencies, attendance at meetings of the Interagency Board and the Advisory Council, and reviews of program documents. Potential approaches were identified through literature review, analysis of our survey results, and consultation with the expert advisers. We did not identify the universe of potential approaches. Instead, our search was focused on those approaches deemed promising and feasible by our expert advisers.

To assess each of the potential approaches, we first identified programs from the same set of sources that helped us define the features of a model program—that is, the literature, our survey of data users, and expert advisers. Where possible, we limited our consideration to programs that had some linkage to nutrition monitoring. For example, for separate studies of subpopulation groups, we looked at the experience with the Hispanic Health and Nutrition Examination Survey in the early 1980s. To describe the strengths and weaknesses of the programs illustrating the potential options, we reviewed program documents and related literature and interviewed managers and staff.

This review was conducted between October 1993 and December 1994 in accordance with generally accepted government auditing standards.

Strengths and Limitations of Our Methodology

This report describes the results of a systematic examination of current and potential approaches to selected features of a model nutrition monitoring program. The strength of the review is its reliance on multiple sources of information. In addition to surveying users of nutrition monitoring data, consulting with experts, and reviewing both technical literature and program documents, we also interviewed officials and program staff in nutrition monitoring programs and in programs illustrating the alternatives. The major limitation of our work is its prospective nature. Because we were examining potential changes to the NNMRRP, hard evidence of the costs or effectiveness of the options was not available. Instead, the strengths and weaknesses of the options relative to the current nutrition monitoring system are supported primarily by logic and stated in tentative terms. Given this limitation, the report makes no recommendations for specific changes to the NNMRRP.

Organization of the Report

In response to the first objective of the project, chapter 2 describes the model program and provides information on the selection of four model features as the focus of the report. (The sources used to develop the model are described above.) The second objective—comparing current and potential approaches to each model feature—is addressed in chapters 3-6, organized by the four model features. Specifically, chapter 3 examines coordination options; chapter 4 compares alternate approaches to providing continuous data; chapter 5 discusses different methods of supporting inferences about subpopulation groups and small geographic areas; and chapter 6 reviews approaches to assisting state and local monitoring activities. Appendix I provides additional detail on the expert advisers to the project, and appendix II describes the sources for the model features. Agency comments on a draft of the report are in appendixes III and IV.

A Model Nutrition Monitoring Program

This chapter responds to the first objective of our review—the definition of a model nutrition monitoring program. First, the major features identified from the sources described in chapter 1 are outlined. From this set of features, we selected four as the focus for our response to the second objective of the review—the comparison of current and other approaches to achieving the model characteristics. The chapter describes our selection process and details the importance of the four features that are the subject of the rest of the report.

Features of a Model Program

Depending on the purposes that the data serve, the specific elements of a model nutrition monitoring program change. For example, researchers and program managers interested in food safety need detailed information on dietary intake, including specific brand names of the foods consumed. In contrast, a nutrition educator may place a higher priority on information about dietary knowledge, attitudes, and behaviors. However, at a more general level, some common ideal characteristics can be identified. Focusing on this general level, we used the sources described in chapter 1 to identify a number of features of a model program. Table 2.1 lists these features and the sources that support them.

All of the features identified are clearly important elements of a comprehensive nutrition monitoring program. However, we selected four characteristics as the focus of our review: (1) a coordinated set of activities that (2) provides data on a continuous basis, (3) supports inferences about important population groups, and (4) assists state and local monitoring activities. These features encompass other desired characteristics, respond to long-standing concerns about federal nutrition monitoring, and generate debate about how they can be achieved.

The first criterion for selection was to focus on the most general concerns. With these features, other desirable characteristics can be considered even though they are not emphasized. Specifically, a mechanism for evaluating the NNMRRP’s content and methods, including a review of the information needs of the data users, is described in chapter 3 as an element of a coordinated program. The theme of evaluating options in relation to the needs for data also underlies the discussion of the alternatives for the other features. Three other features—the comparability of data over time, the collection of longitudinal data, and the timeliness of data release—are related to the continuous collection of data and, as such, are considered briefly in chapter 4.

The four features were also selected because they respond to long-standing criticisms of federal nutrition monitoring activities. Concerns about coordination, the continuity and timeliness of the data, the availability of information on subpopulations, and the role of states and localities were raised as early as 1977 by witnesses before a House subcommittee. In contrast, the concerns with response rates and the level of the data (individual or household) can be traced to the problems with the last NFCS (described in chapter 1), which USDA has taken steps to avoid in the future.

The selected features are also not easily addressed; that is, there is debate about the best approach to achieving each feature. For example, some expert advisers stated that assistance to states and localities should focus on data analysis and interpretation, while others argued for a larger state and local role in data collection. Similarly, to provide information on subpopulation groups, national surveys could be supplemented by such means as the surveillance systems and oversampling or the national surveys could be abandoned and their resources dedicated to special studies of specific groups. In contrast to the debates about how best to achieve these and the other selected features, there has been consensus about the kind of dietary intake methodology that will be used by the NNMRRP surveys and ongoing research to improve these methods through automation and other means.

The Selected Features

An ideal federal nutrition monitoring program would have a coordinated set of activities that provides data on a continuous basis, covers important population groups, and supports state and local monitoring activities. Coordination is the key both to the efficiency of the system and its responsiveness to the needs of data users. A continuous flow of data would ensure that the information on the nation’s nutritional status was up-to-date and would also enable the tracking of dietary behavior and nutritional status over time. Information on population groups that are vulnerable or growing rapidly is needed to plan, manage, and evaluate programs intended to prevent or ameliorate nutritional problems. Because state and local governments are often the location of such programs, they need assistance in either interpreting available data or collecting their own information. The importance of these features is further explained in the following sections.

A Coordinated Nutrition Monitoring Program

The coordination of nutrition monitoring activities has implications for both the utility of the information produced and the costs of the program as a whole. The utility of the information is constrained when data from different data collection activities cannot be easily combined. For example, research on the relationship between diet and health could be strengthened if CSFII’s data on dietary intake could be combined with NHANES’ data on health and nutritional status. However, because of differences in the sampling designs and nutrition measures, combining the data from the two surveys is difficult and controversial. Similarly, poor integration of the data collected by the state-based surveillance systems with the national surveys presents a barrier to meaningful comparisons of state and national populations.

To the extent that the lack of coordination results in unnecessary duplication, a fragmented system can also increase the costs of nutrition monitoring. For example, the current NNMRRP includes two surveys focused on dietary knowledge, attitudes, and behaviors—one operated by USDA and the other by HHS. The Interagency Board plans to review these surveys for duplication. While some overlap can be useful as a quality check on data from different sources, unnecessary redundancy in the system uses resources that could be used for currently unmet data needs—such as the need for information on specific subpopulations at risk for nutrition-related problems.

The need for a coordinated system was supported by three of the four sources used to identify features of a model program. In addition to a general concern about the lack of coordination, past evaluations of federal nutrition monitoring have criticized the incompatibility of the data gathered by different enterprises. Specifically, these evaluations have called for compatible methods of assessing dietary intake, a core set of standardized measures for the major surveys, compatible sampling techniques in the national surveys, and integrated reporting. Another criticism related to coordination focused on the absence of a systematic process for determining the needs for nutrition monitoring data across the different data collection activities. Although needs are assessed for individual activities, no comprehensive assessment of needs in relation to the total system of activities has taken place.

Coordination is also a major theme of the 10-year plan. Four of the six objectives discussed in the plan focus on coordinating data collection and improving the comparability of the data. Of the specific activities listed for each objective, some of those focused on coordination are: coordinating the planning for coverage, tracking, and reporting of findings from surveys and surveillance systems; identifying ways to increase comparability within a dietary method to improve the quality and usefulness of data; and establishing a mechanism for improved coordination among federal agencies that collect and use survey information about knowledge, attitudes, and behavior to assess gaps and duplications in existing surveys.

The third source of support for the need for coordination comes from meetings of the advisory panels. The panelists noted that the different agencies involved in nutrition monitoring have different missions and priorities and, hence, coordination is difficult. Their major suggestions for improving coordination were to give coordination responsibility to a single lead agency, to coordinate from an interagency body with permanent staff and enforcement authority, to locate nutrition monitoring in statistical agencies within the user departments or within a central statistical agency, to centralize the congressional appropriations process for nutrition monitoring activities, and to ensure informed review of data collection plans by qualified staff at the Office of Management and Budget.

Our survey queried respondents about changes to specific data collection activities, rather than the NNMRRP as a whole, so coordination was not a major theme of the comments provided by the data users.

The Continuous Collection of Data

The Interagency Board defines continuous data as data collection that is “repeated regularly and frequently.” Two consequences follow when data are not regularly available. First, because the kinds of foods that are available and the eating patterns of the American people change rapidly, the data become outdated quickly. Compounded by delays between data collection and data release, long intervals between administrations of the surveys diminish the relevance of the data to the current situation and, hence, their utility in program planning and management. For example, if any of the policy changes currently being considered are implemented— such as the consolidation of food assistance programs—up-to-date information at regular intervals before and after the change will be needed to monitor any positive or negative effects of such policy changes on the population.

A second consequence of pauses in data collection is that potential efficiencies of a continuous survey operation are lost. Each implementation of a national survey requires extensive planning, including reviews of the needs of data users, development and testing of data collection procedures, and all the steps involved in approving a contract. An ongoing data collection operation could streamline some of these processes. In addition, when surveys are not in the field continuously or even at dependable intervals, they may attempt to meet as many of the needs of data users as possible when they are administered. For example, the low response rates of the 1987-88 NFCS have been partially attributed to the burden on survey respondents resulting from its attempt to obtain both household and individual data with one interview. In contrast, an ongoing survey could consist of a core set of questions and rotating modules of questions that address the needs of specific users.

The continuous collection of data was a theme in all four of our sources for the features of a model program. Past evaluations of nutrition monitoring have called for the continuous collection and timely release of nutrition-related data. Although the 10-year plan does not list specific activities focused on the continuous collection of data, two of the plan’s objectives indicate the Interagency Board’s concern with the timeliness of the data collected at the federal and state and local levels. The expert advisers also emphasized the need for regularly available data and suggested the following mechanisms for the ongoing collection of the information: • continuous national nutrition surveys, • addition of nutrition-related modules to existing surveys, • reliance on program data that are already collected, and • collection of longitudinal data (that is, data collected from the same sample over time).

Finally, responses to our survey of the users of nutrition data not only stated a desire for continuous data, but also indicated that data that are collected at regular and frequent intervals serve some important purposes. For example, data that are currently available on a regular basis have been used to measure progress toward the Healthy People 2000 objectives and to evaluate policies such as the fortification of infant formula.

Support for Reliable Inferences About Subpopulation Groups and Small Geographic Areas

The need for continued and improved information on subpopulation groups and small geographic areas is supported by several arguments. First, information is needed on specific populations known to be at risk for nutrition-related problems, such as Native Americans or homeless persons, in order to identify their needs and develop and target assistance programs. Second, some subpopulations, including Hispanics and the elderly, are growing rapidly. Their dietary patterns or nutritional needs may be different from the population as a whole; thus, information about these groups is needed to monitor their needs and to understand their effect on estimates of the prevalence of various nutritional problems in the overall population. Finally, the samples for the three major NNMRRP surveys are designed to yield national estimates. However, much of the planning for health and nutrition programs is conducted at the state and local levels. Hence, states and localities also need information on nutrition-related indicators for their populations.

For the reasons given above, past evaluations of federal nutrition monitoring have criticized the program for not covering specific population groups and geographic areas. Although none of the overall objectives of the 10-year comprehensive plan focus on the need for information on subpopulation groups, several of the activities do. For example, one planned action is to develop and implement a plan for improved coverage of groups at nutritional risk. Our advisory panels also discussed the importance of information on subpopulations, noting differences in the kinds of foods consumed in different regions of the country and by different ethnic and racial groups as well as differences in nutritional needs at different ages. Their suggestions for improving the availability of data on subpopulation groups and small geographic areas included different sampling strategies for different populations, contracts with states and localities to gather information on geographically-based populations, and indirect estimation to support inferences about subpopulation groups and small geographic areas.

The availability of data on important subpopulations and states and localities was one of the changes to the current data collection activities requested by respondents to our survey of data users. The users also emphasized the importance of information on subpopulations by describing the uses supported by currently available data on subpopulation groups such as determining dietary needs of the elderly, assessing differences between blacks and whites in the effect of obesity on diabetes, informing policies on the fortification of infant and toddler foods, and targeting a blood pressure screening program to the Mexican-American population.

Assistance to States and Localities for Nutrition Monitoring

State and local governments are interested not only in the applicability of federal nutrition monitoring data to their jurisdictions, but also in having federal assistance in collecting and interpreting their own data. The major justification for the emphasis on state and local monitoring is the wide range of uses that states and localities have for nutrition data. The examples in table 2.2, drawn from responses to our survey of data users, illustrate the utility of existing NNMRRP data for state and local governments. While NNMRRP data collection systems meet some of the state and local needs for nutrition monitoring data, state and local officials have called for additional technical assistance in analyzing and interpreting existing sources of data and for federal support in collecting their own data.

In addition to the support provided by the survey of data users, our other sources indicated the importance of building state and local capacity for nutrition monitoring. For example, past evaluations recommended assisting state and local nutrition monitoring activities. These recommendations are mirrored in the Interagency Board’s 10-year comprehensive plan, which clearly signals the importance of states and localities in the NNMRRP by devoting three of its six objectives to strengthening state and local monitoring activities.

The expert advisers also noted the role of states and localities in nutrition monitoring, but disagreed about the responsibility states should have for data collection. Some argued for state-based data collection that feeds into a federal system, and others argued for less state responsibility for data collection, but for increased consideration of state needs in federal data collection activities. Their suggestions for assisting states and localities include • providing financial assistance to states to determine their own data needs, • creating a federal-state partnership in which states can provide funds for some extra sampling or extra questions on federal surveys, • developing standardized modules of interest for state data collection • assisting state collection of data on subpopulations, and • providing technical assistance in data interpretation.

Approaches to Achieving the Selected Model Features

The rest of the report describes the strengths and limitations of current and potential approaches to achieving the selected model features. These approaches are listed in table 2.3. The alternate strategies were selected from the suggestions generated by the expert advisers using the criteria of responsiveness to criticisms of the current approach and feasibility. For example, the current approach to coordination—the Interagency Board—is criticized for its lack of authority over the member agencies. In contrast, an independent central authority could have influence over the NNMRRP agencies. The options were considered feasible if they were already used in other programs with similar issues (such as the lead agency approach for other cross-agency programs), past activities of the NNMRRP (such as the special study approach for information on subpopulations), or related current activities by NNMRRP agencies (such as indirect estimation).

Approaches to Coordinating the Nutrition Monitoring Program

As described in chapter 2, a model program would have a mechanism for coordinating the various nutrition monitoring activities to maximize the utility of the data and minimize the costs of its collection. This chapter first reviews the status of current NNMRRP activities to improve coordination. Then, two other possible coordination mechanisms— coordination by a central authority and coordination by a single lead agency—are examined. (Table 3.1 provides an overview of the strengths and limitations of the various approaches to coordination.)

Current Approach: the Interagency Board

As described in chapter 1, the NNMRRP meets multiple needs for nutrition-related data. Yet, historically, these needs have not been met by an integrated program with the capacity for evaluating data needs and making adjustments as those needs change. Instead, a fragmented system of activities developed over the decades as new needs for nutrition data were identified. For example, in the early 1930s, USDA developed its first national survey of household food consumption because data on the food supply provided no information about the distribution of food at the household and individual levels. Similarly, the nutrition component was added to the National Health Examination Survey in the early 1970s in response to a need for more information about hunger, and state-based surveillance systems were established in recognition of the primary role of states in providing services to populations at risk of nutritional problems.

To address concerns about the lack of coordination across the agencies involved in nutrition monitoring, the National Nutrition Monitoring and Related Research Act of 1990 required the Secretaries of Agriculture and Health and Human Services to implement a coordinated program of nutrition monitoring. The act specified several tools: an Interagency Board, the development of a comprehensive plan for the program, a council of outside advisers, and an integrated budget.

The Interagency Board created by the act has the difficult task of coordinating numerous data collection and analysis activities across several agencies that have traditionally had separate and distinct missions and operations. The Board has two chairpersons, one selected by the Secretary of Agriculture and one by the Secretary of Health and Human Services. For USDA, the chair is the Under Secretary for Research, Education, and Economics. The HHS chair is the Assistant Secretary for Health. Membership on the Board includes representatives of various agencies in USDA and HHS, as well as the Bureau of the Census, Agency for International Development, Bureau of Labor Statistics, Department of Defense, Department of Veterans Affairs, and Environmental Protection Agency, among others. The Executive Secretary for the Board rotates between USDA and HHS every 2 years. To facilitate coordination, the Interagency Board established working groups focused on survey comparability, food composition, and federal-state linkages and information dissemination.

The Secretaries and the Interagency Board were charged with developing a 10-year strategic plan, which was published in the Federal Register on June 11, 1993. The plan outlines a set of planning activities, including a general time frame and the lead agencies for each activity. The activities are organized around six objectives, listed in chapter 1 (see p. 14). The Interagency Board clearly recognizes the need for improved coordination since four of these objectives focus on either the coordination of the data collection activities or the comparability of the data.

To advise the Board on the development and implementation of the NNMRRP, the act established the National Nutrition Monitoring Advisory Council. The members of the Council represent academic institutions and other interested parties drawn from outside the federal government. The act also required the Interagency Board to submit annually a coordinated budget for nutrition monitoring.

Strengths

Both the concern that preceded passage of the act and the structure it created appear to have improved communication and cooperation among the agencies. The Board and its working groups provide mechanisms for communication and joint decision-making. Specific actions that demonstrate the increased coordination include the development of • common population descriptors for use in conducting and reporting the 1994-96 CSFII and the next NHANES, • a marketing and distribution plan for NNMRRP reports, • an automated dietary intake interview that would facilitate timely data release and linkage across CSFII and NHANES, and • a common set of questions on food security (a concept that addresses the certainty about having enough to eat) to be used in the Current Population Survey.

In addition, a jointly funded research project explored the possible linkage of CSFII and NHANES sampling plans. Alternate sampling designs were evaluated using the criteria of (1) ability to satisfy the separate objectives of the two surveys, (2) benefits in overall costs or analytic power, and (3) feasibility, especially in terms of burden on the survey respondents. The draft report from the contractor on the project emphasized the compromises one or both surveys would have to make to link their sample designs. For example, while the combination of the NHANES and CSFII into a single survey could yield a rich database of information on diet and health, the likely increase in respondent burden could reduce response rates and response quality. Another alternative—using linked samples for NHANES and CSFII—could decrease CSFII’s precision because NHANES’ sampling design is determined by the survey’s reliance on mobile examination centers to conduct physical exams of the respondents. The Interagency Board concluded that the two surveys should remain independent, although work on improving the comparability of the data should continue.

As required by the act, the Interagency Board submits an annual budget for the NNMRRP to the Congress. The budget report includes costs allocated for data collection, related research, information dissemination and exchange, and technical assistance; however, these different types of costs are not distinguished. Instead, the funds dedicated to nutrition monitoring and related activities are reported only by agency, not by type of activity. The budget report for fiscal years 1994-96 indicated that a total of $157.7 million was dedicated to nutrition monitoring or related research in 1994. Of that, $30 million was reported by the Centers for Disease Control and Prevention (which has responsibility for the HHS surveys and state-based surveillance systems) and $9.3 million was reported by the Human Nutrition Information Service (which had responsibility for NFCS and CSFII). The remainder was accounted for by agencies whose primary involvement in nutrition monitoring is related research. For example, HHS’ National Institutes of Health reported $25.9 million dedicated to NNMRRP activities and USDA’s Agricultural Research Service accounted for $50.7 million.

The budget report is useful in communicating to the Congress a general sense of the cost of the NNMRRP across agencies. However, funds for nutrition monitoring cannot always be disaggregated from other purposes of the data collection and research programs. As a result, the budget report contains only approximate amounts dedicated to nutrition monitoring and related research. Moreover, with the recent incorporation of the office responsible for NFCS and CSFII into ARS, determining which funds are dedicated to the monitoring activities and which are used for related research will be more difficult.

Limitations

The literature on the development of objectives to increase accountability for program results indicates that (1) objectives should be written in terms that can be used to judge progress toward achieving them and (2) implementation plans and specific measures of progress should be developed for the goals and objectives. In the 10-year plan, the objectives are stated in general, global terms so that it is not clear when an objective can be considered achieved.

For example, the first objective is to “provide for a comprehensive NNMRRP through continuous and coordinated data collection.” Neither the objective itself, nor the text following it clearly defines what the terms “comprehensive” or “coordinated” mean. Similarly, the activities are too vague to be considered implementation plans for the objectives; for example, no activity directly relates to the development of continuous data collection.

As another illustration, the activity, “identify ways to increase comparability within a dietary method to improve the quality and usefulness of data,” specifies neither the degree of quality required nor the uses that will be facilitated. Without more concrete, measurable objectives, there is little accountability for the program because progress toward the objectives cannot be assessed. (Other examples of activities listed in the 10-year plan can be found in table II.2 in appendix II.)

In addition, the plan did not describe how activities would be ranked by importance or addressed within current fiscal constraints. However, since the plan was published, the Interagency Board drafted an approach for ranking the 68 activities listed in the plan into three categories of priorities: (1) essential (mandatory, legislatively required), (2) necessary (critical but not mandatory), and (3) beneficial. An interagency implementation group, involving around 60 agency representatives, applied the approach. Twenty-five of the activities were ranked in the high-priority category, 33 were ranked as next most important, and only 10 were ranked as beneficial but not critical. While this is an important first step in setting priorities, the Interagency Board has not yet linked the top-ranked tasks to the costs and benefits of completing them. Such a framework could be used to understand the trade-offs in selecting one or another approach to each task.

Since an assessment of the effectiveness of the different approaches to the objectives requires information on the uses of the data, another obstacle to the Interagency Board’s implementation of a coordinated system is the absence of a comprehensive assessment of how nutrition monitoring data are used across the federal government and by data users in other settings.

For example, although USDA and HHS held a joint workshop to assess the needs of users of dietary intake data in August 1994, the workshop included only representatives of federal agencies.

Finally, the ability of the Interagency Board to coordinate the NNMRRP is limited by the lack of resources dedicated to coordination. The Interagency Board has no staff, although two people—one from USDA and one from HHS—have been given primary responsibility for organizing NNMRRP activities. The NNMRR Act gives the Secretaries the option of appointing an administrator for the program, but so far they have chosen not to exercise that option.

Potential Options

While recognizing the progress made by the Interagency Board, we also considered other coordination mechanisms. Two options for improving coordination—through a central authority or by a lead agency—are reviewed in detail in this section. In addition, other options discussed by our expert advisers are briefly presented.

Coordination Through a Central Authority

The suggestion of coordination through a central authority came in response to the lack of enforcement power held by the Interagency Board over the member agencies. The kind of central, coordinating agency envisioned by the expert advisers is most clearly exemplified by the executive offices in the White House. Therefore, to examine the advantages and disadvantages of having a central authority provide coordination, we reviewed the literature (including prior GAO reports and congressional hearings) and interviewed agency officials in three White House offices that have coordination responsibilities: Office of Management and Budget (OMB), Office of Science and Technology Policy (OSTP), and Office of National Drug Control Policy (ONDCP). As indicated by the brief descriptions of each office provided in table 3.2, the coordination tools used by the White House offices are similar to those used by NNMRRP’s Interagency Board, including interagency committees and working groups, development of plans, and review of budgets.

One potential advantage of elevating the coordination of nutrition monitoring to a high-level central authority is the increased participation of high-level officials in coordination activities. This participation could, in turn, increase the ability of the coordinating body to establish priorities. Currently, some of the agencies participating in the Interagency Board are represented by administrators or directors, while other agencies are represented by staff members who have no authority to change agency activities or establish priorities. In contrast, the political visibility of a program under White House management could encourage agencies to send representatives in positions of authority, capable of establishing priorities and committing resources to support them.

The political visibility that comes with the participation of a high-level central authority may also contribute to the effectiveness of the various coordination tools. For example, the Interagency Board compiles budgets obtained from each of the agencies involved in nutrition monitoring without reviewing them for consistency with the activities identified in the 10-year comprehensive plan. In contrast, the White House offices use the budget review process to bring activities of the agencies into line with overarching policy goals. For example, ONDCP can threaten to decertify an agency’s drug budget if it is not consistent with the National Drug Control Strategy. Decertification has no practical ramifications, but it sends a politically important message about the priority given to drug control activities by the White House.

An additional potential advantage of having a central authority provide coordination is that it can be a central location for assistance to all data users. For example, ONDCP has a Bureau of State and Local Affairs that works with state and local government agencies involved in drug control activities. The Bureau serves as a clearinghouse for information about state and local activities and uses conferences to increase communication with and among state and local officials. In addition, the Bureau can communicate the concerns of state and local governments and community groups to the federal agencies involved in drug control programs. For nutrition monitoring, such an office could provide a central contact point for users in a variety of settings, including federal, state, and local governments, food industry, and health care organizations.

Potential Limitations

Although the increased political visibility of a high-level central coordination office may facilitate the development of priorities, it may also increase the potential for political pressure on the data collection and research. A conflict between ONDCP and HHS over the data collection and reporting of drug data illustrates this issue. HHS was concerned about the degree of ONDCP’s involvement in how the data were collected and reported, while ONDCP expected HHS to meet its data needs. Political influence on the scientific agenda is also a concern for OSTP, where priorities may change with changing administrations even if scientific issues remain the same.

In addition, although White House offices have more influence over the budgets reported by the different agencies, the budgets developed for other programs that cut across agency jurisdictions share the limitation of the coordinated NNMRRP budget. Specifically, agency activities may serve multiple purposes, thus making it difficult to determine how much of the overall costs are dedicated to the interagency program. For example, Coast Guard patrol boats serve drug interdiction purposes, but are also used in search and rescue missions that are not related to drug control.

Thus, the Coast Guard can only estimate the portion of its resources dedicated to supporting national drug control efforts. Ambiguity about the portion of a multipurpose program that serves the interagency purpose makes it difficult to monitor the costs of the program.

Although central authorities like the White House offices are financed separately from the agencies they oversee, resource limitations persist. For example, HHS staff attributed part of their conflict with ONDCP to a lack of technical and substantive expertise among ONDCP staff. Similarly, a past director of OSTP identified limited staff resources as a reason why long-term planning received less attention than short-term problems.

Coordination Through a Lead Agency

Another possible approach to coordinating the NNMRRP is locating the responsibility within a single agency. To investigate this approach, we discussed the option with our expert panels and used program documents and evaluations conducted by GAO and the Congressional Budget Office to examine the experience of the High Performance Computing and Communications Program (HPCCP). Like the NNMRRP, the HPCCP involves multiple agencies with different strengths and missions. Unlike the NNMRRP, the oversight of the HPCCP is located in OSTP, which delegated the responsibility for coordinating the activities across the agencies to a single agency, the National Library of Medicine. While not a major player in high-performance computing, the National Library of Medicine was seen as an independent, unbiased participant with interest in and knowledge about the technology. The Library of Medicine’s role has been to pull together materials for program reports, convene meetings, and provide a clearinghouse.

Potential Strengths

HPCCP’s National Coordination Office shares two of the strengths of the Interagency Board: It appears to have facilitated communication among the agencies and coordination of individual activities. In addition, it has provided the Congress with a budget that looks at the costs of high- performance computing activities across agencies. Moreover, the National Coordination Office has the added advantage of providing the Congress and the public with a central contact point for information on high-performance computing.

Potential Limitations

“an explicit technical agenda, identifying and prioritizing specific technology challenges and establishing a framework of expected costs and results . . . clarify the program’s goals and objectives, focus efforts on critical areas, and serve as a baseline for measuring program progress and results.”

In addition, HPCCP did not have uniform guidelines for which research activities should be included in the budgets submitted by the different agencies, mirroring the difficulty the other coordination mechanisms have in tracking funds used for multipurpose programs. The National Coordination Office also shares the NNMRRP’s lack of budget and staff resources for coordination.

An additional concern about the lead agency model was raised by our expert advisers: If responsibilities for nutrition monitoring were located in one agency, nutrition monitoring might become a monopoly, serving the needs of only one agency. Its current dispersion across agencies allows the different components to serve different purposes. However, safeguards—such as that used for the HPCCP when it was located in an agency that did not have a large investment in high-performance computing, relative to some of the other agencies—could address this concern.

Other Coordination Suggestions

In addition to coordination by an interagency body with permanent staff and enforcement authority or a single lead agency, other suggestions for improving coordination of the NNMRRP were to locate nutrition monitoring in statistical agencies within the user departments, centralize the congressional appropriations process for nutrition monitoring activities, and ensure informed review of data collection plans by qualified staff at the Office of Management and Budget.

The first idea—locating nutrition monitoring in statistical agencies within the departments that use the data—was intended to focus the program on the quality of the data collected. However, a concern about this suggestion was that it could result in decreased responsiveness to the needs of the data users. We did not pursue it because the major surveys are already located in statistical or research branches of HHS and USDA. (NHANES is operated by the National Center for Health Statistics, and NFCS and CSFII were recently relocated to ARS.)

Similarly, OMB already has responsibility for reviewing the data collection activities of the agencies, although OMB staff reported that they rely on the agencies to describe coordination efforts. Finally, we did not review the advantages and disadvantages of centralizing the congressional appropriations process for nutrition monitoring activities because its relevance to the utility and efficiency of the data collection activities was not clear.

Conclusions

Lack of coordination across program activities has implications for both the effectiveness and the efficiency of the NNMRRP as a whole. The Interagency Board has made progress toward coordinating activities across the different agencies. However, the other approaches to coordination suggest mechanisms that could further strengthen the work of the Interagency Board.

First, because its responsibilities are shared by USDA and HHS, the Interagency Board does not provide a central contact point for users of nutrition monitoring data. A central contact could be established if the Secretaries of USDA and HHS used their option to appoint an administrator of the program or if responsibility for responding to requests for information about the program was assigned to a single agency, as it was for high-performance computing.

Second, the Interagency Board does not review the agency budgets it compiles for consistency with the overarching priorities of the program. Before such a review could occur, specific objectives and priority activities would need to be identified. Then, the Interagency Board could work with OMB to secure funding of NNMRRP priorities.

Approaches to the Continuous Collection of Data

Evaluating programs intended to reduce diet-related chronic disease, tracking progress toward health objectives, and monitoring changes in our diets are the kinds of activities that require continuous data. The advantages and disadvantages of the current and alternate approaches to collecting continuous data are summarized in table 4.1 and detailed below.

Current Approach: the State-Based Surveillance Systems

Although the provision of continuous data is one of the objectives identified in the Interagency Board’s 10-year plan, not one of the three major NNMRRP surveys—NFCS, CSFII, or NHANES—is implemented continuously. In fact, planned future administrations of two of the surveys have been postponed, potentially compromising the ability to monitor trends in diet-related health risks over time and evaluate the effect of any changes in food assistance policy.

NFCS has been administered at approximately 10-year intervals since the 1930s. Its next implementation was planned to begin in 1996, but is now tentatively scheduled for 1998, depending on funding. CSFII, originally intended to provide continuous data on dietary intake, has had three separate administrations: 1985-86, 1989-91, and 1994-96. After a 1-year pause in 1997, it is expected to resume for another 3-year period in 1998. Since the National Health Examination Survey gained a nutrition component in 1971, NHANES has been fielded three times: 1971-75, 1976-80, and 1988-94. Like NFCS, its future implementation is uncertain because of budget constraints. Its planned implementation in 1997 is now expected to be postponed.

Because of the lack of certainty about the implementation of the national nutrition surveys, the state-based surveillance systems are currently the primary source of continuous data in the NNMRRP. As described in chapter 1, PedNSS and PNSS rely on data from clinic records from publicly funded health, nutrition, and food assistance programs and BRFSS collects information through telephone interviews, with respondents (adults 18 years and over) identified through random digit dialing. While a valuable source of quick information for state and local program managers, the surveillance systems do not meet the needs of researchers or program decisionmakers who require either national data or in-depth food intake data.

Strengths

Compared to the national surveys, one of the strengths of the state-based surveillance systems is that they not only provide data continuously, but they are also able to process and report the data relatively quickly. For PedNSS and PNSS, information is collected as part of the process of receiving services from WIC and other publicly funded health, nutrition, and food assistance programs. Because they depend on program records, PedNSS and PNSS do not burden respondents the way surveys dependent on interviews do. The information is transmitted from the records of local health and nutrition programs to the state, which then forwards the records to HHS for analysis. Similarly, the data collected by the states for BRFSS are sent to HHS for processing. According to HHS officials, all three systems report data back to the states within a year and generally in less than 9 months. HHS has also helped states conduct their own analyses by distributing a standardized software package.

Another strength of the state-based surveillance systems is that the data they collect are directly linked to program decisions. Although PedNSS and PNSS include only a few indicators of nutritional deficiencies and behaviors, they are selected to support state data needs for program planning and management. For example, the data are used to target resources for the WIC program. Similarly, BRFSS data have been used to inform decisions about nutrition education programs, such as campaigns to encourage the consumption of five servings of fruits and vegetables a day.

Limitations

In contrast to the national surveys, the surveillance systems do not permit examination of diverse diet-health associations across the entire population. Instead of collecting extensive biochemical, anthropometric, and interview data, they focus on a narrow range of variables relevant to specific programs or nutritional risks. For example, PedNSS collects clinical data on weight and height, monitors infant feeding practices, and assesses anemia. PNSS collects information on anemia and behaviors associated with low-birthweight babies. BRFSS asks respondents to report on their consumption of fat, fruits, and vegetables. This focus limits the breadth of uses that can be supported by the data; however, as noted above, it also limits the burden placed on respondents.

While the systems are currently limited in the amount of dietary data they collect, HHS is exploring other methods of gathering these data. For example, with HHS support, the University of Texas examined the use of bar code data to look at dietary patterns. They concluded that the technology is not yet ready for use, but through a partnership with food manufacturers could be a promising method for the future. In addition, with USDA, HHS is evaluating the feasibility of collecting additional dietary data in the clinics that provide the PedNSS and PNSS records.

Another concern about surveillance systems is the quality and completeness of the data across the different states. For example, for PedNSS, error can be introduced by variations in practice in weighing infants, such as with or without the baby’s winter clothes, or by clerical errors in entering the data in states that do not have automated data systems. PNSS, which attempts to collect a wider range of information than PedNSS, suffers from missing data on several variables, such as the pregnancy risk factors of smoking and alcohol consumption. However, HHS provides technical assistance to help states standardize their data collection procedures. In fact, by flagging biologically implausible values for the physical measures, HHS analyses of the surveillance data help identify clinics that may have poor procedures.

The surveillance systems are also limited because, within the participating states, only certain groups of the population are covered. As described earlier, PedNSS and PNSS primarily provide information on mothers and children participating in the WIC program. As a result, data may not be available on other populations that are potentially at risk, such as homeless people or older children not eligible for WIC. In contrast, BRFSS has a wider target population, collecting data from randomly selected adults 18 years and over. However, neither adults in households without telephones nor children are covered by BRFSS. Since there is evidence that some health risk factors, such as smoking, are associated with living in a household without a telephone, this could affect estimates of the extent of diet-related risk factors as well.

Finally, not all states participate in the PedNSS and PNSS surveillance systems and, therefore, they are not good sources of national-level data on their populations of interest. In 1993, 38 states participated in PedNSS and only 20 states participated in PNSS. (In contrast, BRFSS has good state coverage; in 1993, only Wyoming did not participate.)

Potential Options

In addition to the state-based surveillance systems, we reviewed two other approaches to providing continuous data. The first option is a national nutrition-related survey that is operated continuously. To describe the strengths and weaknesses of this approach, we consulted with our expert advisers and interviewed managers of the current national nutrition monitoring surveys about the continuous operation of the surveys. The second option is the inclusion of nutrition-related questions on continuous non-NNMRRP surveys. The strengths and weaknesses of this alternative were explored by examining the NNMRRP’s recent experience developing food insecurity questions for the Current Population Survey (CPS). In addition, our expert advisers and the data users who responded to the survey made other suggestions, which are also briefly presented.

A Continuous NNMRRP Survey

An ongoing national survey is a possible approach to providing continuously updated information that addresses some of the limitations of the state-based surveillance systems. One of the current national surveys, the CSFII, was developed in response to calls for the continuous collection of individual data on food intake. It has not been implemented regularly in the past; however, current plans are for the ongoing implementation of CSFII as a 3-year survey followed by a 1-year pause for planning and development before the next 3-year period.

Potential Strengths

An ongoing survey has the potential to yield continuously updated, timely data for monitoring the nutritional status of the nation’s population. The increased timeliness of the data could decrease the current pressure from collaborating federal agencies to include components on each implementation of the periodic surveys. For example, in contrast to the recent implementation of NHANES, which attempted to meet as many data needs as possible within a single survey, the ongoing implementation of NHANES could contain a core set of data items that would be collected continuously, supplemented at intervals by rotating modules. Such a streamlining would also reduce the burden on respondents.

Moreover, the surveys could become more flexible by distinguishing between variables that change rapidly, variables that need regular but not continuous monitoring, and variables of emerging policy importance. The core set of items in continuous implementation could gather information on rapidly changing variables. Topical modules on issues that do not change as rapidly could be included on a regular schedule. Finally, as new issues arise, additional questions could be added.

In addition to increased timeliness and flexibility, a continuous survey operation could be more efficient than periodic surveys because current costly start-up activities, such as planning, designing sampling strategies, and training interviewers, would be diminished. Thus, the data could be collected for less cost per respondent. However, as described below, without concomitant streamlining of the survey, overall costs could increase.

Potential Limitations

While an ongoing survey could save money on start-up costs, it could increase costs overall because activities that are now funded and staffed sequentially would require a continuous flow of resources to be conducted concurrently. For example, as NHANES is currently implemented, staff change their activities as the survey moves through the phases of planning, implementation, analysis, and dissemination. If NHANES was implemented continuously, all of these activities would be going on at the same time. According to both USDA and HHS officials, the primary constraints on the continuous operation of their surveys are the need for dependable funding and sufficient staff resources. As described above, the absence of dependable funding has affected the frequency with which the national surveys are now implemented.

The development of a survey that continuously collects data on a core set of items and intermittently collects data on other issues raises two difficult issues. First, the definition of the core items is complex. An expert panel convened by the Federation of American Societies for Experimental Biology was charged with identifying a set of core indicators to assess the nutritional status of difficult-to-sample populations. The report summarizing the work of the panel noted that (1) the suitability of an indicator changes with the purposes for the data and (2) information on the determinants and the consequences of each indicator is also needed.The panel ended by identifying three sets of indicators—minimal, intermediate, and comprehensive—without recommending specific measures for the indicators.

The second issue with a continuous survey is the potential inflexibility of the core items once they are selected and implemented. While opportunities to test new methods are enhanced because the survey is in the field continually, making changes to the data collection procedures can be difficult because of the pressure to ensure that the measures are consistent over time. Without such assurance, changes in an indicator such as obesity may be the result of changes in how obesity is measured rather than changes in the prevalence of the condition itself. Our expert advisers suggested a survey with built-in periods of transition to allow the survey to incorporate new methods and new data elements as they emerge.

Adding Nutrition Questions to Existing Continuous Surveys

A third approach to obtaining frequently up-dated information is the addition of nutrition-related questions to existing continuous surveys. The potential of existing surveys to provide data regularly on some variables is demonstrated by plans that included NNMRRP questions on food security in the April 1995 Current Population Survey conducted by the Census Bureau. Food security is a concept intended to go beyond the idea of hunger to measure the availability of food for a family or individual.

The food security questions were developed by an interagency working group cochaired by HHS’ National Center for Health Statistics and USDA’s Food and Consumer Service. While the working group developed questions, USDA reserved space on the CPS for 1995. The question development process included determining how the data would be used and soliciting input from both federal and other data users. The food security module contains both core questions and supplemental ones. The Census Bureau will include the complete module. NNMRRP surveys could include the smaller core set of questions. If the initial implementation of the questions yields useful data, USDA plans to continue to support the inclusion of the food security questions annually in the CPS.

The same strategy of piggy-backing nutrition-related questions on continuous surveys could be used with other surveys, such as the Survey of Income and Program Participation or the National Health Interview Survey. The latter has the added advantage of collecting health data that could be linked to nutritional indicators.

Potential Strengths

The major advantage of this approach is the efficiency with which data on specific issues can be gathered. USDA will pay for the cost that the food security questions add to the CPS without having the responsibility or the cost of fielding and managing the survey itself.

Potential Limitations

A potential disadvantage of this approach is that, just as core questions on nutritional status could be inflexible, existing continuous surveys can be hard to change because of their momentum. Moreover, existing surveys have their own set of constraints and limitations. For example, while CPS could accommodate the addition of questions about food security, it probably could not accommodate a module obtaining data on an individual’s dietary intake over the last 24 hours. A 24-hour recall instrument requires considerable training to administer and adds substantially to the burden on the respondent. For these reasons, opportunities to piggy-back nutrition-related questions on other surveys may be limited.

Other Options for Continuous Data

In addition to the approaches reviewed, other actions were suggested by our expert advisers and data users who responded to our survey. Specifically, longitudinal surveys were suggested as an efficient way to collect data over time because a new sample does not have to be selected every time a survey is fielded. Since original respondents are followed up in subsequent administrations of the survey, longitudinal surveys can also be useful for tracking individual-level changes in food consumption behavior. However, longitudinal surveys have their own costs, including the need to collect additional data so that respondents can be found for later surveys and the likelihood of attrition as respondents either drop out or cannot be found.

Another suggestion was the collection of survey data using automated survey technology. Because direct entry of the data into an automated system can speed the processing and aggregation of the survey, it can also accelerate the release of the data for analysis. However, it does not affect the regularity with which the data are collected in the first place. HHS already uses automated data collection for NHANES. USDA is planning to automate the next administration of the CSFII.

Conclusions

The funding constraints that have caused the postponement of two of the national surveys jeopardize the availability of periodic data on the population as a whole. Although the state-based surveillance systems provide continuous information, they are inadequate to meet the need for data on the population at large or on in-depth nutrition and health status. Moreover, if approved, proposals to collapse funds for the WIC program with other food assistance programs into a block grant for the states could affect the major source of data for two of the state-based surveillance systems, PedNSS and PNSS.

Although the NNMRRP has pursued such creative solutions as including food security questions on the CPS, the availability of up-to-date information could worsen. To decide how best to meet the needs for continuous data in the future, the NNMRRP would first need to analyze the purposes that require frequently collected data and the current mechanisms for supporting those purposes. Within this framework, the strengths and limitations of the different approaches to increasing the frequency with which important indicators are measured could be weighed according to which purposes are supported and which are diminished.

Approaches to Supporting Reliable Inferences About Subpopulations

Subpopulations can be defined by geographic location as well as by age and sex (such as infants or elderly women), physiological characteristics (such as pregnancy), ethnicity or race (such as Hispanic or Native American), income, and the intersection of any of these groups (such as low-income children). As described in chapter 2, information on subpopulations is needed to appropriately target and evaluate programs that address nutrition-related programs. As summarized in table 5.1, this chapter describes current approaches and some potential options to responding to the calls for better information on subpopulation groups and small geographic areas in the NNMRRP. Other issues of federal assistance to states and localities for nutrition monitoring are discussed in chapter 6.

Current Approaches

Subpopulation groups are covered in two ways by current NNMRRP activities. First, the three national surveys use oversampling of certain groups to ensure the selection of enough respondents to support subpopulation estimates. NHANES focuses on racial and ethnic subpopulations, while CSFII and NFCS have included subpopulations defined by income, reflecting USDA’s focus on food assistance to low-income populations. The second way in which data on subpopulations are gathered is through the state-based surveillance systems—PedNSS, PNSS, and BRFSS.

Oversampling

Oversampling includes members of subpopulation groups in a sample at a rate greater than their proportion in the population. The purpose of oversampling is to ensure that data will be collected on enough group members to support inferences about the group as a whole. Oversampling is already used in the major NNMRRP surveys. NHANES III (1988-94) oversampled non-Hispanic blacks and Mexican-Americans, as well as persons 60 years or older and children 1-5 years old. NHANES staff indicated that two groups (Hispanics and persons 75 years and older) are likely to be important groups in the next implementation of the survey because both are growing and have significant health-related issues to study. In addition to estimates of the general population, the current administration of CSFII (1994-96) is expected to produce estimates for low-income populations through oversampling. Also, staff of both USDA and HHS surveys stated that the national surveys could oversample a state’s population, but that the state would have to finance the added costs.

Strengths

Oversampling has two major strengths. First, because it can be used in conjunction with a national survey, it has efficiencies of scale. Specifically, the planning and implementation costs are diminished because they are part of a larger survey. Second, data on both the group and the rest of the population are collected at the same time and with the same survey procedures, facilitating the comparison of the two population groups.

Limitations

Oversampling implies that the subpopulation group of interest is included in the sampling frames used to identify participants in the national surveys. However, not all special populations are well covered by a national sampling frame. For example, homeless individuals, persons who live in institutions, and American Indians and Alaska Natives living on reservations would not be included in the national household sampling frame.

Even for those individuals who are included in the sampling frame, oversampling may not be appropriate because of the costs incurred in screening for members of the group. Screening is the process of asking questions at sampled households to identify whether they represent (or include representatives of) the subpopulation of interest. Screening adds to the costs of the survey because enough households have to be sampled and screened to identify the smaller number of households or individuals that meet the definition of the subpopulation. To reduce screening costs, oversampling is most effective for subpopulations that are geographically clustered or fairly well represented in the general population, such as persons with low income who are often clustered by neighborhood. In contrast, oversampling is not appropriate for groups that are few in number or geographically dispersed, such as pregnant women.

A possible response to some of the limitations of oversampling is the use of multiple sampling frames. For some subpopulations, alternative frames or lists may be available that can be used in conjunction with the national sampling frame. Samples can be selected from both the subpopulation frame and the general population frame, and weighted estimates can then compensate for the fact that some group members could be selected from two different sources. For example, to oversample for the frail elderly, the elderly individuals identified in the sample drawn from the national sampling frame could be supplemented by samples drawn from lists of elderly who participate in congregate meals programs. This approach is useful because, relative to screening in the general population, it is an inexpensive way to identify members of the subpopulation.

State-Based Surveillance Systems

Although identified as a limitation in the previous chapter, the focus of the state-based surveillance systems on particular subpopulations can also be seen as a strength. As described in chapters 1 and 4, states use BRFSS to collect data on the health behaviors of their adult population. PedNSS and PNSS are sources of information on the nutritional status of low-income mothers and children. In addition, HHS is exploring opportunities to expand the program to other populations, such as schoolchildren. However, the other disadvantages of the systems, such as the limited amount of nutritional data they collect and the incomplete participation of the states, diminish their utility as a source of information on subpopulations.

Potential Options

The two major alternate approaches to providing subpopulation data are special studies and indirect estimation. Special studies are those that use a separate sampling frame from a national survey and are not necessarily conducted at the same time as a national survey. To describe the strengths and weaknesses of special studies as an approach to collecting information on subpopulations, we examined HHS’ past experience with the Hispanic Health and Nutrition Examination Survey (HHANES) and discussed the option with our expert advisers.

Indirect estimation uses data that are not direct observations of the group of interest to develop inferences about the subpopulation. Our review of this approach relied on reviews of technical literature, interviews with USDA and Bureau of Census staff responsible for indirect estimation programs, and consultation with methodological experts on our panel of advisers and in HHS. These and the current approaches address the suggestions made by the expert advisers and data users who responded to our survey.

Special Studies: the HHANES Model

An alternate approach to covering subpopulations is conducting a special study. A model of this approach is the Hispanic Health and Nutrition Examination Survey conducted in 1982 by HHS. HHS developed HHANES in response to recommendations made by the National Academy for Public Administration, which identified the Hispanic population as growing, likely to have low income, and potentially at risk for health and nutrition problems. HHANES was conducted as a separate study rather than integrated into the national survey because NHANES II was already completed and funding was not available to conduct HHANES as part of a national survey.

Potential Strengths

One of the advantages of conducting HHANES as a separate study was the opportunity to change the content of the survey instrument to address issues of special relevance to the Hispanic population. For example, unlike NHANES II, HHANES gathered information on health services use and gallstone disease. In addition, HHS took steps to address the cross-cultural issues of applying HHANES to different Hispanic populations. The survey instrument was translated into the idiomatic Spanish of each of the three Hispanic groups surveyed (Cuban-Americans, Mexican-Americans, and Puerto Ricans) and an appropriate plan for reaching out to the respondents was developed. For example, unlike the regular NHANES, which relies primarily on press releases and the formal leaders (such as mayors) of the places they have sampled to communicate the importance of participation in the survey, HHANES used informal leaders (such as church leaders) and Spanish-language media. The opportunity to tailor a special survey to the population of interest is an advantage of the approach, but it can be complicated for a population whose members speak multiple distinct languages and have varying degrees of assimilation into the U.S. population.

Although not demonstrated by HHANES, another advantage of a special study is the ability to study populations that are not appropriately studied through oversampling. As described above, this includes groups that are not included in the national sampling frame, such as homeless persons, and groups that are geographically dispersed or not well represented in the general population, such as pregnant and lactating women.

Potential Limitations

Unlike oversampling, in which the subpopulation is surveyed at the same time and usually with the same procedures as the population as a whole, the data collected by special studies may not be comparable to the population as a whole. For example, differences found between HHANES data on Hispanic groups and NHANES data on the nation could be the result of national changes in health and nutrition status during the gap between the two surveys rather than actual differences between the groups. Of course, conducting special studies in tandem with a national survey is possible. In fact, HHS is considering conducting special subpopulation surveys at the same time as the next NHANES.

Special studies that are conducted in addition to the national surveys clearly add to the overall cost of data collection. Specific costs for surveys of ethnic populations include bilingual interviewers, the translation of the survey instrument, and outreach to the group. Other costs are the development of a separate sampling frame and screening potential respondents to identify those that belong to the group of interest. Finally, if a special study is conducted at the same time as the national survey, the burden on the survey support facilities, such as the laboratories that analyze blood and urine specimens, is increased, which may slow down the processing and release of the data.

If a special study is conducted because the group of interest is not well covered by the national sampling frame, it needs a sampling frame that allows for generalization to the subpopulation group, which may be difficult to construct. For example, to use survey results to draw conclusions about the population of people who are without homes, one needs to sample from a complete list of the members of that group. For the homeless population, such a list would be very costly to construct. As a result, other means—such as sampling shelters—would have to be used. The survey results, though, would probably not be generalizable to the overall population of homeless people because these other means are likely to be incomplete.

Indirect Estimation

Indirect estimation (also known as small area or synthetic estimation) refers to procedures that use values of the variable of interest from an area or time other than the area and time of interest. For example, to develop an indirect estimate of the prevalence of iron-deficiency anemia in a county, the national estimate of iron-deficiency anemia can be adjusted based on the county’s demographic profile. Both USDA and HHS have experience with indirect estimation. Since the early part of the century, USDA has had a program to develop indirect estimates of crop yields. Although some information is available from state surveys of nonprobability samples of farmers, the USDA program adjusts these less dependable estimates so that they aggregate to the more reliable regional and national estimates that are based on a survey of a national probability sample of farmers. HHS has no regular program to produce indirect state estimates, but since 1968, it has supported the occasional development and evaluation of indirect estimates from National Health Interview Survey data. In addition, HHS produced state estimates from the National Natality and National Fetal Mortality Surveys conducted in 1980 using demographic data from the states to adjust the national estimates. Indirect estimation models range from the relatively straightforward adjustments of national or regional estimates to match local demographic profiles to more complex models.

Potential Strengths

The major strength of indirect estimation is that, compared to increasing survey sample sizes to obtain data for direct estimates at state and local levels, it is far less costly. In addition, it is a means of extending the usefulness of costly national survey data to inform decisions made at state or local levels of government. Moreover, it is an established approach. Indirect estimates of such variables as state and local populations, employment and unemployment, and crop yields are already used by the federal government in formulas for determining eligibility and benefit amounts for federal programs. Some state governments have also used indirect estimation to conduct analyses for economic and other types of programs.

Indirect estimation also responds to limitations of the data on which direct estimates might be based. Program records, such as those used by PedNSS and PNSS, have the advantage of timeliness since they are usually collected continuously, but their relevance may be limited because the data are collected for specific administrative purposes, not just for nutrition monitoring. Sample survey data, however, have the advantage of relevance, but the data can be costly to obtain at a level of detail that will support estimates for states, localities, or other subpopulation groups. In contrast, indirect estimation is an approach to producing timely, relevant, and detailed information without a major increase in cost.

Potential Limitations

The major limitation of indirect estimation is the difficulty of determining the quality of the estimate. The best way is to compare it to the true value for the population. For example, comparing an indirect estimate of a county’s population to census data on the county’s population would enable an assessment of the bias of the estimate. However, since indirect estimation is used when data are not available for direct estimation, such a comparison is usually not feasible. Moreover, the quality of the estimates yielded by a model changes for different populations and for different times. In other words, even if a model yields estimates that prove to be unbiased in comparison to direct observations for the same year, estimates from the same model for subsequent years may be biased if the relationships between the variables change over time.

This limitation has an implication for the use of indirect estimates. Specifically, indirect estimates may be difficult to defend in the political arena because they are based on models rather than direct observations. However, in the absence of direct data from other, more expensive approaches, indirect estimation is preferable to no information at all.

Although indirect estimation has been used successfully with other federal surveys, there are constraints on the development of indirect estimation programs for the major nutrition monitoring surveys. According to USDA and HHS staff, major obstacles include the lack of staff resources to support the program and the complexity of the surveys. Despite these concerns, both agencies have long experience with indirect estimation in other arenas, which could be applied to nutrition monitoring.

Conclusions

Information on subpopulation groups and small geographic areas is used to identify nutrition-related problems that are associated with specific populations and to target programs to those most at risk. Different strengths and weaknesses are associated with the four approaches to meeting this need that we reviewed. Surveillance systems and indirect estimation programs are likely to be less expensive than oversampling or special surveys. However, oversampling and special surveys can yield more detailed information than the surveillance systems that rely on program records and more dependable estimates than indirect estimation models. Of the four approaches, only indirect estimation has not been a part of NNMRRP activities at one time or another. From programs in other areas, this approach appears to be a potentially efficient means of expanding the information available on subpopulations and small geographic areas. However, before the Interagency Board can determine what priority to give to this promising approach, a complete picture of what needs would be met by each option is required.

Approaches to Assisting States and Localities

State and local governments are interested not only in the applicability of federal nutrition monitoring data to their jurisdictions, but also in having federal assistance in collecting and interpreting their own data. This chapter follows on the previous discussion about NNMRRP support for reliable inferences about state and local populations to describe current and potential options for assisting states and localities in their own monitoring activities.

The NNMRRP currently assists states through the state-based surveillance systems, which provide technical and other assistance to states. The alternative that we examined, community-based nutrition monitoring, is a response to states and localities that are interested in building their capacity to collect their own data. The strengths and weaknesses of these two approaches to assisting states and localities are summarized in table 6.1 and detailed below.

Other suggestions by our expert panelists—such as providing financial and technical assistance, developing standardized modules of interest for state data collection activities, and assisting state collection of data on subpopulations—can be implemented as part of either of the two approaches.

Current Approach: the State-Based Surveillance Systems

The strengths and limitations of the surveillance systems as sources of continuous data and of information on state and other population groups have been discussed in chapters 4 and 5. However, they have additional strengths and limitations related to their ability to meet state and local needs for assistance.

Strengths

Surveillance systems currently balance the federal interest in information that is collected in a standard format across states and the states’ needs for flexibility. For example, with BRFSS, HHS supplies states with standardized modules of questions, training in collecting the data, and support in analysis and reporting. In addition, changes to the survey content are made in consultation with state participants in the surveillance system, and states have the opportunity to add their own questions to the survey.

The surveillance systems have also played a role in building state capacity for data collection and analysis. Specifically, some states have used their experience with BRFSS to implement their own telephone surveys using the random digit dialing procedure. In addition, standardized software developed by HHS for PNSS has enabled states to generate their own reports.

Yet another strength of the surveillance systems is the foundation of federal-state partnership it provides for future improvements of federal technical assistance to state and local nutrition monitoring activities. For example, as mentioned in previous chapters, HHS is exploring ways to use the surveillance systems to gather additional nutritional data, such as more in-depth information on dietary intake, and to cover new populations, such as schoolchildren. Additional technical assistance in data analysis and interpretation could also be provided through the surveillance system structure.

Limitations

While the surveillance systems could be expanded and improved to further respond to the interest of states and localities in receiving more technical assistance, they are limited in their flexibility because of the federal interest in standardization across states. In addition, users of surveillance system data who responded to our survey identified some concerns about the surveillance systems. Specific issues included the availability of the data to localities, timeliness of HHS’ processing of the data, and the formats of the reports that HHS provided. Recommendations for improving the surveillance systems included increasing local access to data, reducing the time it takes HHS to process the data, simplifying reports for local users, and providing additional technical and financial assistance in data collection and interpretation.

Potential Option: Community-Based Nutrition Monitoring

The federal-state linkage forged by the surveillance systems could be further extended to support community-based nutrition monitoring. To explore the strengths and limitations of this approach, we reviewed the literature on two models of community-based programs. HHS has funded Planned Approach to Community Health (PATCH) projects, which used the BRFSS survey instrument as the basis for a community needs assessment. The survey data, in combination with interviews with knowledgeable informants in the community, were used to plan health promotion programs. HHS provided technical assistance in the needs assessment portion of the projects and small awards of a few thousand dollars for project activities.

Researchers at Cornell University developed a similar model specifically for nutrition monitoring that was pilot-tested in three New York counties with funding from the State Department of Health and technical assistance from the university. In their approach, a coalition of interested community members first articulates potential information needs. Then, the group selects specific needs on the basis of the likelihood that the data will be used by the community. To meet the selected information needs, feasible sources of routinely available data are identified. According to the model, data collection should depend on procedures that are already in place, from such sources as program participation records, patient charts, and school health screenings. In its reliance on program records, this approach is similar to PedNSS and PNSS. However, there is no expectation that the same issues will be targeted or the same sources of data used in each community. To facilitate the final component of the system—the communication of monitoring results—a network of users must be cultivated and the appropriate vehicle for communicating the results must be used. Drawing on the experience of these projects and HHS’ PATCH program, the following strengths and limitations of community-based nutrition monitoring were identified.

Potential Strengths

Evaluations of both types of community-based programs found evidence that local capacity for collecting and using data was developed. By involving community members in assessing needs, PATCH built skills in identifying health risks and cultivated community and organizational supports for health promotion programs. In the New York counties that pilot-tested the Cornell model of community-based nutrition monitoring, data were collected and compiled to describe access to food and nutrition services and nutritional health of county residents. To disseminate the information and guide decisions based on the data, interagency coalitions were formed. These coalitions brought together local nutrition-related professionals, so that nutrition interventions were better coordinated and information was shared.

The limited federal investment in the community-based programs is another strength. HHS supported PATCH with small grants and technical assistance, while the community-based nutrition monitoring projects were sponsored by the state and received technical assistance from the state’s land grant institution.

“At the end of the pilot stage, one county obtained local funding to continue their monitoring activities, including the issue-based coalitions; in the other counties, local nutrition councils are coordinating continued monitoring efforts.”

The program has also spread to other counties, some of which have initiated monitoring activities without outside funding.

Potential Limitations

The experience with community-based programs indicates that communities require a set of resources—specifically, technical skills and dedicated personnel—to fully benefit from the projects. One evaluation of PATCH found that the projects were most effective in communities that already had human services and health professionals who were involved in community health promotion efforts. In addition, the projects that had directors seemed to have the greatest implementation successes. Both kinds of projects required considerable time and effort to collect and interpret data. Moreover, while HHS provided technical assistance in using the BRFSS survey in the local needs assessments, additional assistance was needed to help communities set priorities based on the data.

The reliance on BRFSS was a limitation for PATCH because BRFSS did not necessarily include the issues of primary concern for the community. For example, one PATCH project was specifically interested in water quality, which is not addressed in BRFSS. In contrast, the Cornell model emphasized local sources of data. Based on the initial experience with PATCH, HHS no longer expects communities to use BRFSS for its needs assessment.

Conclusions

Data from the HHS surveillance systems are already extensively used in state and local program planning. In addition, the systems are flexible and are exploring ways to increase the information they collect on dietary intake. However, they seem less responsive to the needs of localities than to the needs of states. Recent efforts to build local capacity for data collection and interpretation indicate that community-based programs are a promising approach to responding to local needs for nutrition information. Before the Interagency Board can decide what priority to place on community-based nutrition monitoring, however, it must first identify the objectives of the NNMRRP that would be furthered and the importance of these objectives relative to others that are competing for program resources.