Food Assistance: Performance Measures for Assessing Three WIC Services
Released: 2001-02-28T00:00:00

Letter

Background

As part of its administration of the WIC program, FNS makes grants to the 88 state agencies that in turn provide program benefits—supplemental foods and a variety of nutrition services—to participants through more than 1,800 local WIC agencies. The state agencies develop guidelines intended to ensure that local agencies effectively deliver WIC benefits to eligible participants, and monitor local agencies’ compliance with these guidelines. Local agencies serve participants directly or through one or more service delivery sites or clinics located within their service areas. Staff at local WIC agencies and clinics approve applicants for participation; provide food benefits, typically in the form of vouchers that can be exchanged for WIC-approved supplemental foods at specified grocers; provide nutrition education; and make health referrals to eligible individuals.

For fiscal year 2000, grants for this program totaled $4.1 billion—$3 billion for supplemental foods and $1.1 billion for nutrition services and program administration costs. Grants for nutrition services and administration (NSA) are typically used to support activities in four broad categories: participant services, including health care referrals; nutrition education; breastfeeding promotion and support; and program administration, such as planning and budgeting. This report focuses on three specific services— nutrition education, breastfeeding promotion and support, and health referrals. Table 1 describes some of the specific activities that local WIC agencies provide as part of these services.

FNS Has an Outcome- Based Measure for One of Its Three Nutrition Services

FNS established the breastfeeding initiation rate as an outcome-based measure for the WIC program’s breastfeeding promotion and support activities, but has no outcome measures for its nutrition education or health referral services. While this single measure allows FNS to examine one aspect of the impact of its services on WIC clients, it does not measure ther important aspects of this service’s impact—such as the length of time that WIC mothers breastfeed their infants and the percentage of daily nutrition an infant obtains from breastfeeding. FNS has attempted to develop outcome-based measures for the WIC program’s nutrition education and health referral services but has not yet been successful in implementing these measures. FNS has identified a measure for nutrition education but has not implemented it because of resource constraints. For health referrals, FNS has been unable to identify a measure that, among other things, would permit it to appropriately link the service’s activities with a desired outcome.

Breastfeeding Initiation Rate Measures Outcomes of Breastfeeding Promotion and Support Activities

FNS established the breastfeeding initiation rate as its outcome-based measure for breastfeeding promotion and support activities as part of its fiscal year 2000 performance plan. The measure represents the percentage of WIC infants between 7 and 11 months of age who, at some time, have been breastfed. FNS first reported the results of this measure in May 2000 as part of its 1998 WIC Participant and Program Characteristics Study (Characteristics Study), a biennial report on the characteristics of WIC participants and program activities. The overall rate of 42 percent for 1998 will serve as a baseline against which FNS will gauge future years’ progress. FNS has collected data for fiscal year 2000, which it is in the process of analyzing and expects to be available in the spring of 2002.

Although FNS established an outcome measure for breastfeeding initiation, this measure does not assess the full range of potential impacts of breastfeeding promotion and support activities on WIC participants. The breastfeeding initiation rate includes only WIC infants who are breastfed at some time. It does not address other aspects of breastfeeding, such as duration or the percentage of daily nutrition that an infant obtains through breastfeeding. FNS attempted to collect data on breastfeeding duration as part of its 1998 Characteristics Study. However, only 40 of the 88 state agencies were able to provide this information for at least 75 percent of WIC infants in their states. According to FNS, these state agencies were unable to provide the data because their automated information systems did not contain complete data on each participant. FNS anticipates the number of state agencies reporting sufficient data to increase as states become accustomed to reporting these data. FNS has not attempted to measure the percentage of daily nutrition an infant obtains through breastfeeding. While acknowledging the importance of this measure, FNS has not yet identified an appropriate means of implementing it.

FNS’ Efforts to Implement Outcome-Based Measures for Nutrition Education and Health Referrals Have Been Unsuccessful

FNS established outcome-based measures for both the nutrition education and health referral services in its fiscal year 2000 performance plan. However, it was unable to implement these measures and did not include outcome-based measures for the two services in its 2001 performance plan.

FNS has identified an outcome measure for WIC nutrition education services, but has not implemented it primarily because of resource constraints. FNS intended to evaluate the impact of its nutrition education services by assessing the nutritional quality of meals consumed by a sample of WIC participants. FNS planned to implement this measure by extracting data from a national dietary survey of the U.S. population. This periodic survey, the Continuing Survey of Food Intakes of Individuals, includes WIC participants. However, FNS determined that the survey did not contain a large enough sample of WIC participants to establish national estimates. FNS lacked the resources needed to develop another sample with a large enough number of WIC participants to establish national estimates. FNS officials said that the agency is exploring ways to overcome the sample size problem for future implementation of this measure.

In addition, FNS has not established a measure for WIC health referral activities that appropriately links the service’s activities with a desired outcome. Initially, FNS intended to use the percentage of WIC child participants who were immunized as the outcome measure for its health referral services. To implement this measure, FNS planned to use data gathered by the federal Centers for Disease Control and Prevention (CDC) from its National Immunization Survey. In its 1999 Survey, the CDC began reporting immunization data for children participating or not participating in WIC. However, with its 2001 performance plan, FNS discontinued this measure. Agency officials gave two reasons for this decision. First, they noted that FNS is not responsible for ensuring that children participating in WIC are actually immunized, but rather those children needing immunizations are referred to doctors as appropriate. Second, they pointed out that immunization data provided by CDC on WIC participants did not permit FNS to measure the extent to which WIC health referrals actually contributed to the immunization of WIC child participants.

FNS officials noted that the problems they faced in finding a measure that appropriately linked WIC health referral activities to a desired outcome are not unique to the health referral services. They pointed out that it is difficult to measure the impact of any of the individual services WIC offers because, among other things, other variables also influence the behavior these services are intended to promote. For example, there are several other state and local programs that, like WIC, are aimed at improving health through nutrition education. Separating the effects of these efforts from those of the WIC program is difficult at best.

FNS Relies on a Variety of Output Measures to Monitor Program Performance

FNS employs a large number of program output measures to gauge performance of the WIC nutritional services it offers. Generally, these measures are used to examine the types and quantities of services the state agencies provide and whether the agencies are in compliance with grant expenditure and other program requirements. (A detailed list of output measures can be found in app. I.) To monitor these measures, FNS depends in large part on data provided by state and local agencies through routine financial reports, its biennial Characteristics Study, and FNS’ management evaluations of state and local agency operations. Of these monitoring methods, FNS relies heavily on its management evaluations to measure state and local WIC agency compliance with program requirements. However, weaknesses in the procedures for conducting evaluations of WIC agencies reduce its value as a performance oversight tool. FNS recently completed an examination of its practices for conducting management evaluations for all FNS programs, including WIC, and made several recommendations for improvements.

FNS uses several financial reports submitted by state agencies to measure the extent to which state and local agencies meet expenditure requirements. The principal measure is whether states are adhering to a legislatively established requirement for spending on nutrition education activities and breastfeeding promotion and support activities. Combined spending for these two activities must equal at least one-sixth of the NSA expenditures plus a target amount for breastfeeding promotion and support established by FNS at the beginning of each fiscal year. The Addendum to the WIC Program Annual Closeout Report, prepared by each state, serves as the primary source of data on the state’s compliance with this expenditure requirement. For fiscal year 1999, the most recent year for which complete data were available, FNS reported that two states and four Indian tribal organizations did not meet this minimum spending requirement. In addition, the WIC Program Annual Closeout Report each state prepares provides data on the final status of the state’s grant and costs for the report year.

To measure the nature and amount of program services WIC agencies provide, FNS primarily uses data obtained from state and local agencies and reported in FNS’ Characteristics Study. As noted earlier, the Characteristics Study provides information on a variety of program activities. Table 2 lists some of the key program activities in the three areas of service included in the Characteristics Study as well as information on the percentage of local WIC agencies offering the activity.

According to FNS officials, survey data from local agencies, of the sort seen in table 2, may no longer be collected for the Characteristics Study for 2002 and beyond because research funds are unavailable for that purpose. These officials explained that the Agriculture, Rural Development, Food and Drug Administration, and Related Agencies Appropriations Act of 1998 (P.L. 105- 86) effectively transferred primary responsibility for research on food assistance and nutrition programs from FNS to the Department’s Economic Research Service, which has not subsequently funded the collection of the local WIC agency data. FNS officials believe that these data are important for monitoring the nature and level of services being delivered by local agencies. To address this and other research concerns, FNS has attempted to regain direct responsibility over research funding for food assistance and nutrition programs.

To measure overall compliance with program requirements, including those related to nutrition education, breastfeeding promotion and support, and referral services, FNS regional offices conduct management evaluations at each of the 88 state WIC agencies and at a sample of local agencies. In general, each of the state agencies and selected local agencies are reviewed every 3 years. According to FNS, management evaluations are the primary means of measuring state and local WIC agency compliance with program requirements. As part of the management evaluations, staff in FNS’ regional offices conduct interviews, review planning documents, observe interactions, and examine participants’ files. Listed below are some of the types of nutrition education, breastfeeding promotion and support, and health referral service program requirements that can be assessed during management evaluations: State agency WIC plan includes nutrition education goals and action Local agencies offer each participant a minimum of two nutrition education contacts during each 6-month certification period and documents the provision of service or the participant’s refusal or inability to participate, State agency has developed or identified nutrition education resources and materials for use by local agencies, and Nutrition education material provided is easily understood by participants.

State and local agencies have a designated breastfeeding promotion and support coordinator who provides technical assistance to local agencies, State agencies have developed standards for local agencies for use in providing breastfeeding promotion and support, and Local agency staff has received training on promotion and management of breastfeeding promotion and support.

Local agency staff refers participants, as appropriate, to private or public health care services.

FNS has not made full use of management evaluations to measure program performance and provide needed oversight. In large part this is due to inconsistencies in the procedures used to conduct the evaluations, which have caused problems in interpreting evaluation results as well as in comparing and analyzing results across evaluations. FNS recently examined its practices for conducting management evaluations for all FNS programs, including WIC. Results of this review were highlighted in a January 2001 report to the Administrator. The report noted that (1) management evaluations at state and local levels were not being conducted uniformly across FNS regional offices; (2) standard vocabulary did not exist for identification of certain concepts; (3) standard definitions were needed for such key terms as deficiency (a judgment that the agency being reviewed is not in compliance with program requirements and must take corrective action); (4) additional staff time needed to be allocated to testing and training prior to implementation; and (5) a learning, feedback, and modification period would be needed to support consistent data collection and the development of reporting methods. The task force recommended a number of actions aimed at obtaining more consistent information that, if implemented, could provide FNS with better data for identifying national trends, quantifying deficiencies, developing policies, and measuring mission effectiveness. Agency officials told us that a response from the Administrator is expected by April 2001.

Agency Comments and Our Evaluation

We provided a draft of this report to USDA’s Food and Nutrition Service for review and comment. We met with FNS officials, including the Director, Office of Analysis and Nutrition Evaluation; the Director, Supplemental Food Program Division; and officials in the Grants Management Division. These officials generally agreed with the information presented in this report. However, they cautioned that the report addressed performance measures as they pertained to three nutrition service components of the WIC program and not the program in its entirety, which includes supplemental foods and other aspects of nutrition services. FNS officials believed that research on the effects of the WIC program as a whole indicated that it has contributed to a number of desirable outcomes including improved diets and infant feeding practices, reduced incidence of anemia among children, reduced incidence of infant mortality and low birthweight infants, and reduced postpartum medical expenses. Also, these officials provided us with a number of technical comments that we incorporated into the report, as appropriate.

Scope and Methodology

In developing information for this report, we spoke with and obtained data from officials at FNS’ headquarters in Alexandria, Virginia, and the seven FNS regional offices—Northeast in Boston, Massachusetts; Mid-Atlantic in Robbinsville, New Jersey; Southeast in Atlanta, Georgia; Midwest in Chicago, Illinois; Mountain Plains in Denver, Colorado; Southwest in Dallas, Texas; and, Western in San Francisco, California. We reviewed legislation, policies, procedures, and practices; reviewed reports issued by GAO and others related to WIC; and interviewed various officials responsible for administering the WIC program. We also reviewed FNS’ strategic and annual performance plans and other documents developed as part of implementing the Results Act. We conducted our work from September 2000 through January 2001 in accordance with generally accepted government auditing standards.

We are sending copies of this report to the appropriate congressional committees; interested members of the Congress; the Honorable Ann M. Veneman, the Secretary of Agriculture; and other interested parties. We will also make copies available to others upon request.

If you or your staff have any questions about this report, please contact me at (202) 512-7215. Key contributors to this report are listed in appendix II.

Output Measures for Monitoring Program Requirements and Activities for the Three Services

Special report (biennial survey by public health and nutrition program directors)

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Abiud Amaro, Clifford Diehl, Judy Hoovler, Angela Miles, and Ron Wood made key contributions to this report.

Ordering Information

The first copy of each GAO report is free. Additional copies of reports are $2 each. A check or money order should be made out to the Superintendent of Documents. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders by phone: (202) 512-6000 fax: (202) 512-6061 TDD (202) 512-2537 Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

To Report Fraud, Waste, or Abuse in Federal Programs

Web site: http://www.gao.gov/fraudnet/fraudnet.htm e-mail: fraudnet@gao.gov 1-800-424-5454 (automated answering system)