Food Assistance: Performance Measures for Assessing Three WIC Services
Published: 2001-02-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO examined the performance measures that the Food and Nutrition Service (FNS) uses to assess the nutrition education, breastfeeding promotion and support, and health referral services provided to participants in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) program.
section_title: What GAO Found
paragraphs: GAO found that FNS has an outcome-based measure for one of the three nutrition services--breastfeeding promotion and support. However, the measure, breastfeeding initiation rate, examines only one of several important aspects of the service's possible impact on WIC participants. In addition, several obstacles have hindered FNS' efforts to develop and implement outcome-based measures for nutrition education and health referral services for the WIC program. These include difficulties in identifying measures that would allow the agency to appropriately link a particular service's activity to a desired outcome and resource constraints affecting FNS' ability to collect data needed to implement a proposed measure.

section_title: Letter
paragraphs: The Special Supplemental Nutrition Program for Women, Infants and  Children (WIC) is a federally funded $4.1 billion-a-year nutrition assistance  program administered by the U.S. Department of Agriculture’s Food and  Nutrition Service (FNS). During fiscal year 2000, this program provided $3  billion for supplemental foods and $1.1 billion for nutrition services and  administration to assist lower-income pregnant, breastfeeding, and  postpartum women, infants, and children up to age 5 who are at nutritional  risk. FNS provided annual cash grants to support program operations at 88  state-level agencies. These 88 agencies administered the program through  more than 1,800 local WIC agencies.
To help the Congress better understand the costs of delivering nutrition  services and administering WIC, the William F. Goodling Child Nutrition  Reauthorization Act of 1998 (P.L. 105-336) directed GAO to assess various  aspects of WIC nutrition services and administration. This report, the third  in a series responding to this request, examines the performance measures  that FNS uses to assess the nutrition education, breastfeeding promotion  and support, and health referral services provided to WIC program  participants.
We based our evaluation on the performance measurement framework  contained in the Government Performance and Results Act of 1993 (Results  Act), which, among other things, encourages agencies to measure program  performance by determining the extent to which intended program  outcomes have been achieved. Program outcomes are the results of  delivering a program’s products and services, while program outputs are  the products and services delivered. An example of the distinction between  the two concepts can be seen in the case of a job-training program. In such  a program, an output could be the number or percentage of program  participants who completed the training. A program outcome, on the other  hand, could be the number or percentage of program participants  employed 1 year after the training. Outcome measures on which objective  data can be collected at reasonable cost can be difficult to develop,  particularly for programs that are intended to influence the behavior of  individuals. While emphasizing the use of outcome measures, the Results  Act recognized that the output measures traditionally used by agencies for  measuring performance remain critical to program management.
Consistent with the Results Act’s approach to performance measurement,  this report discusses FNS’ use of both outcome and output performance  measures. Specifically, it provides information on how FNS measures (1)  the outcomes of three services provided by the WIC program—nutrition  education, breastfeeding promotion and support, and health referral  services; and (2) program outputs for these three service areas.
subsections: section_title: Background
paragraphs: As part of its administration of the WIC program, FNS makes grants to the  88 state agencies that in turn provide program benefits—supplemental  foods and a variety of nutrition services—to participants through more  than 1,800 local WIC agencies. The state agencies develop guidelines  intended to ensure that local agencies effectively deliver WIC benefits to  eligible participants, and monitor local agencies’ compliance with these  guidelines. Local agencies serve participants directly or through one or  more service delivery sites or clinics located within their service areas.  Staff at local WIC agencies and clinics approve applicants for participation;  provide food benefits, typically in the form of vouchers that can be  exchanged for WIC-approved supplemental foods at specified grocers;  provide nutrition education; and make health referrals to eligible  individuals.
For fiscal year 2000, grants for this program totaled $4.1 billion—$3 billion  for supplemental foods and $1.1 billion for nutrition services and program  administration costs. Grants for nutrition services and administration  (NSA) are typically used to support activities in four broad categories:  participant services, including health care referrals; nutrition education;  breastfeeding promotion and support; and program administration, such as  planning and budgeting. This report focuses on three specific services— nutrition education, breastfeeding promotion and support, and health  referrals. Table 1 describes some of the specific activities that local WIC  agencies provide as part of these services.
section_title: FNS Has an Outcome- Based Measure for One of Its Three Nutrition Services
paragraphs: FNS established the breastfeeding initiation rate as an outcome-based  measure for the WIC program’s breastfeeding promotion and support  activities, but has no outcome measures for its nutrition education or  health referral services. While this single measure allows FNS to examine  one aspect of the impact of its services on WIC clients, it does not measure  ther important aspects of this service’s impact—such as the length of time  that WIC mothers breastfeed their infants and the percentage of daily  nutrition an infant obtains from breastfeeding. FNS has attempted to  develop outcome-based measures for the WIC program’s nutrition  education and health referral services but has not yet been successful in  implementing these measures. FNS has identified a measure for nutrition  education but has not implemented it because of resource constraints. For  health referrals, FNS has been unable to identify a measure that, among  other things, would permit it to appropriately link the service’s activities  with a desired outcome.
subsections: section_title: Breastfeeding Initiation Rate Measures Outcomes of Breastfeeding Promotion and Support Activities
paragraphs: FNS established the breastfeeding initiation rate as its outcome-based  measure for breastfeeding promotion and support activities as part of its  fiscal year 2000 performance plan. The measure represents the percentage  of WIC infants between 7 and 11 months of age who, at some time, have  been breastfed. FNS first reported the results of this measure in May 2000  as part of its 1998 WIC Participant and Program Characteristics Study  (Characteristics Study), a biennial report on the characteristics of WIC  participants and program activities. The overall rate of 42 percent for 1998  will serve as a baseline against which FNS will gauge future years’  progress. FNS has collected data for fiscal year 2000, which it is in the  process of analyzing and expects to be available in the spring of 2002.
Although FNS established an outcome measure for breastfeeding initiation,  this measure does not assess the full range of potential impacts of  breastfeeding promotion and support activities on WIC participants. The  breastfeeding initiation rate includes only WIC infants who are breastfed at  some time. It does not address other aspects of breastfeeding, such as  duration or the percentage of daily nutrition that an infant obtains through  breastfeeding. FNS attempted to collect data on breastfeeding duration as  part of its 1998 Characteristics Study. However, only 40 of the 88 state  agencies were able to provide this information for at least 75 percent of  WIC infants in their states. According to FNS, these state agencies were  unable to provide the data because their automated information systems  did not contain complete data on each participant. FNS anticipates the  number of state agencies reporting sufficient data to increase as states  become accustomed to reporting these data. FNS has not attempted to  measure the percentage of daily nutrition an infant obtains through  breastfeeding. While acknowledging the importance of this measure, FNS  has not yet identified an appropriate means of implementing it.
section_title: FNS’ Efforts to Implement Outcome-Based Measures for Nutrition Education and Health Referrals Have Been Unsuccessful
paragraphs: FNS established outcome-based measures for both the nutrition education  and health referral services in its fiscal year 2000 performance plan.  However, it was unable to implement these measures and did not include  outcome-based measures for the two services in its 2001 performance plan.
FNS has identified an outcome measure for WIC nutrition education  services, but has not implemented it primarily because of resource  constraints. FNS intended to evaluate the impact of its nutrition education  services by assessing the nutritional quality of meals consumed by a sample  of WIC participants. FNS planned to implement this measure by extracting  data from a national dietary survey of the U.S. population. This periodic  survey, the Continuing Survey of Food Intakes of Individuals, includes WIC  participants. However, FNS determined that the survey did not contain a  large enough sample of WIC participants to establish national estimates.  FNS lacked the resources needed to develop another sample with a large  enough number of WIC participants to establish national estimates. FNS  officials said that the agency is exploring ways to overcome the sample size  problem for future implementation of this measure.
In addition, FNS has not established a measure for WIC health referral  activities that appropriately links the service’s activities with a desired  outcome. Initially, FNS intended to use the percentage of WIC child  participants who were immunized as the outcome measure for its health  referral services. To implement this measure, FNS planned to use data  gathered by the federal Centers for Disease Control and Prevention (CDC)  from its National Immunization Survey. In its 1999 Survey, the CDC began  reporting immunization data for children participating or not participating  in WIC. However, with its 2001 performance plan, FNS discontinued this  measure. Agency officials gave two reasons for this decision. First, they  noted that FNS is not responsible for ensuring that children participating in  WIC are actually immunized, but rather those children needing  immunizations are referred to doctors as appropriate. Second, they pointed  out that immunization data provided by CDC on WIC participants did not  permit FNS to measure the extent to which WIC health referrals actually  contributed to the immunization of WIC child participants.
FNS officials noted that the problems they faced in finding a measure that  appropriately linked WIC health referral activities to a desired outcome are  not unique to the health referral services. They pointed out that it is  difficult to measure the impact of any of the individual services WIC offers  because, among other things, other variables also influence the behavior  these services are intended to promote. For example, there are several  other state and local programs that, like WIC, are aimed at improving  health through nutrition education. Separating the effects of these efforts  from those of the WIC program is difficult at best.
section_title: FNS Relies on a Variety of Output Measures to Monitor Program Performance
paragraphs: FNS employs a large number of program output measures to gauge  performance of the WIC nutritional services it offers. Generally, these  measures are used to examine the types and quantities of services the state  agencies provide and whether the agencies are in compliance with grant  expenditure and other program requirements. (A detailed list of output  measures can be found in app. I.) To monitor these measures, FNS depends  in large part on data provided by state and local agencies through routine  financial reports, its biennial Characteristics Study, and FNS’ management  evaluations of state and local agency operations. Of these monitoring  methods, FNS relies heavily on its management evaluations to measure  state and local WIC agency compliance with program requirements.  However, weaknesses in the procedures for conducting evaluations of WIC  agencies reduce its value as a performance oversight tool. FNS recently  completed an examination of its practices for conducting management  evaluations for all FNS programs, including WIC, and made several  recommendations for improvements.
FNS uses several financial reports submitted by state agencies to measure  the extent to which state and local agencies meet expenditure  requirements. The principal measure is whether states are adhering to a  legislatively established requirement for spending on nutrition education  activities and breastfeeding promotion and support activities. Combined  spending for these two activities must equal at least one-sixth of the NSA  expenditures plus a target amount for breastfeeding promotion and  support established by FNS at the beginning of each fiscal year. The  Addendum to the WIC Program Annual Closeout Report, prepared by each  state, serves as the primary source of data on the state’s compliance with  this expenditure requirement. For fiscal year 1999, the most recent year for  which complete data were available, FNS reported that two states and four  Indian tribal organizations did not meet this minimum spending  requirement. In addition, the WIC Program Annual Closeout Report each  state prepares provides data on the final status of the state’s grant and costs  for the report year.
To measure the nature and amount of program services WIC agencies  provide, FNS primarily uses data obtained from state and local agencies  and reported in FNS’ Characteristics Study. As noted earlier, the  Characteristics Study provides information on a variety of program  activities. Table 2 lists some of the key program activities in the three areas  of service included in the Characteristics Study as well as information on  the percentage of local WIC agencies offering the activity.
According to FNS officials, survey data from local agencies, of the sort seen  in table 2, may no longer be collected for the Characteristics Study for 2002  and beyond because research funds are unavailable for that purpose. These  officials explained that the Agriculture, Rural Development, Food and Drug  Administration, and Related Agencies Appropriations Act of 1998 (P.L. 105- 86) effectively transferred primary responsibility for research on food  assistance and nutrition programs from FNS to the Department’s Economic  Research Service, which has not subsequently funded the collection of the  local WIC agency data. FNS officials believe that these data are important  for monitoring the nature and level of services being delivered by local  agencies. To address this and other research concerns, FNS has attempted  to regain direct responsibility over research funding for food assistance  and nutrition programs.
To measure overall compliance with program requirements, including  those related to nutrition education, breastfeeding promotion and support,  and referral services, FNS regional offices conduct management  evaluations at each of the 88 state WIC agencies and at a sample of local  agencies. In general, each of the state agencies and selected local agencies  are reviewed every 3 years. According to FNS, management evaluations are  the primary means of measuring state and local WIC agency compliance  with program requirements. As part of the management evaluations, staff in  FNS’ regional offices conduct interviews, review planning documents,  observe interactions, and examine participants’ files. Listed below are  some of the types of nutrition education, breastfeeding promotion and  support, and health referral service program requirements that can be  assessed during management evaluations:   State agency WIC plan includes nutrition education goals and action   Local agencies offer each participant a minimum of two nutrition  education contacts during each 6-month certification period and  documents the provision of service or the participant’s refusal or  inability to participate,  State agency has developed or identified nutrition education  resources and materials for use by local agencies, and  Nutrition education material provided is easily understood by  participants.
State and local agencies have a designated breastfeeding promotion  and support coordinator who provides technical assistance to local  agencies,  State agencies have developed standards for local agencies for use in  providing breastfeeding promotion and support, and  Local agency staff has received training on promotion and  management of breastfeeding promotion and support.
Local agency staff refers participants, as appropriate, to private or  public health care services.
FNS has not made full use of management evaluations to measure program  performance and provide needed oversight. In large part this is due to  inconsistencies in the procedures used to conduct the evaluations, which  have caused problems in interpreting evaluation results as well as in  comparing and analyzing results across evaluations. FNS recently  examined its practices for conducting management evaluations for all FNS  programs, including WIC. Results of this review were highlighted in a  January 2001 report to the Administrator. The report noted that (1)  management evaluations at state and local levels were not being conducted  uniformly across FNS regional offices; (2) standard vocabulary did not  exist for identification of certain concepts; (3) standard definitions were  needed for such key terms as deficiency (a judgment that the agency being  reviewed is not in compliance with program requirements and must take  corrective action); (4) additional staff time needed to be allocated to  testing and training prior to implementation; and (5) a learning, feedback,  and modification period would be needed to support consistent data  collection and the development of reporting methods. The task force  recommended a number of actions aimed at obtaining more consistent  information that, if implemented, could provide FNS with better data for  identifying national trends, quantifying deficiencies, developing policies,  and measuring mission effectiveness. Agency officials told us that a  response from the Administrator is expected by April 2001.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to USDA’s Food and Nutrition Service for  review and comment. We met with FNS officials, including the Director,  Office of Analysis and Nutrition Evaluation; the Director, Supplemental  Food Program Division; and officials in the Grants Management Division.  These officials generally agreed with the information presented in this  report. However, they cautioned that the report addressed performance  measures as they pertained to three nutrition service components of the  WIC program and not the program in its entirety, which includes  supplemental foods and other aspects of nutrition services. FNS officials  believed that research on the effects of the WIC program as a whole  indicated that it has contributed to a number of desirable outcomes  including improved diets and infant feeding practices, reduced incidence of  anemia among children, reduced incidence of infant mortality and low  birthweight infants, and reduced postpartum medical expenses. Also, these  officials provided us with a number of technical comments that we  incorporated into the report, as appropriate.
section_title: Scope and Methodology
paragraphs: In developing information for this report, we spoke with and obtained data  from officials at FNS’ headquarters in Alexandria, Virginia, and the seven  FNS regional offices—Northeast in Boston, Massachusetts; Mid-Atlantic in  Robbinsville, New Jersey; Southeast in Atlanta, Georgia; Midwest in  Chicago, Illinois; Mountain Plains in Denver, Colorado; Southwest in  Dallas, Texas; and, Western in San Francisco, California. We reviewed  legislation, policies, procedures, and practices; reviewed reports issued by  GAO and others related to WIC; and interviewed various officials  responsible for administering the WIC program. We also reviewed FNS’  strategic and annual performance plans and other documents developed as  part of implementing the Results Act. We conducted our work from  September 2000 through January 2001 in accordance with generally  accepted government auditing standards.
We are sending copies of this report to the appropriate congressional  committees; interested members of the Congress; the Honorable Ann M.  Veneman, the Secretary of Agriculture; and other interested parties. We will  also make copies available to others upon request.
If you or your staff have any questions about this report, please contact me  at (202) 512-7215. Key contributors to this report are listed in appendix II.
section_title: Output Measures for Monitoring Program Requirements and Activities for the Three Services
paragraphs: Special report (biennial survey by public  health and nutrition program directors)
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Abiud Amaro, Clifford Diehl, Judy  Hoovler, Angela Miles, and Ron Wood made key contributions to this  report.
section_title: Ordering Information
paragraphs: The first copy of each GAO report is free. Additional copies of  reports are $2 each. A check or money order should be made out to  the Superintendent of Documents. VISA and MasterCard credit  cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are  discounted 25 percent.
Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders by phone: (202) 512-6000 fax: (202) 512-6061 TDD (202) 512-2537 Each day, GAO issues a list of newly available reports and  testimony. To receive facsimile copies of the daily list or any list  from the past 30 days, please call (202) 512-6000 using a touchtone  phone. A recorded menu will provide information on how to obtain  these lists.
section_title: To Report Fraud, Waste, or Abuse in Federal Programs
paragraphs: Web site: http://www.gao.gov/fraudnet/fraudnet.htm  e-mail: fraudnet@gao.gov  1-800-424-5454 (automated answering system)