Program Evaluation: OMB's PART Reviews Increased Agencies' Attention to Improving Evidence of Program Results
Published: 2005-10-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Office of Management and Budget (OMB) designed the Program Assessment Rating Tool (PART) as a diagnostic tool to draw on program performance and evaluation information for forming conclusions about program benefits and recommending adjustments to improve results. To assess progress in improving the evidence base for PART assessments, GAO was requested to examine (1) agencies' progress in responding to OMB's recommendations to evaluate programs, (2) factors facilitating or impeding agencies' progress, and (3) whether agencies' evaluations appear to be designed to yield the information on program results that OMB expects.
section_title: What GAO Found
paragraphs: GAO examined agency progress on 20 of the 40 evaluations OMB recommended in its PART reviews at four federal agencies: the Department of Energy, Department of Health and Human Services, Department of Labor, and Small Business Administration. About half the programs GAO reviewed had completed an evaluation in the 2 years since those PART reviews were published; 4 more were in progress and 3 were still being planned. Program restructuring canceled plans for the remaining 2 evaluations. Several agencies struggled to identify appropriate outcome measures and credible data sources before they could evaluate program effectiveness. Evaluation typically competed with other program activities for funds, so managers may be reluctant to reallocate funds to evaluation. Some agency officials thought that evaluations should be targeted to areas of policy significance or uncertainty. However, all four agencies indicated that the visibility of an OMB recommendation brought agency management attention--and sometimes funds--to get the evaluations done. Moreover, by coordinating their evaluation activities, agencies met these challenges by leveraging their evaluation expertise and strategically prioritizing their evaluation resources to the studies that they considered most important. Because the OMB recommendations were fairly general, agencies had flexibility in interpreting the kind of information OMB expected. Some program managers disagreed with OMB on the purpose of their evaluations, their quality, and the usefulness of "independent" evaluations by third parties unfamiliar with their programs. Agency officials concerned about an increased focus on process said that they were more interested in learning how to improve program results than in meeting an OMB checklist. Since a few programs did not discuss their evaluation plans with OMB, it is not certain whether OMB will find their ongoing evaluations useful during the programs' next PART review. GAO concludes that the PART review process stimulated agencies to increase their evaluation capacity and available information on program results. Further, agencies are likely to design evaluations to meet their own needs--that is, in-depth analyses that inform program improvement. If OMB wants evaluations with a broader scope, such as information that helps determine a program's relevance or value, it will need to take steps to shape both evaluation design and execution. Finally, because agency evaluation resources tend to be limited, they are most usefully focused on important areas of uncertainty. Regular performance reporting is key to good management, but requiring all federal programs to conduct frequent evaluation studies is likely to result in superficial reviews of little utility and to overwhelm agency evaluation capacity.

section_title: Letter
paragraphs: In the 1990s, Congress and the executive branch laid out a statutory and  management framework for strengthening government performance and  accountability. The Government Performance and Results Act of 1993  (GPRA) was its centerpiece. The act was designed to provide  congressional and executive decision makers with objective information  on the relative effectiveness and efficiency of federal programs and  spending. The current administration has made integrating performance  information into budget deliberations one of five governmentwide  management priorities under its President’s Management Agenda.
A central element of this initiative is the Program Assessment Rating Tool  (PART), designed by the Office of Management and Budget (OMB) to  provide a consistent approach to assessing federal programs in the  executive budget formulation process. PART is a standard series of  questions meant to serve as a diagnostic tool, drawing on available  program performance and evaluation information to form conclusions  about program benefits and recommend adjustments that may improve  results.
However, PART’s ability to do this relies on OMB’s access to credible  information on program performance and on policy makers’ confidence in  the credibility of their analysis. In our January 2004 review of PART, we  found that limited availability of credible evidence on program results  constrained the ability of OMB staff to use PART to rate programs’  effectiveness. When OMB first applied PART, for the fiscal year 2004  budget, it judged fully half the programs it reviewed as not having  adequate information on results. Moreover, although OMB’s assessments  recommended improvements in program design, management, and  assessment, half the recommendations were to improve program  assessment—to identify outcome measures and obtain improved  performance data or program evaluations.
To examine progress in improving the evidence base for the PART  assessments, you asked us to examine  1.  progress agencies have made in responding to OMB’s PART  recommendations that they obtain program evaluations,  2.  factors that facilitated or impeded agencies’ progress in obtaining  3.  whether the evaluations appear to have been designed to yield the  information on program results that OMB anticipated.
To answer these questions, we examined progress on 20 of the 40  evaluation recommendations in the President’s fiscal year 2004 budget  proposal. These 20 recommendations reflect a diverse array of programs  concentrated in the Department of Energy (DOE), the Department of  Health and Human Services (HHS), the Department of Labor (DOL), and  the Small Business Administration (SBA). We reviewed OMB and agency  documents and interviewed officials in the four agencies to learn the  status of the evaluations and the factors that influenced how they were  conducted. We also reviewed the available evaluation plans and reports to  assess whether they were likely to yield the desired information on results.  We conducted our review from December 2004 through August 2005 in  accordance with generally accepted government auditing standards. A list  of the programs reviewed and their evaluation recommendations appears  in appendix I. OMB provided written comments on a draft of this report  that are reprinted in appendix III.
subsections: section_title: Background
paragraphs: PART’s standard series of questions is designed to determine the strengths  and weaknesses of federal programs by drawing on available program  performance and evaluation information. OMB applies PART’s 25  questions to all programs under four broad topics: (1) program purpose  and design, (2) strategic planning, (3) program management, and (4)  program results (that is, whether a program is meeting its long-term and  annual goals). During the fiscal year 2004, 2005, and 2006 budget cycles,  OMB applied PART to approximately 20 percent of programs each year  and gave each program one of four overall ratings: “effective,” “moderately  effective,” “adequate,” or “ineffective,” depending on the program’s scores  on those questions. OMB gave a fifth rating of “results not demonstrated”  when it decided that a program’s performance information, performance  measures, or both were insufficient or inadequate.
The summary assessments published with the President’s annual budget  proposal include recommended improvements in program design,  management, and assessment. For example, a summary of the review’s  findings might be followed by the clause “the administration will conduct  an independent, comprehensive evaluation of the program,” or “the Budget  includes  to conduct independent and quality evaluations,” both of  which we interpreted as an OMB recommendation to the agency to  conduct such an evaluation. In our previous analysis of the fiscal year  2004 PART reviews, we analyzed over 600 recommendations made for the  234 programs assessed and found that half of those recommended  improvements in program assessment.
PART not only relies on previous program evaluation studies to answer  many of the questions but also explicitly asks, in the strategic planning  section, “Are independent evaluations of sufficient scope and quality  conducted on a regular basis or as needed to support program  improvements and evaluate effectiveness and relevance to the problem,  interest, or need?” Program evaluations are systematic studies that assess  how well a program is working, and they are individually tailored to  address the client’s research question. Process (or implementation)  evaluations assess the extent to which a program is operating as intended.  Outcome evaluations assess the extent to which a program is achieving its  outcome-oriented objectives; they focus on program outputs and  outcomes but may also examine program processes to understand how  outcomes are produced.
OMB first applied PART to the fiscal year 2004 budget during 2002, and the  assessments were published with the President’s budget in February 2003.  In January 2004, we reported on OMB and agency experiences with PART  in the fiscal year 2004 budget formulation process. We noted that PART  had helped structure OMB’s use of performance information in its budget  review and had stimulated agency interest in budget and performance  integration. However, its effectiveness as a credible, objective assessment  tool was challenged by inconsistency in OMB staff application of the  guidance and limited availability of credible information on program  results. Moreover, PART’s influence on agency and congressional decision  making was hindered by failing to recognize differences in focus and  issues of interest among the various parties involved in programmatic,  policy, and budget decisions. We noted that PART’s potential value lay in  recommended changes in program management and design but would  require sustained attention if the anticipated benefits were to be achieved.
To strengthen PART and its use, in our January 2004 report we  recommended that OMB (1) centrally monitor and report on agency  progress in implementing the PART recommendations; (2) improve PART  guidance on determining the unit of analysis, and defining program  outcomes and “independent, quality evaluation”; (3) clarify expectations  regarding agency allocation of scarce evaluation resources among  programs; (4) target future reviews based on the relative priorities, costs,  and risks associated with clusters of programs; (5) coordinate assessments  to facilitate comparisons and trade-offs between related program; (6)  consult with congressional committees on performance issues and  program areas for review; and (7) articulate an integrated, complementary  relationship between GPRA and PART.
Requesting that we follow up on the findings in our January 2004 report,  you asked that we examine (1) OMB and agency perspectives on the  effects of PART recommendations on agency operations and results, (2)  OMB’s efforts at ensuring an integrated relationship between PART and  GPRA, and (3) steps OMB has taken to involve Congress in the PART  process. A companion report addresses all three objectives—including  OMB’s outreach to Congress—with regard to all PART reviews. Because  of the fundamental role that the availability of program evaluations plays  in conducting PART assessments, we conducted an in-depth analysis of  agencies’ responses to OMB recommendations that they conduct program  evaluations. These recommendations were identified through the analysis  of recommendations for our January 2004 review. This report focuses on  agencies’ progress on those evaluations and the issues involved in  obtaining them. For both analyses, we examined the same four agencies’  experiences with PART. The four agencies were selected to represent a  range of program types (such as research and regulatory programs), large  and small agencies, and, for the purposes of this report, a large proportion  of the OMB evaluation recommendations.
section_title: About Half the Programs Completed Evaluations, and Three Evaluations Were Being Planned
paragraphs: All but two of the programs we reviewed had responded to some extent to  OMB’s recommendations to conduct an evaluation; agencies did not plan  evaluations of the other programs because they were canceled or  restructured. However, after 2 years, only about half the programs had  completed evaluations, partly because of lengthy study periods and partly  because of some lengthy planning phases. The evaluations used a variety  of study designs, reflecting differences in the programs and in the  questions posed about program performance.
subsections: section_title: All Programs Responded to OMB’s Recommendations, but Only Half Completed Evaluations
paragraphs: About half of the programs we reviewed (11 of the 20) had completed an  evaluation by June 2005—2 years after the fiscal year 2004 PART reviews  and recommendations were published. Four evaluations were in progress,  while 3 were still in the planning stage. Agencies did not plan an  evaluation of 2 programs because those programs had been canceled or  restructured. (See table 1.) Most of OMB’s evaluation recommendations  asked for evaluation of the specific program reviewed, while some PART  reviews at DOE and DOL asked the agencies to develop a plan for  conducting multiple evaluations. At DOL, where two entire regulatory  agencies had been assessed, these agencies had completed multiple  studies.
OMB gave DOE seven evaluation recommendations in its fiscal year 2004  PART reviews. Six were for research programs in basic science and  nuclear energy and one was for its formula grant program to weatherize  the homes of low-income families. Since one research program in the  Office of Science had previously been evaluated by a panel of external  experts called a committee of visitors, OMB explicitly recommended that  the other research programs in that office also institute such a process by  September 2003.
In response, DOE completed evaluations of five of the six research  programs, but it did not plan to evaluate the sixth, the Nuclear Energy  Research Initiative, because it considered this not a stand-alone program  but, rather, a source of funding for follow-up projects to other nuclear  energy research programs. DOE revised this program’s objective and now  authorizes funds for its projects through the other nuclear energy research  programs; thus it is no longer considered a separately funded program to  be evaluated. Finally, DOE officials indicated that they had only recently  gained funding for planning the evaluation of the Weatherization  Assistance program. (A bibliography of related agency evaluation reports  appears in app. II.)
OMB gave DOL five evaluation recommendations for fiscal year 2004. Two  were for evaluations of specific DOL programs: grants to state and local  agencies to provide employment-related training to low-income youths  and administration of the Federal Employees Compensation Act regarding  work-related injuries and illnesses. The three others were regulatory  enforcement offices or agencies of DOL that were reviewed in their  entirety: the Office of Federal Contract Compliance Programs, regarding  equal employment opportunity; the Employee Benefits Security  Administration; and the Occupational Safety and Health Administration  (OSHA). OMB recommended that the last, which is a large regulatory  agency, develop plans to evaluate the results of its regulatory and  nonregulatory programs.
The two DOL regulatory administrations each completed several  evaluations of their enforcement activities by spring 2005, as did two of  the three other DOL programs we reviewed. DOL is waiting to conduct an  evaluation of the fifth—the youth employment program—until after its  reauthorization because that is expected to result in an increased focus on  out-of-school youths and a significant change in program activities. In  addition, OSHA completed two regulatory “lookback” reviews—assessing  the cumulative effects of a regulation over time—one in 2004 and another  in 2005. Program officials indicated that they had developed a plan for  conducting lookback reviews of employee benefit regulations beginning in  fiscal year 2006.
OMB recommended evaluations for four diverse HHS programs: (1) grants  and technical assistance to states to increase childhood disease  immunization, (2) grants to states to help recently arrived refugees find  employment, (3) education loan repayment and scholarships for nurses in  return for serving in facilities facing a nursing shortage, and (4) direct  assistance in constructing sanitation facilities for homes for American  Indians and Alaskan Natives. Evaluations of the two state grant programs  were still in progress during our review, although an interim report on the  immunization program was available. Reports from the two other program  evaluations had recently been completed and were under departmental  review.
OMB recommended evaluations for four SBA programs: (1) support for  existing Business Information Centers that provide information and access  to technology for small businesses; (2) use of volunteer, experienced  business executives to provide basic business counseling and training to  current and prospective entrepreneurs; (3) Small Business Development  Centers that provide business and management technical assistance to  current and prospective entrepreneurs; and (4) the small business loan  program that provides financing for fixed assets. OMB also asked all three  counseling programs to develop outcome-oriented annual and long-term  goals and measures.
SBA is conducting customer surveys and had recently initiated a  comprehensive evaluation of one its counseling programs, and is planning  one for the other in fiscal year 2006. Another evaluation has begun to  compare the costs, benefits, and potential duplication of its business loan  programs. SBA planned no evaluation of the Business Information Centers  program because the program was canceled, partly as a result of the PART  review and an internal cost allocation study. In reassessing the need for  the program, SBA decided that because of the increase in commercially  available office supplies and services and the accessibility of personal  computers over the years, such a program no longer needed federal  government support.
section_title: Evaluation Design and Focus Differed, Reflecting Different Program Purposes and Structures
paragraphs: Because evaluations are designed around programs and what they aim to  achieve, the form of the evaluations reflected differences in program  structure and anticipated outcomes. The evaluations were typically  multipurpose, including questions about results as well as the agency  processes that managers control in order to achieve those results, and  designed to respond to OMB and yield actionable steps that programs  could take to improve results.
The Nursing Education Loan Repayment and Scholarship programs aim to  increase the recruitment and retention of professional nurses by providing  financial incentives in exchange for service in health care facilities that are  experiencing a critical shortage of nurses. The ongoing evaluation of the  two programs combined was shaped by the reporting requirements of the  Nurse Reinvestment Act of 2002. The act requires HHS to submit an  annual report to Congress on the administration and effect of the  programs. Each yearly report is to include information such as the number  of enrollees, scholarships, loan repayments and grant recipients,  graduates, and recipient demographics to provide a clear description of  program beneficiaries. Program beneficiaries are compared with the  student, nurse applicant and general populations to assess success in  outreach. Information pertaining to beneficiaries’ service in health care  facilities is important for determining whether program conditions and  program goals have been met. The number of defaulters, default rate,  amount of outstanding default funds, and reasons for default are reported  for each year. These data as well as follow-up data on whether  beneficiaries remain in targeted facilities after their term of commitment  will be important in assessing the overall cost-benefit of the program.  Subsequent data collection will establish trends and allow for a cost- benefit analysis in the future.
The Indian Health Service Sanitation Facilities Construction delivers  construction and related program services to provide drinking water and  waste disposal facilities for American Indian and Alaska Native homes, in  close partnership with tribes. Among other issues, the evaluation  examined key areas of service delivery, while the health benefits of clean  water were assumed. Specifically, project needs identification and project  portfolio management were evaluated to see how well construction efforts  are prioritized and targeted to areas of greatest need, and whether  facilities construction projects are competently designed, timely, and cost- effective. The completed evaluation recommended that the agency  consider integrating its separate data systems into a single portfolio  management system to represent all projects or, at least, to adopt  standardized project management and financial tracking systems.
The primary responsibility of DOL’s Office of Federal Contract  Compliance Programs is to implement and enforce rules banning  discrimination and establishing affirmative action requirements for federal  contractors and subcontractors. Because of the time and expense involved  in conducting compliance reviews and complaint investigations, the office  is attempting to target establishments for review based in part on an  analytic prediction that they will be found to discriminate. The focus of its  effectiveness evaluation, therefore, was on identifying a targeting  approach and measuring change in the rate of discrimination among  federal contractors during the period of oversight. The logic for this choice  of outcome measure was based on the expectation that overall rates of  discrimination would decrease if the oversight programs were effective.  Using data on the characteristics of establishments that had already been  reviewed, evaluators used statistical procedures to estimate a model of the  probability of discrimination. The coefficients from that model were then  used to predict rates of discrimination among contractors who had not  been reviewed and among noncontractors. The analysis showed that the  office effectively targeted selected establishments for review, but there  was no measurable effect on reducing employment discrimination in the  federal contractor workforce overall. To improve the office’s  effectiveness, the evaluators recommended that the office focus on  establishments with the highest predicted rates of discrimination rather  than employ its previous approach, targeting larger establishments that are  likely to affect a greater number of workers.
The DOE Office of Science used a peer review approach to evaluating its  basic research programs, adapting the committee of  visitors model that  the National Science Foundation had developed. Because it is difficult to  predict the findings of individual basic research projects, science  programs have adapted the peer review model they use for merit selection  of projects to evaluate their portfolios of completed (and ongoing)  research. The Office of Science convenes panels of independent experts as  external advisers to assess the agency’s processes for selecting and  managing projects, the balance in the portfolio of projects awarded, and  progress in advancing knowledge in the research area and in contributing  to agency goals. Panel reviews generally found these programs to be  valuable and reasonably well-managed and recommended various  management improvements such as standardizing and automating  documentation of the proposal review process, adopting program-level  strategic planning, and increasing staffing or travel funds to increase  grantee oversight.
OSHA, pursuant to section 610 of the Regulatory Flexibility Act and  section 5 of Executive Order 12866, must conduct lookback studies on  OSHA standards, considering public comments about rules, the continued  need for them, their economic impacts, complexity, and whether there is  overlap, duplicity, or conflict with other regulations. OSHA recently  concluded a lookback review on its Ethylene Oxide standard and issued a  final report on another lookback review that examined the Presence  Sensing Device Initiation standard for mechanical power presses.A press  equipped with a sensing device initiates a press cycle if it senses that the  danger zone is empty, and if something should enter the zone, the device  stops the press. Accidents with mechanical presses result in serious  injuries and amputations to workers every year.
In the sensing device lookback review, OSHA examined the continued  need for the rule, its complexity, complaints levied against the rule,  overlap or duplication with other rules, and the degree to which  technology, economic conditions, or other factors have changed in the  area affected by the rule. Typically, once a standard is selected for a  lookback review, the agency gathers information on experience with the  standard from persons affected by the rule and from the general public  through an announcement in the Federal Register. In addition, available  health, safety, economic, statistical, and feasibility data are reviewed, and  a determination is made about any contextual changes that warrant  consideration. In conducting such reviews, OSHA determines whether the  standards should be maintained without change, rescinded, or modified.  OSHA found that there was a continued need for the rule but that to  achieve the expected benefits of improved worker safety and employer  productivity, the rule needed to be changed. Although the technology for  sensing device systems had not changed since their adoption in 1988, the  technology for controlling mechanical presses had changed considerably,  with press operation now often controlled by computers, introducing  hazards that were not addressed initially by the standard.
section_title: Management Attention, Caught by OMB’s Recommendations, Overcame Measurement and Funding Barriers
paragraphs: Agency officials described two basic barriers to completing the  evaluations that OMB recommended: obtaining valid measures of program  outcomes to assess effectiveness and obtaining the financial resources to  conduct independent evaluations. Although most of the program officials  claimed that they had wanted to conduct such evaluations anyway, they  noted that the visibility of an OMB recommendation brought evaluation to  the attention of their senior management, and sometimes evaluation funds,  so that the evaluations got done. Indeed, in response to the PART reviews  and recommendations, two of the agencies initiated strong, centrally led  efforts to build their evaluation capacity and prioritize evaluation  spending.
subsections: section_title: Measurement Challenges Delayed Evaluation Starts
paragraphs: To evaluate program effectiveness, agencies needed to identify  appropriate measures of the outcomes they intended to achieve and  credible data sources for those measures. However, as noted in our  previous report, many programs lacked these and needed to develop new  outcome-oriented performance measures in order to conduct evaluations.
Agency officials identified a variety of conceptual and technical barriers to  measuring program outcomes similar to those previously reported as  difficulties in implementing performance reporting under GPRA. SBA  officials acknowledged that before the PART reviews, they generally  defined their programs’ performance in terms of outputs, such as number  of clients counseled, rather than in outcomes, such as gains in small  business revenue or employment. SBA revised its strategic plan in fall 2003  and worked with its program partners to develop common definitions  across its counseling programs, such as who is the client or what  constitutes a counseling session or training. Since SBA had also had  limited experience with program evaluation, it contracted for assistance in  designing evaluations of the economic impact of its programs.
DOL had difficulty conceptualizing the outcomes of regulations in  monetary terms to produce the cost-benefit analyses that PART (and the  Regulatory Flexibility Act) asks of regulatory programs. For instance,  OSHA has historically considered the likely controversy of quantifying the  value of a human life in calculating cost-benefit ratios for developing  worker health and safety regulations. OSHA officials explained that the  Assistant Secretary had helped to mitigate such a controversy by issuing a  July 2003 memorandum that directed OSHA staff to identify costs,  benefits, net benefits, and the impact of economically significant  regulations and their significant alternatives, as well as discuss significant  nonmonetized costs and benefits.
DOL officials noted that designing a cumulative assessment of the net  benefits of employer reporting requirements for pension and health benefit  plans was complicated. For example, a primary benefit of reporting is to  aid the agency’s ability to enforce other benefit plan rules and thereby  protect or regain employees’ benefits. They also pointed out that although  health and safety regulations are mandatory, employers are not required to  offer benefit plans, so a potential cost of regulators’ overreaching in their  enforcement actions could be discouraging employers from offering these  pension and health benefits altogether.
DOE officials acknowledged that they could not continue to use state  evaluations to update the national estimates of energy savings from a  comprehensive evaluation of weatherization assistance conducted a  decade ago. They recognized that assumptions from the original national  evaluation could no longer be supported and that a new, comprehensive  national evaluation design was needed. They noted new hurdles to  measuring reductions in home heating costs since the previous evaluation:  (1) monthly electric bills typically do not isolate how much is spent on  heating compared with other needs, such as lighting, and (2) the increased  privatization of the utility industry is expected to reduce government  access to the utilities’ data on individual household energy use.
Other barriers were more operational, such as the features of a program’s  data system that precluded drawing the desired evaluative conclusions.  For one, regulations need to be in place for a period of years to provide  data adequate for seeing effects. HHS officials noted that their databases  did not include the patient outcome measures OMB asked for and that  they would need to purchase a longitudinal study to capture those data.  They also noted that variation in the form of states’ refugee assistance  programs and data systems, as well as regional variation in refugees’  needs, made it difficult to conduct a national evaluation. Their evaluation  especially relied on the cooperation of state program coordinators. DOL  officials pointed out that the federal employees’ compensation program’s  data system was developed for employee and management needs and did  not lend itself to making comparisons with the very different state  employee compensation programs.
section_title: Agencies, with Limited Funds, Delayed or Narrowed Evaluations and Questioned the Need to Evaluate All Programs
paragraphs: Evaluation generally competes for resources with other program and  department activities. Contracts for external program evaluations that  collect and analyze new data can be expensive. In a time of tight  resources, program managers may be unwilling to reallocate resources to  evaluation. Agencies responded to such limitations by delaying evaluations  or cutting back on an evaluation’s scope. Some agency officials thought  that evaluations should not be conducted for all programs but should be  targeted instead to areas of uncertainty.
HHS’s Office of Refugee Resettlement—which was allotted funds  especially for its evaluation—is spending $2 million to evaluate its refugee  assistance program over 2 years. Costs are driven primarily by the  collection of data through surveys, interviews, and focus groups and the  need for interpreters for many different languages. Given the size and  scope of the program, even with $2 million, program officials would have  liked to have more time and money to increase the coverage of their  national program beyond the three sites they had selected.
DOL program officials explained that although they had had a large  program evaluation organization two decades ago, the agency downsized  in 1991, the office was eliminated, and now they must search for program  evaluation dollars. The program spent $400,000 for an 18-month evaluation  of the Federal Employees Compensation Act program, which relied  heavily on program administrative data, but they also spent a large amount  of staff time educating and monitoring the contractor. Program officials  were disappointed with the lack of depth in the evaluation. They believed  that their evaluation contractor did not have enough time to plan and  conduct a systematic survey, and consequently, their selective interview  data were less useful than they would have liked.
DOE program officials indicated that they have been discussing an  evaluation of Weatherization Assistance since spring 2003, but not having  identified funds for an evaluation, they have not been able to develop a  formal evaluation plan. They had no budget line item for evaluation, so  they requested one in their fiscal year 2005 appropriations. Although there  was congressional interest in an evaluation, additional funds were not  provided in fiscal year 2005. DOE instructed program officials to draw  money for evaluation from the 10 percent of the program’s funds that are  set aside for training and technical assistance, increase the federal share  from 1.5 percent to 2 percent, and reduce the states’ share to 8 percent.  Program officials indicated that the amount from the technical assistance  account would cover only planning and initial implementation activities,  not the bulk of the evaluation itself. And they were concerned about  displacing existing training, so they were still looking for an evaluation  funding commitment.
Agency officials also questioned PART’s assumption that all programs  should have evaluations. SBA officials indicated that some agency  appropriations generally precluded SBA’s spending program funds on any  but specifically identified program activities. Thus, evaluations had to be  funded from agency administrative funds. They thought that it was  unreasonable to ask a small agency to finance several program  evaluations, as might be expected of a larger agency. SBA dealt with this  by conducting evaluations sequentially as funds became available. DOL  program officials also thought that spending several hundred thousand  dollars for a comprehensive evaluation study was a reasonable investment  for a $2.5 billion program but not for small programs. They did not believe  that all programs need to be evaluated—especially in a time of budget  deficits. They recommended that OMB and agencies should “pick their  shots” and should be more focused in choosing evaluations to conduct.  They suggested a risk-based approach, giving higher priority to evaluating  programs for which costs are substantial and effectiveness uncertain.
section_title: OMB’s Recommendations Increased Management Attention and Investment in Evaluation
paragraphs: Most of the agency officials we interviewed declared that they valued  evaluation. For example, HHS and DOE officials described evaluation as  part of their culture. Many said they had already been planning to do  something similar to the evaluation that OMB had recommended. In a  couple of cases, OMB’s recommendation appeared to have been shaped by  planned or ongoing activities. However, officials in all four agencies  indicated that the visibility of a PART recommendation and associated  OMB pressure brought management attention, and sometimes funds, to  getting the evaluations done.
HHS departmental officials said that the agency was a federal leader in  terms of evaluation capacity, and that they spend approximately $2.6  billion a year on agency-initiated research, demonstrations, and  evaluation. They stated that it is part of their culture to conduct  evaluations—because their program portfolio is based in the physical and  social sciences. DOE officials said that they embraced the PART process  because, as an agency with a significant investment in advancing science  and technology, DOE had already been using similar processes, such as  peer review, to evaluate its programs. DOE officials noted that DOE had  developed a basic evaluation mechanism—independent peer review—that  all its research programs undertake. Officials in the Office of Energy  Efficiency and Renewable Energy developed a corporate peer review  guide summarizing best practices in this field and considered their peer  review process as “state of the art,” as it is used as a model nationally and  globally.
In other cases, agency or congressional interest in evaluation seemed to  set the stage for OMB evaluation recommendations. For example, while  OMB was reviewing the Nursing Education Loan Repayment program, the  Nursing Reinvestment Act of 2002 was enacted, expanding the program  and instituting a requirement for annual reports after the first 18 months.  The reports were to include data on the numbers of loan applicants and  enrollees, the types of facilities they served in, and the default rates on  their loans and service commitments and an evaluation of the program’s  overall costs and benefits. OMB then recommended that the agency  evaluate the program’s impact, develop outcome measures, and begin to  track performance against newly adopted benchmarks. To respond to  OMB’s request for a long-term outcome measure, the agency agreed to also  collect information on how long beyond their service commitment nurses  stay in service in critical shortage facilities. In another example previously  discussed, the DOE Office of Science had already initiated committee of  visitors reviews for its Basic Energy Sciences program, which OMB then  recommended for other research programs in that office.
The PART and President’s Management Agenda pressed agencies to report  progress on the recommendations. OMB published the cumulative set of  completed PART review summaries, including the recommendations, in  the President’s budget proposals for fiscal years 2004 through 2006. In the  fiscal year 2006 budget, OMB reported on the status of its previous  recommendations in the PART summaries, whether action had been taken  or completed. OMB also asked agencies to report on their progress in  implementing PART recommendations to provide input into its quarterly  scorecards on agencies’ progress in implementing the President’s  Management Agenda initiatives. In addition, OMB precluded agencies from  being scored “green” on Budget and Performance Integration if more than  10 percent of their programs were rated “results not demonstrated” 2 years  in a row. DOE and DOL program officials reported being asked to update  the status of the recommendations every 2 to 3 months. HHS officials  noted that since fall 2004, they have been reporting on PART  recommendations to OMB twice a year, tracking approximately 100 PART  recommendations (with about 200 separate milestones) for the 62  programs reviewed for fiscal years 2004 through 2006.
Most of the officials we interviewed believed that because of PART and  the President’s Management Agenda, their agencies were paying greater  attention to program results and evaluation. Officials at DOL noted that  the department spends much time and effort making sure it scores green  on the next President’s Management Agenda assessment; for example, the  department’s management review board, chaired by Labor’s Assistant  Secretary for Management and Administration, discusses these issues  monthly. In addition, DOL’s Center for Program Planning and Results  reviews programs’ progress on OMB’s recommendations, scores programs  internally on the Budget and Performance Integration scorecard, and  provides agencies with training and preparation before their PART  reviews. The SBA Administrator initiated a series of steps after August  2003 to increase the agency’s focus on achieving results. SBA rewrote its  strategic plan to focus on a limited number of strategic goals and  integrated its strategic plan, annual performance plan, and performance  report. The agency formed a central Office of Analysis, Planning, and  Accountability to help each program office develop results-oriented  performance measures and conduct program assessments.
Although HHS officials said that the department had invested in evaluation  long before the PART reviews, Indian Health Service program officials  indicated that they had not planned an evaluation of their sanitation  facilities program before the PART review. However, they thought it was a  good idea and said that the recommendation brought their lack of a recent  evaluation to HHS’s attention, making it easier to justify efforts to quantify  their program’s benefits.
section_title: Centralized Coordination Helped Agencies Leverage Their Evaluation Resources
paragraphs: SBA and DOL responded to demands for more performance information  by centrally coordinating their assessment activities, helping to address  evaluation’s measurement and funding challenges. Centralization helped  the agencies to leverage their evaluation expertise throughout the agency  and helped them prioritize spending on the evaluations they considered  most important.
SBA program offices had little experience with outcome measurement and  evaluation before the 2002 PART reviews. The central planning office was  formed to help the program offices develop outcome measures linked to  the agency’s strategic goals and collect and validate their performance  data. The office also conducts an annual staff activity survey to support  cost allocation across programs, a key step toward performance  budgeting. This office took advantage of the similarity in outcome goals  across SBA’s programs and the evaluation methodology developed for the  counseling programs to contract for the development of a standard  methodology for assessing other SBA programs’ economic impacts on  small businesses. The central office is also funding the subsequent  evaluations. For a small agency, this type of coordination can result in  important savings in contract resources as well as staff time.
DOL, much larger than SBA, has measurement and evaluation experience,  but capacity had declined over time. DOL established the Center for  Program Planning and Results in 2001 to provide leadership, policy advice,  and technical assistance to GPRA-related strategic and performance  planning. The center was expanded in fiscal year 2003 to respond to the  President’s Management Agenda and manage the PART process. With a  budget of $5 million a year, the center solicits and selects evaluation  proposals focusing on program effectiveness submitted by DOL’s  component agencies, funds the studies, and helps oversee the external  contractors. The center’s officials claimed that the Secretary’s and  Assistant Secretary’s support for evaluation, combined with pressure from  OMB, has led to increased interest by the component agencies in  evaluation, resulting in $6 million to $7 million in proposals competing for  $5 million in evaluation funds. Some DOL agencies retained their  evaluation expertise and design, fund, and oversee their own evaluations.  In addition to helping program offices develop research questions and  evaluation designs, the center helps develop agency evaluation capacity by  holding “Vendor Days,” when evaluation contractors are invited to exhibit  for agency staff the specialized design, data collection, and analysis skills  that could inform future studies.
section_title: Where OMB and Program Managers Do Not Share Expectations, Evaluations May Not Meet OMB Needs
paragraphs: Because the OMB evaluation recommendations were fairly general,  agencies had flexibility in interpreting the information OMB expected and  the evaluations to fund. Some program managers disagreed with OMB on  the scope and purpose of their evaluations, their quality, and the  usefulness of evaluations conducted by independent third parties.  Program managers concerned about an increased focus on process said  that they were more interested in learning how to improve program  performance than in meeting an OMB checklist. Since a few programs did  not discuss their evaluation plans with OMB, it is not certain whether OMB  will accept their ongoing evaluations.
subsections: section_title: Agencies Have Flexibility in Determining Evaluation Timing and Content
paragraphs: Agencies had a fair amount of flexibility to design their evaluations.  Except for the recommendations to the DOE Office of Science to conduct  committee of visitors reviews, OMB’s evaluation recommendations were  fairly general, typically telling agencies to conduct an independent  evaluation of a program’s effectiveness. Agencies reported little guidance  from OMB on how to conduct these evaluations, beyond the PART written  guidance and the rationale the examiner provided for not accepting their  previous evaluations or measures of program outcomes. They said that  follow-up on previous PART recommendations was generally limited to  providing responses to the OMB reporting template, unless OMB  conducted a second formal PART review.
Agencies also had flexibility to determine the timing of their evaluations.  Agency officials reported that OMB did not prioritize its recommendations  within or among programs. Moreover, because evaluation resources were  limited, DOL and SBA officials reported that they had to choose which  evaluations to conduct first. The recommendations for the two DOL  regulatory agencies explicitly acknowledged their need to balance  responsibility for several programs. OMB asked these agencies to develop  plans to evaluate their programs or expand existing efforts for more  comprehensive and regular evaluation. In the reviews of recommendation  status for the fiscal year 2006 budget, OMB credited both agencies with  having conducted one or more program reviews and planning others.  Agencies were free to choose which programs to evaluate but were likely  to be influenced by the potential effect of PART reassessments on their  President’s Management Agenda scores and, thus, to attempt to reduce the  number of programs rated “results not demonstrated.” Research and  development programs were held to a somewhat higher standard than  other programs were, since their agencies could not be scored “green” on  the separate R&D Investment Criteria Initiative if less than 75 percent of  their programs received a score of “moderately effective” or better. DOE  officials noted that their Office of Energy Efficiency and Renewable  Energy now requires programs to outline their plans for evaluations in  their multiyear plans.
section_title: Disagreements on the Scope and Purpose of Evaluation
paragraphs: OMB and the agencies significantly differed in defining evaluation scope  and purpose. Program officials were frustrated by OMB’s not accepting  their prior evaluations of program effectiveness in the PART review. Some  of the difficulties seemed to derive from OMB expecting to find, in the  agencies’ external evaluation studies, comprehensive judgments about  program design, management, and effectiveness, like the judgments made  in the OMB PART assessments.
PART’s criteria for judging the adequacy of agency evaluations are  complex and may have created some tension as to the importance of one  dimension over another. For example, question 2.6 read: “Are independent  evaluations of sufficient scope and quality conducted on a regular basis or  as needed to support program improvements and evaluate effectiveness  and relevance to the problem, interest, or need?” OMB changed the  wording of the question to help clarify its meaning and added the  reference to “relevance.” However, while OMB’s revised guidance for this  question defines quality, scope, and independence, it does not address the  assessment of program “relevance.”  Specifically, sufficient scope is  defined as whether the evaluation focuses on achievement of performance  targets and the cause and effect relationship between the program and  target—i.e., program effectiveness. This is different from assessing the  relevance—i.e., appropriateness—of the program design to the problem or  need. Instead, questions in section 1 ask whether the design is free of  major flaws and effectively targeted to its purpose.
Another potential contribution to differences between OMB and agency  expectations for program evaluations is that evaluations designed for  internal audiences often have a different focus than evaluations designed  for external audiences. Evaluations that agencies initiate typically aim to  identify how to improve the allocation of program resources or the  effectiveness of program activities. Studies requested by program  authorizing or oversight bodies are more likely to address external  accountability—to judge whether the program is properly designed or is  solving an important problem.
HHS officials reported differences with OMB over the acceptability of HHS  evaluations. HHS officials were particularly concerned that OMB  sometimes disregarded their studies and focused exclusively on OMB’s  own assessments. One program official complained that OMB staff did not  adequately explain why the program’s survey of refugees’ economic  adjustment did not qualify as an “independent, quality evaluation,”  although an experienced, independent contractor conducted the  interviews and analysis. In the published PART review, OMB  acknowledged that the program surveyed refugees to measure outcomes  and monitored grantees on-site to identify strategies for improving  performance. In our subsequent interview, OMB staff explained that the  outcome data did not show the mechanism by which the program  achieved these outcomes, and grantee monitoring did not substitute for  obtaining an external evaluation, or judgment, of the program’s  effectiveness. Other HHS officials said that OMB had been consistent in  applying the standards for independent evaluation, but these standards  were set extremely high.
In reviewing the vaccination program, OMB did not accept the several  research and evaluation studies offered, since they did not meet all key  dimensions of “scope.” OMB acknowledged that the program had  conducted several management evaluations to see whether the program  could be improved but found their coverage narrow and concluded “there  have previously been no comprehensive evaluations looking at how well  the program is structured/managed to achieve its overall goals.” OMB also  did not accept an external Institute of Medicine evaluation of how the  government could improve its ability to increase immunization rates  because the evaluation report had not looked at the effectiveness of the  individual federal vaccine programs or how this program complemented  the other related programs. However, in reviewing recommendation  status, OMB credited the program with having contracted for a  comprehensive evaluation that was focused on the operations,  management, and structure of this specific vaccine program.
DOE Office of Science officials described much discussion with OMB  examiners about what was or was not a good committee of visitors review  in following up on the status of the evaluation recommendations. Although  OMB had revised and extended its guidance on what constituted quality in  evaluation, program officials still found this guidance difficult to apply to  research programs. They also acknowledged that their first committee of  visitors reviews might have been more useful to the program than to OMB.
section_title: Disagreements about the Quality of Evaluation Designs
paragraphs: OMB and agencies differed in identifying which evaluation methods were  sufficiently rigorous to provide high-quality information on program  effectiveness. OMB guidance encouraged the use of randomized controlled  trials, or experiments, to obtain the most rigorous evidence of program  impact but also acknowledged that these studies are not suitable or  feasible for every program. However, as described above, without  guidance on which—and when—alternative methods were appropriate,  OMB and agency staff disagreed on whether specific evaluations were of  acceptable quality. To help develop shared understandings and  expectations, federal evaluation officials and OMB staff held several  discussions on how to assess evaluation quality according to the type of  program being evaluated.
When external factors such as economic or environmental conditions are  known to influence a program’s outcomes, an impact evaluation attempts  to measure the program’s net effect by comparing outcomes with an  estimate of what would have occurred in the absence of the program  intervention. A number of methodologies are available to estimate  program impact, including experimental and quasi-experimental designs.  Experimental designs compare the outcomes for groups that were  randomly assigned to either the program or to a nonparticipating control  group prior to the intervention. The difference in these groups’ outcomes  is believed to represent the program’s impact, assuming that random  assignment has controlled for any other systematic difference between the  groups that could account for any observed difference in outcomes. Quasi- experimental designs compare outcomes for program participants with  those of a comparison group not formed through random assignment, or  with participants’ experience prior to the program. Systematic selection of  matching cases or statistical analysis is used to eliminate any key  differences in characteristics or experiences between the groups that  might plausibly account for a difference in outcomes.
Randomized experiments are best suited to studying programs that are  clearly defined interventions that can be standardized and controlled, and  limited in availability, and where random assignment of participants and  nonparticipants is deemed feasible and ethical. Quasi-experimental  designs are also best suited to clearly defined, standardized interventions  with limited availability, and where one can measure, and thus control for,  key plausible alternative explanations for observed outcomes. In mature  full-coverage programs where comparison groups cannot be obtained,  program effects may be estimated through systematic observation of  targeted measures under specially selected conditions designed to  eliminate plausible alternative explanations for observed outcomes.
Following our January 2004 report recommendation that OMB better  define an “independent, quality evaluation,” OMB revised and expanded its  guidance on evaluation quality for the fiscal year 2006 PART reviews. The  guidance encouraged the use of randomized controlled trials as  particularly well suited to measuring program impacts but acknowledged  that such studies are not suitable or feasible for every program, so it  recommended that a variety of methods be considered. OMB also formed  an Interagency Program Evaluation Working Group in the summer of 2004  to provide assistance on evaluation methods and resources to agencies  undergoing a PART review that discussed this guidance extensively.  Evaluation officials from several federal agencies expressed concern that  the OMB guidance materials defined the range of rigorous evaluation  designs too narrowly. In the spring of 2005, representatives from several  federal agencies participated in presentations about program evaluation  purposes and methods with OMB examiners. They outlined the types of  evaluation approaches they considered best suited for various program  types and questions (see table 2). However, OMB did not substantively  revise its guidance on evaluation quality for the fiscal year 2007 reviews  beyond recommending that “agencies and OMB should consult evaluation  experts, in-house and/or external, as appropriate, when choosing or  vetting rigorous evaluations.”
section_title: Disagreements about Requiring Independent Third-Party Evaluations
paragraphs: A related source of tension between OMB and agency evaluation interests  was the importance of an evaluation’s independence. PART guidance  stressed that for evaluations to be independent, nonbiased parties with no  conflict of interest, for example, GAO or an Inspector General, should  conduct them. OMB subsequently revised the guidance to allow  evaluations to be considered independent if the program contracted them  out to a third party or they were carried out by an agency’s program  evaluation office. However, disagreements continued on the value and  importance of this criterion.
HHS officials reported variation among examiners in whether their  evaluations were considered independent. Two programs objected to OMB  examiners’ claims that an evaluation was not independent if the agency  paid for it. OMB changed the fiscal year 2005 PART guidance to recognize  evaluations contracted out to third parties and agency program evaluation  offices as possibly being sufficiently independent, subject to examination  case by case. But HHS officials claimed that they were still having issues  with the independence standard in the fiscal year 2006 reviews and that  OMB’s guidance was not consistently followed from one examiner to the  next.
DOL program officials stated that using an external evaluator who was not  familiar with the program resulted in an evaluation that was not very  useful to them. In part, this was because program staff were burdened  with educating the evaluator. But more important, they claimed that the  contractor designed the scope of the work to the broad questions of PART  (such as questions on program mission) rather than focusing on the results  questions the program officials wanted information on. In combination,  this led to a relatively superficial program review, in their view, that  provided the external, independent review OMB wanted but not the  insights the program managers wanted.
In reviewing the status of its PART recommendations, OMB did not accept  advisory committee reviews for two research programs that DOE offered  in response because OMB did not perceive the reviews as sufficiently  independent. These two program reviews involved standing advisory  committees of approximately 50 people who review the programs every 3  years. The OMB examiner believed that the committee was not truly  independent of the agency. DOE program officials objected, noting the  committee’s strong criticisms of the program, but have reluctantly agreed  to plan for an external review by the National Academies. Program  officials expressed concern that because evaluators from the National  Academies may not be sufficiently familiar with their program and its  context, such reviews may not address questions of interest to them about  program performance.
HHS program officials were also concerned about the usefulness of an  evaluation of the sanitation facilities program if it was conducted by a  university-based team inexperienced with the program. The agency  deliberately guarded against this potential weakness by including two ex- agency officials (one an engineer) on the evaluation team, and by taking  considerable effort with the team to define the evaluation questions.
section_title: Agencies Not Consulting with OMB on Evaluation Plans May Not Meet OMB’s Expectations
paragraphs: Agencies’ freedom to design their evaluations, combined with differences  in expectations between agencies and OMB, raises the strong possibility  that the evaluations that agencies conduct may not provide OMB with the  information it wants. Most of the agency officials we interviewed said that  they had discussed their evaluation plans with their OMB examiners, often  as part of their data collection review process. SBA and DOL, in particular,  appeared to have had extensive discussions with their OMB examiners.  However, a few programs have not discussed their plans with OMB,  presumably on the assumption that they will meet OMB’s requirements by  following its written guidance.
Officials in SBA’s and DOL’s central planning offices described extensive  discussions of their evaluation plans with their OMB examiners. SBA  vetted the evaluation design for SBA’s counseling programs with OMB in  advance, as well as the questionnaire used to assess client needs. DOL  planning and evaluation officials noted that they had worked with OMB  examiners to moderate their expectations for agencies’ evaluations. They  said that OMB understands their “real world” financial constraints and is  allowing them to “chip away” at their outcome measurement issues and  not conduct net impact evaluations in program areas where they do not  have adequate funds to do this type of evaluation.
HHS program officials were concerned about whether OMB will accept  their ongoing evaluation of the immunization program when they receive  their next PART review. The evaluation recommendation was general, so  they based their design on the fiscal year 2004 criteria and to provide  information useful to the program. However, the officials had heard that  the fiscal year 2007 evaluation quality criteria were more rigid than those  previously used, so they were concerned about whether the program will  meet OMB’s evaluation criteria when it is reviewed again. They said they  would have liked OMB to consider its evaluation progress and findings so  far and to have given them input as to whether the evaluation will meet the  current criteria. OMB officials denied that the PART criteria for evaluation  quality had changed much in the past two years. They also expected, from  their review of the design, that this new evaluation would meet current  PART criteria, assuming it was carried out as planned.
Several program officials expressed the view that in designing their  evaluations, they were more concerned with learning how to improve their  programs than in meeting an OMB checklist. Program officials complained  that OMB’s follow-up on whether evaluations were being planned sent the  message that OMB was more interested in checking off boxes than in  having a serious discussion about achieving results. When one program  official was asked for the program’s new evaluation plan, he answered  “Who needs a plan? I’ve got an evaluation.” DOE program officials  indicated that they believe a comprehensive evaluation of Weatherization  Assistance should include all the questions that state, regional, and local  officials would like to ask and not just establish a new national energy  savings estimate. Those questions—also of interest to DOE—include:  Which weatherization treatments correlate with energy savings? Should  they use their own crews or hire contractors? What are the nonenergy  benefits, such as improved air quality or employment impacts? Program  officials indicated that they had conducted a great deal of planning and  discussion with their stakeholders over the past 5 to 6 months and expect  to conduct five or six studies to meet those needs.
section_title: Conclusions
paragraphs: The PART review process has stimulated agencies to increase their  evaluation capacity and available information on program results. The  systematic examination of the array of evidence available on program  performance has helped illuminate gaps and has helped focus evaluation  questions. The public visibility of the results of the PART reviews has  brought management attention to the development of agency evaluation  capacity.
Evaluations are useful to specific decision makers to the degree that the  evaluations are credible and address their information needs. Agencies are  likely to design evaluations to meet their own needs—that is, in-depth  analyses that inform program improvement. If OMB wants evaluations  with a broader scope, such as information that helps determine a  program’s relevance or value, it will need to take steps to shape both  evaluation design and execution.
Because agency evaluation resources tend to be limited, they are most  usefully focused on illuminating important areas of uncertainty. While  regular performance reporting is key to good program management and  oversight, requiring all federal programs to conduct frequent evaluation  studies is likely to result in many superficial reviews that will have little  utility and that will overwhelm agency evaluation capacity.
section_title: Recommendations for Executive Action
paragraphs: In light of our findings and conclusions in this report, we are making the  following recommendations to OMB reiterating and expanding on  recommendations in our previous report:  OMB should encourage agencies to discuss their plans for program  evaluations—especially those in response to an OMB recommendation— with OMB and with congressional and other program stakeholders to  ensure that their findings will be timely, relevant, and credible and that  they will be used to inform policy and management decisions.
OMB should engage in dialogue with agencies and congressional  stakeholders on a risk-based allocation of scarce evaluation resources  among programs, based on size, importance, or uncertain effectiveness,  and on the timing of such evaluations.
OMB should continue to improve its PART guidance and training of  examiners on evaluation to acknowledge a wide range of appropriate  methods.
section_title: Agency Comments
paragraphs: We provided a draft of this report to OMB and the agencies for review and  comment. OMB agreed that evaluation methodology should be appropriate  to the size and nature of the program and that randomized controlled trials  may not be valuable in all settings. It noted its intent to provide additional  guidance in this area. OMB disagreed with the reference to the PART as a  checklist. This view was not ours but the view of agency officials who  expressed concern about the focus of the assessment process. OMB also  provided a number of technical comments, which we incorporated as  appropriate throughout the report. OMB’s comments appear in appendix  III. We also received technical comments from DOE, DOL, and HHS that  we incorporated where appropriate throughout the report. SBA had no  comments.
We are sending copies of this report to the Director of the Office of  Management and Budget; the Secretaries of Energy, Labor, and Health and  Human Services; the Administrator of the Small Business Administration;  appropriate congressional committees; and other interested members of  Congress. We will also make copies available to others on request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have questions about this report, please contact me at  (202) 512-2700 or KingsburyN@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix IV.
section_title: Appendix I: Agency Programs OMB Recommended Evaluations For in PART Reviews
subsections: section_title: Advanced Fuel Cycle Initiative
section_title: Appendix II: Related Agency Program Evaluation Reports
subsections: section_title: Department of Energy Agency Reports
paragraphs: Advanced Fuel Cycle Initiative: Nuclear Energy Research Advisory  Committee (NERAC) Evaluation Subcommittee. Evaluation of DOE  Nuclear Energy Programs. Washington, D.C.: Sept. 10, 2004.
Advanced Scientific Computing Research Program: Advanced  Scientific Computing Research. Committee of Visitors Report.  Washington, D.C.: April 2004.
Generation IV Nuclear Energy Systems Initiative: Nuclear Energy  Research Advisory Committee (NERAC) Evaluation Subcommittee.  Evaluation of DOE Nuclear Energy Programs. Washington, D.C.: Sept. 10,  2004.
High Energy Physics Program: Committee of Visitors to the Office of  High Energy Physics. Report to the High Energy Physics Advisory Panel.  Washington, D.C.: Apr. 7, 2004.
Nuclear Physics Program: Committee of Visitors. Report to the Nuclear  Science Advisory Committee. Washington, D.C.: Department of Energy,  Office of Science, Feb. 27, 2004.
section_title: Department of Health and Human Services Agency Reports
paragraphs: 317 Immunization Program: RTI International. Section 317 Grant  Immunization Program Evaluation: Findings from Phase I. Draft  progress report. Atlanta, Ga.: Centers for Disease Control and Prevention,  January 2005.
Indian Health Service Sanitation Facilities Program: Department of  Health and Human Services, U.S. Public Health Service, Federal  Occupational Health Service. Independent Evaluation Report Summary.  Prepared for Indian Health Service Sanitation Facilities Construction  Program, Rockville, Maryland. Seattle, Wash.: Mar. 8, 2005.
Nursing Education Loan Repayment and Scholarship Program:  Department of Health and Human Services, Health Resources and Services  Administration, Bureau of Health Professions. HRSA Responds to the  Nursing Shortage: Results from the 2003 Nursing Scholarship Program  and the Nursing Education Loan Repayment Program: 2002–2003. First  report to the United States Congress. Rockville, Md.: n.d.
section_title: Department of Labor Agency Reports
paragraphs: Employee Benefits Security Administration Reports:    Mathematica Policy Research, Inc. Case Opening and Results Analysis  (CORA) Fiscal Year 2002: Final Report. Washington, D.C.: Mar. 31,  2004.
Royal, Dawn. U.S. Department of Labor, Employee Benefits Security  Administration: Evaluation of EBSA Customer Service Programs  Participant Assistance Program Customer Evaluation. Washington,  D.C.: The Gallup Organization, February 2004.
Royal, Dawn. U.S. Department of Labor, Employee Benefits Security  Administration: Evaluation of EBSA Customer Service Programs  Participant Assistance Mystery Shopper Evaluation. Washington,  D.C.: The Gallup Organization, January 2004.
Royal, Dawn. U.S. Department of Labor, Employee Benefits Security  Administration: Evaluation of EBSA Customer Service Programs  Participant Assistance Outreach Programs Evaluation. Washington,  D.C.: The Gallup Organization, January 2004.
Royal, Dawn. U.S. Department of Labor, Employee Benefits Security  Administration: Evaluation of EBSA Customer Service Programs  Participant Assistance Web Site Evaluation. Washington, D.C.: The  Gallup Organization, January 2004.
Federal Employees Compensation Act Program: ICF Consulting.  Federal Employees Compensation Act (FECA): Program Effectiveness  Study. Fairfax, Va.: U.S. Department of Labor, Office of Workers’  Compensation Programs, Mar. 31, 2004.
Office of Federal Contract Compliance Programs: Westat. Evaluation  of Office of Federal Contract Compliance Programs: Final Report.  Rockville, Md.: December 2003.
Occupational Safety and Health Administration Reports:    ERG. Evaluation of OSHA’s Impact on Workplace Injuries and  Illnesses in Manufacturing Using Establishment–Specific Targeting  of Interventions. Final report. Lexington, Mass.: July 23, 2004.
Marker, David and others. Evaluating OSHA’s National and Local  Emphasis Programs. Draft Final Report for Quantitative Analysis of  Emphasis Programs. Rockville, Md.: Westat, Dec. 24, 2003.
OSHA, Directorate of Evaluation and Analysis. Regulatory Review of  OSHA’s Presence Sensing Device Initiation (PSDI) Standard [29  CFR 1910.217(h)]. Washington, D.C.: May 2004.  www.osha.gov/dcsp/compliance_assistance/lookback/psdi_final2004.ht ml (Oct. 21, 2005).
section_title: Appendix III: Comments from the Office of Management and Budget
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Stephanie Shipman, Assistant  Director, and Valerie Caracelli made significant contributions to this  report. Denise Fantone and Jacqueline Nowicki also made key  contributions.
section_title: Related GAO Products
paragraphs: Performance Budgeting: PART Focuses Attention on Program  Performance, but More Can Be Done to Engage Congress. GAO-06-28.  Washington, D.C.: Oct. 28, 2005.
Managing for Results: Enhancing Agency Use of Performance  Information for Managerial Decision Making. GAO-05-927. Washington,  D.C.: Sept. 9, 2005.  21st Century Challenges: Performance Budgeting Could Help Promote  Necessary Reexamination. GAO-05-709T. Washington, D.C.: June 14, 2005.
Performance Measurement and Evaluation: Definitions and  Relationships. GAO-05-739SP. Washington, D.C.: May 2005.
Results-Oriented Government: GPRA Has Established a Solid  Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.:  Mar. 10, 2004  Performance Budgeting: Observations on the Use of OMB’s Program  Assessment Rating Tool for the Fiscal Year 2004 Budget. GAO-04-174.  Washington, D.C.: Jan. 30, 2004.
Program Evaluation: An Evaluation Culture and Collaborative  Partnerships Help Build Agency Capacity. GAO-03-454. Washington,  D.C.: May 2, 2003.
Program Evaluation: Strategies for Assessing How Information  Dissemination Contributes to Agency Goals. GAO-02-923. Washington,  D.C.: Sept. 30, 2002.
Program Evaluation: Studies Helped Agencies Measure or Explain  Program Performance. GAO/GGD-00-204. Washington, D.C.: Sept. 29,  2000.
Performance Plans: Selected Approaches for Verification and Validation  of Agency Performance Information. GAO/GGD-99-139. Washington, D.C.:  July 30, 1999.
Managing for Results: Measuring Program Results That Are Under  Limited Federal Control. GAO/GGD-99-16. Washington, D.C.: Dec. 11,  1998.