Managing for Results: Observations on NASA's Fiscal Year 1999 Performance Plan
Released: 1998-06-10T00:00:00

Letter

Background

The Results Act requires that the Office of Management and Budget (OMB) direct each executive agency to prepare an annual performance plan, beginning with fiscal year 1999. The performance plan is one of three components of the Results Act, the others being the strategic plan, submitted by agencies in September 1997, and the annual report due March 31, 2000. The performance plan should describe (1) annual performance goals and measures, (2) the strategies and resources to achieve those goals, and (3) procedures to verify and validate reported performance. The act requires that performance goals be linked to the program activities in agencies’ budgets and be expressed in an objective, quantifiable, and measurable form. OMB is to use agency performance plans to develop the overall federal government performance plan submitted annually to Congress, beginning with the President’s fiscal year 1999 budget.

NASA’s Performance Plan Could Provide a Clearer Picture of Intended Performance Across the Agency

Generally, the performance goals in NASA’s plan are objective and the performance measures are quantifiable and useful for assessing progress. While the plan reflects the mission statement and strategic goals in NASA’s strategic plan and provides clear links between strategic goals and the annual performance goals and measures, it does not clearly associate the plan’s performance goals and measures with program activities in the budget. Although NASA’s plan recognizes that other agencies and partners have complementary missions and activities and acknowledges that NASA must work closely with them to achieve its goals, it does not discuss the extent to which it has coordinated either its strategies for achieving goals or its development of performance measures.

Defining Expected Performance

The performance plan provides near-term goals for each of NASA’s four business enterprises and at least one performance target to address each of the goals for fiscal year 1999. The four enterprises, which were characterized by NASA in its fiscal year 1999 strategic plan as analogous to business units in the private sector, are Space Science; Earth Science; Human Exploration and Development of Space; and Aeronautics and Space Transportation Technology. In addition, the performance plan includes goals and measures for the internal processes it characterizes as crosscutting.

NASA officials told us that the enterprise performance targets are indicators of progress toward the agency’s near-term goals. Generally, the enterprise near-term goals are objective and free from bias. The plan provides both quantitative and qualitative measures for aspects of some of the goals. For example, in addition to quantitative measures for exploring the role of gravity in physical, chemical, and biological processes, the plan provides for using “the data obtained by fluid physics experiments on suspensions of colloidal particles on MSL-1 to answer fundamental questions in condensed matter physics.....” NASA officials acknowledged that this would require the use of qualitative measures to determine whether the fundamental questions have been answered.

OMB recommends that outcome goals be included in a performance plan whenever possible, but recognizes that agencies will supplement outcome goals with output goals. Some of NASA’s performance measures are outcome oriented; others are output oriented. NASA officials told us that outputs are used when demonstrating intermediate progress in support of future outcomes and often reflect program commitments made in conjunction with science users. For instance, they said that the performance target of orbiting the asteroid Eros closer than 50 kilometers—a performance measure associated with NASA’s objective of exploring the solar system—matches Near-Earth Rendezvous program expectations and specifications. A few performance goals are not specific to fiscal year 1999; rather, they span multiple years. For example, the plan provides for reducing “average spacecraft cost for Space Science and Earth Science missions to $190 million from $590 million.” However, the performance target is based on comparing spacecraft programs completed in the time period of fiscal years 1990 through 1994 with those completed between fiscal years 1995 through 1999. NASA officials told us that expressing multiple year programs in a single-year context required by the Results Act was difficult and a source of frustration for the agency.

In order to provide a more comprehensive understanding of the importance of the goals and performance measures chosen for its crosscutting processes, it would have been helpful for the plan to acknowledge the agency’s major management challenges and associated corrective actions. For example, NASA officials told us that the plan’s target of achieving 70 percent or greater of resource authority obligated relates to funding carryovers—an issue we have identified as a major challengeand which the NASA Administrator has recognized in congressional testimony as a top priority. We have also identified other management challenges. These include: improving systems and information for monitoring contractor activities and complying with contract management requirements; implementing a new accounting system and the importance of this effort to providing Congress and agency management with better financial information and allowing the agency to move to full cost accounting; and resolving uneven progress toward better NASA and Department of Defense cooperation and developing a national perspective on aerospace test facilities.

OMB guidance states that performance goals for corrective steps for major management problems should be included for problems whose resolution is mission-critical, such as problems that could materially impede the achievement of program goals. Recognizing those management challenges appears to meet the intent of OMB’s guidance.

Furthermore, although the performance plan includes performance targets related to contracting, such as increasing the obligated funds for Performance-Based Contracts to 80 percent, the performance measure can be better understood by referencing the agency’s efforts to ensure that it has relevant and reliable methods for timely and accurate monitoring of its contract management activities. For example, the performance plan is silent on NASA’s planned corrective efforts in fiscal year 1999 to evaluate the effectiveness of its contract management activities at its field centers.

Connecting Goals, Missions, and Activities

NASA’s performance goals are closely aligned with the agency-wide and enterprise strategic mission, goals, and objectives, and provide good linkages between the agency’s near-term strategic goals and the enterprise near-term goals, objectives, and performance targets. This is accomplished through well-laid out charts summarizing these relationships.

OMB Circular A-11 provides that the annual performance plan should also show how specific performance goals are related to the specific program activities contained in the agency’s program and financing (P&F) schedules in the President’s Budget. NASA’s performance plan contains a table that allocates NASA’s $13.5 billion fiscal year 1999 budget request among its four enterprises and several other consolidated budget accounts. Each of the enterprises is related to a set of performance goals. However, the plan does not clearly explain how NASA’s program activities relate to the budget totals presented for each enterprise. There is no crosswalk to facilitate an understanding of the relationship between the specific program activities in the budget and the goals and measures in the performance plan. Therefore, we believe NASA’s performance plan cannot help Congress understand which performance goals and measures in the plan cover which program activities or whether all program activities in NASA’s budget are covered by its performance goals.

NASA officials with whom we spoke during our review indicated that these relationships can be determined by simultaneously analyzing information in its “trilogy” of documents, namely the annual budget justification, strategic plan, and performance plan. However, they recognize that a clearer connection between performance goals and the program activities in the budget needs to be provided in future performance plans to facilitate analysis. According to those officials, an initiative is underway to align these documents.

Recognizing Crosscutting Efforts

The performance plan points out that collaborative efforts with other agencies and international partners will continue to be key to the successful implementation of enterprise strategies. Although the plan identifies the specific agency or international partner involved in carrying out specific efforts, it does not discuss the extent to which other agencies have been coordinated with in establishing the goals, objectives, and associated performance targets. For example, in describing the objective of developing next-generation computational design tools, the plan indicates that NASA’s efforts are part of the Federal High Performance Computing and Communications (HPCC) initiative. However, there is no discussion as to whether NASA coordinated its performance target of a 200-fold improvement with other federal partners, nor is there an explanation of how NASA’s effort will contribute to the overall federal initiative.

NASA’s performance plan would be strengthened if it identified activities being undertaken to address crosscutting issues of broad national concern and discussed how achieving the agency’s goals will contribute to addressing the crosscutting issues. To indicate how NASA’s programs contribute to addressing a crosscutting issue, the plan could present output goals, output measures, intermediate outcome goals, and intermediate outcome measures that reflect the agency’s contributions to addressing the crosscutting issues.

The performance plan could be improved if it discussed efforts to coordinate NASA’s programs with other federal programs performing related activities and provided evidence of coordination such as joint planning and coordination or development of partnerships aimed at improving program efficiency and effectiveness.

NASA’s Performance Plan Does Not Fully Discuss How the Agency’s Strategies and Resources Will Help Achieve Its Goals

The performance plan does not fully discuss how the agency’s strategies and resources will help achieve its goals. The implementation strategy descriptions in each enterprise section are unclear and do not provide sufficient information on how NASA will go about accomplishing its near-term enterprise goals. The plan only partially discusses resources; it does not describe the operational processes, skill and technology, human, capital, and information resources required to achieve the performance goals.

Connecting Strategies to Results

The NASA plan does not fully describe how the agency will achieve its performance goals, nor does it reference other documents that show how the strategies will contribute to achieving the agency’s performance goals. Enterprise implementation strategy descriptions lack detailed information on how NASA will accomplish its enterprise goals. In several instances, these descriptions appear to outline philosophies rather than describe enabling actions. For instance, application of the “faster, better, cheaper” approach to spacecraft development identified in both the Space Science and Earth Science implementation strategy sections is not clearly explained. At a minimum, it would have been helpful if the plan had included an explicit discussion on what is involved in applying that approach and how the establishment of “prudent risks” is part of it.

While not required by the Results Act, we believe that a similar discussion of external factors and discussion of how NASA will mitigate or use the identified factors to achieve performance goals could provide additional context regarding anticipated performance. With the exception of stating that NASA assumes that International Space Station partners will meet their own schedules, the plan does not explain how NASA will address external factors that could affect the agency’s performance during the coming fiscal year. In its strategic plan, NASA had identified several key external factors, such as the stability of future budgets. We believe that a similar discussion in the performance plan would be helpful. This is particularly pertinent for NASA’s budgetary programmatic priorities, including the International Space Station, which could potentially consume a large portion of NASA’s future resources and affect the timely implementation of other programs.

Connecting Resources to Strategies

NASA’s performance plan partially discusses the resources it will use to achieve the performance goals. Characterization of resources needed by NASA to meet its performance goals is limited to identifying fiscal year 1999 funding requirements by enterprise. In addition, the plan does not discuss operational processes, skills and technology, human and capital resources required to meet the performance goals.

Information technology issues are discussed in the context of the internal crosscutting objective entitled “improving technology capability and services.” The performance target is identified as improving information technology return on investment and customer satisfaction by maintaining a positive return on investment and a “satisfactory” rating from information technology customers. The performance measure of “maintaining a positive return on investment” is vague and does not indicate how the target of improving information technology return on investment will be achieved. Likewise, although maintaining a “satisfactory” rating from customers is a positive measure, it does not indicate whether any improvement is being achieved.

We observe that the information technology goal and measure identified in the performance plan do not specifically address any of the important information technology management issues that NASA currently faces. Specifically, the plan does not discuss how

NASA plans to measure compliance with the Clinger-Cohen Act of 1996, which calls for agencies to implement a framework of modern technology management based on practices followed by leading private-sector and public-sector organizations that have successfully used technology to dramatically improve performance and meet strategic goals; the agency’s new Chief Information Officer (CIO) management structure, which depends on the cooperation of NASA’s diverse enterprises and field centers, will measure its success in providing strategic direction to information technology investments; and

NASA plans to address the “year 2000 problem” (which requires that computer systems be changed to accommodate dates beyond the year 1999) as well as any significant information security weaknesses—two issues that we have identified as high risk across the federal government.Both the “year 2000 problem” and information security weaknesses could seriously undermine the performance and integrity of NASA’s information systems if not adequately addressed.

In addition, NASA’s performance plan contains no discussion of its strategy for achieving improvements in information technology capability and services, nor does it refer to a separate information technology annual plan that might contain such information. As a result, there is no way to assess whether NASA is reasonably prepared to undertake any significant improvement in information technology capability and services or whether such a goal is attainable. Furthermore, the performance plan gives no indication of the resources associated with the information technology improvement goal.

Furthermore, the performance of NASA’s major capital investments in information technology are not specifically addressed in its performance plan. For example, the Earth Science Enterprise section discusses an objective of improving dissemination of Earth Science research results but does not mention the Earth Observing System Data and Information System (EOSDIS). NASA has requested $257 million for EOSDIS in fiscal year 1999. No planned measures of the performance of EOSDIS are included.

NASA’s Performance Plan Partially Provides Confidence That Its Performance Information Will Be Credible

While NASA’s performance plan identifies internal and external organizations that will evaluate performance, it does not include a discussion of the procedures that will be used to verify and validate performance data and the limitations that could exist with internal sources of data that NASA plans to use to assess performance. Although the Results Act does not require that limitations of external sources of data be discussed in performance plans, we believe NASA’s performance plan would be improved with the inclusion of such a discussion.

Verifying and Validating Performance

NASA’s performance plan states that performance will be evaluated by internal and external processes. Internal assessments will be conducted by agency management councils throughout the year and semi-annually by the Senior Management Council. External assessments will be conducted annually by the NASA Advisory Council and organizations such as the National Academy of Sciences. For example, the plan indicates that the Aeronautics and Space Transportation Technology Committee of the NASA Advisory Council will conduct annual assessments of the progress made by the Aeronautics and Space Transportation Technology Enterprise. This committee will provide a qualitative progress measurement, including commentary to clarify and supplement the qualitative measures. Such commentary would constitute a means to verify and validate measured values.

Recognizing Data Limitations

The plan falls short in this area. NASA’s plan does not address limitations to the data from internal or external sources. To gauge its progress, NASA enterprises will likely rely extensively on internal sources of data, such as program management reports. However, the plan does not discuss any limitations that these data sources have. For example:

As previously discussed, the performance target of “improving information technology” will require assessments of the trend in the ratio of benefits achieved to information technology costs. The plan would be more complete if it identified the challenges associated with determining benefits attributable to information technology investments. In describing its internal crosscutting process characterized as “Communicate Knowledge,” one of the objectives described is the development of educational outreach programs. One of the associated performance targets in fiscal year 1999 is to increase the number of students reached through its NEWEST/NEWMAST programs to 42,000 students from 37,000 students. It is not clear whether NASA will do its own survey or need to rely on other external information sources.

NASA’s performance plan could also benefit from a discussion of the accuracy and reliability of data from external sources, even though such a discussion is not required by the Results Act. As previously discussed, the plan recognizes that securing collaboration and cooperation with other agencies and research partners is key to the successful accomplishment of its goals. In its description of the objective of making major scientific contributions to national and international environmental assessments, the plan can be improved by indicating whether NASA will rely on information provided by its partner, the Federal Aviation Administration (FAA), to assess atmospheric effects of aviation, and if so, how the accuracy and reliability of FAA’s data will be assessed. Since the performance target associated with this objective is to make significant contributions—further defined by NASA officials as being referenced in technical literature—such added information could influence greater use of the information generated.

As previously noted, a critical component to NASA’s evaluation of its performance is the establishment of a financial management system and its integration with full cost accounting. Full cost accounting facilitates decision making and increases accountability by automating the agency’s operations and linking data results and costs to major agency activities, budgets, accounts, and reports. Unfortunately, all the information NASA needs to measure the costs associated with its performance is currently not available. NASA’s plans to implement full cost accounting are contingent upon the timely completion of its new financial management system, which NASA plans to implement in 1999. Until the new integrated financial management system is operational, performance assessments relying on cost data may be incomplete.

Agency Comments and Our Evaluation

We provided a draft of this report to NASA for review and comment. In its comments, NASA stated that it will continually improve the content of its annual plan and recognizes the usefulness of our comments and recommendations. NASA intends to use its strategic plan, budget justification, and performance plan as a “trilogy of documents” to be reviewed as a package. Even though the areas we identified in our report as lacking specificity may be addressed in NASA’s strategic plan, budget justification and other documents, we maintain that the performance plan could be more useful by better linking performance goals and measures to program activities in the budget; more fully explaining procedures for verifying and validating performance data and recognizing the limitations that affect the credibility of data that will be used to measure performance; and by acknowledging major management challenges and associated corrective actions. Including these areas in the plan would provide a more comprehensive understanding of the importance of NASA’s goals and performance measures, making it more useful for purposes of the Results Act.

NASA’s comments and our evaluation of them are presented in appendix I.

Scope and Methodology

The scope of our work was limited to the information contained in NASA’s fiscal year 1999 performance plan and clarifying discussions we had with NASA officials. The criteria we used to conduct this review were the Results Act, OMB’s guidance on developing the plans (Circular A-11, part 2), our February 1998 guidance for congressional review of the plans (GGD/AIMD 10.1.18), and the December 17, 1997, letter to OMB Director Raines from several congressional leaders. For purposes of our analysis, we collapsed the six requirements for annual performance plans in the Results Act and the related guidance into three core questions: (1) To what extent does the agency’s performance plan provide a clear picture of intended performance across the agency? (2) How well does the performance plan discuss the strategies and resources the agency will use to achieve its performance plan? (3) To what extent does the agency’s performance plan provide confidence that its performance information will be credible?

We conducted our work between December 1997 and April 1998 in accordance with generally accepted government auditing standards.

As requested, we plan no further distribution of this report until 7 days after its issue date. At that time we will send copies to the Director of the Office of Management and Budget, the Administrator of NASA, and appropriate congressional committees. We will also make copies available to other interested parties on request.

If you or your staff have any questions concerning this report, please contact me on (202) 512-4841. The major contributors to this letter are listed in appendix II.

Comments From the National Aeronautics and Space Administration

The following are GAO’s comments on NASA’s letter dated April 24, 1998.

GAO Comments

1. Responding to the issues raised in this report, NASA concludes that it has complied with the requirements of GPRA. We agree. Our point in the report was that the plan contains areas that could be strengthened, making it more useful to the Congress. In discussing the strengths of NASA’s performance plan, our report recognizes that the plan provided good linkage between strategic goals and the goals in the performance plan and that performance measures were generally objective, quantifiable, and useful for assessing progress. However, our report also clearly states that NASA’s plan could provide a clearer picture of intended performance across the agency and does not fully portray how NASA’s strategies and resources will help it achieve the plan’s performance goals. The report also states that NASA’s plan does not clearly explain how program activities relate to the budget; and that there is no crosswalk to facilitate an understanding of the relationship between the specific program activities in the budget and the goals and measures in the performance plan. As stated in our report, NASA is in agreement to merge the budget justification and performance plan process in an effort to provide a crosswalk showing the relationship of performance targets to program activities and resources. 2. In commenting on measures for major management challenges, NASA recognizes that the plan does not include measures to improve the management challenges we address. While these management issues may be addressed in other management documents, such as the Chief Financial Officer’s Five-Year Plan, Chief Information Officer’s Information Technology Implementation Plan and implementation planning documents being developed by the Office of Procurement, to which NASA alludes, we did not review these documents; neither does the performance plan include reference to them. Referencing these documents in the plan would have been useful in providing specific sources for obtaining a more comprehensive understanding of the importance of these issues. We continue to believe that an acknowledgement of major management challenges and associated corrective actions would make the plan more useful for purposes of the Results Act. 3. In addressing the issue of resource requirements to achieve performance, NASA correctly restates our concern that the plan only addresses funding requirements for fiscal year 1999 at the Enterprise level. As in their comment to our point on not identifying management challenges, NASA makes the assumption that, since a discussion of operational processes, skills and technology, human and capital resources required to meet the performance goals is provided in the budget justification, it need not be included in the performance plan. Referencing these documents in the plan would have been useful in providing specific sources for obtaining a more comprehensive understanding of the importance of these issues. 4. Responding to our concern that the plan’s implementation strategy of applying the “faster, better, cheaper” approach to spacecraft development is not clearly explained, NASA contends that the statement that follows in the plan is a clear enough description of its approach. We do not agree. The statement NASA alludes to says that “Program managers are encouraged to accept prudent risk, shorten development time of technologies and missions, explore new conceptual approaches, streamline management, and incorporate innovative methods and technologies to enhance efficiency and effectiveness.” In our opinion, this statement does not provide a clear explanation of NASA’s implementation strategy. We continue to believe that a more explicit discussion of this approach is needed. 5. Our issue regarding EOSDIS is not in its lack of identification in the section discussing crosscutting information technology. Rather, we take issue with EOSDIS not being specifically mentioned either in the references cited in NASA’s comments or anywhere else in the performance plan, despite the fact that NASA plans to spend well over $1 billion on the system. NASA’s comment assumes that the reader of the plan would recognize EOSDIS as the vehicle for improving dissemination of Earth Science research results. We believe that a large investment such as EOSDIS should be explicitly addressed in the performance plan in order to ensure that its actual performance is understood. For example, a test of one part of EOSDIS in March 1998 revealed software problems that will force the launch of the EOS AM-1 satellite to be delayed by at least 6 months. We believe that the cost and schedule impact of this and other EOSDIS development problems needs to be explicitly and objectively assessed. 6. Responding to our concern about the lack of any discussion in the plan about the credibility of performance information, NASA officials expressed the belief that the data it collects will be credible and that we may be premature in passing judgement. As support, NASA references the audit of its fiscal year 1996 and fiscal year 1997 Accountability Reports by Arthur Andersen LLP, which the agency says found no significant issues with the credibility of NASA’s performance information plans. We have three points regarding NASA’s comment. First, the plan would have been strengthened by referencing the results of the Arthur Andersen LLP audit in the performance plan. Second, while a review of the Accountability Reports was not in the scope of our review, we believe that the objectives and performance measures in the performance plan are likely to be more results oriented than those included in the Accountability Reports NASA references. Third, while the plan does identify internal and external organizations that will evaluate its performance, we believe that it is not premature for NASA’s performance plan to include a discussion of procedures that will be used to verify and validate performance data and the limitations that could exist with internal sources of data that NASA plans to use to assess performance.

Major Contributors to This Report

National Security and International Affairs Division, Washington, D.C.

Office of the General Counsel

Related GAO Products

NASA Budget: Carryover Balances in Selected Programs (GAO/NSIAD-96-206, July 16, 1996).

High-Risk Program: Information on Selected High-Risk Areas (GAO/HR-97-30, May 16, 1997).

Results Act: Observations on NASA’s May 1997 Draft Strategic Plan (GAO/NSIAD-97-205R, July 22, 1997).

NASA: Major Management Problems (GAO/T-NSIAD-97-178, July 24, 1997).

Managing for Results: Agencies’ Annual Performance Plans Can Help Address Strategic Planning Challenges (GAO/GGD-98-44, Jan. 30, 1998).

Aerospace Testing: Promise of Closer NASA/DOD Cooperation Remains Largely Unfulfilled (GAO/NSIAD-98-52, Mar. 11, 1998).

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.