IG-01-020

AUDIT

REPORT VALIDATION AND VERIFICATION OF

SELECTED NASA FY 2000 PERFORMANCE

DATA RELATED TO THE GOVERNMENT

PERFORMANCE AND RESULTS ACT (GPRA)

March 30, 2001

National Aeronautics and

Space Administration

OFFICE OF INSPECTOR

GENERAL
ii

Additional Copies

To obtain additional copies of this report, contact the Assistant Inspector General for Auditing

at (202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General for

Auditing. Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing

Code W

NASA Headquarters

Washington, DC 20546-0001

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800)

424-9183, (800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form; or

write to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, Washington,

DC 20026. The identity of each writer and caller can be kept confidential, upon request, to the

extent permitted by law.

Reader Survey

Please complete the reader survey at the end of this report or at

www.hq.nasa.gov/office/oig/hq/audits.html.

Acronyms

CFO Chief Financial Officer

EOCAP Earth Observing Commercialization Applications Program

FY Fiscal Year

GAO General Accounting Office

GPRA Government Performance and Results Act

HEDS Human Exploration and Development of Space

OIG Office of Inspector General

OMB Office of Management and Budget
W March 30, 2001

TO: A/Administrator

FROM: W/Inspector General

SUBJECT: INFORMATION: Validation and Verification of Selected NASA FY 2000

Performance Data Related to the Government Performance and

Results Act (GPRA)1

Report Number IG-01-020

The NASA Office of Inspector General (OIG) has completed an audit of the accuracy and

reliability of performance data for selected GPRA performance targets2 that will be in the

Agency's Fiscal Year (FY) 2000 Performance Report.3 The audit is a continuation of our

oversight of NASA's implementation of GPRA as described in our Results Act Review Plan.4,5

The annual Performance Report is a document that the Congress and Office of Management

and Budget (OMB) will use to assess NASA's overall performance and may make decisions on

Agency programs and funding levels. NASA's FY 2000 Performance Report will contain the

Agency's assessment of its actual performance against 211 performance targets. The

supporting data and information on 19 of 23 performance targets we reviewed6 were adequate,

and we did not identify any significant problems with reported actual performance for those

targets.

 

1

 Congress enacted GPRA in 1993 to improve public confidence in the Federal Government by holding

agencies accountable through setting program goals, measuring performance against those goals, and

reporting publicly on progress. This act is contained in Public Law 103-62.

2

 Performance target is the term NASA uses in the Performance Plan for those measures or metrics that were

established to accomplish (and measure) the individual goals and objectives. Target, as used in this report,

generally equates to the terms “measure” or “indicator” as used in the GPRA.

3

In March 2000, NASA published its first Performance Report covering FY 1999. At the time we performed

the audit, NASA was collecting data for its Performance Report covering FY 2000.

4

 An October 1998 letter signed by the House Majority Leader and Chairmen of the House Committee on

Government Reform and Oversight; the House Subcommittee on Government Management, Information,

and Technology; and the Results Caucus asked the NASA OIG to establish a GPRA review plan to assess

Agency controls. In response to the request, the OIG included a plan in its Semiannual Report for

March 31, 1999.

5

 The NASA OIG has issued reports on previous audits of NASA's implementation of GPRA. Additional

details on the audits are in Appendix B.

6

 We reviewed 23 targets that related to 6 critical areas: Procurement, Financial Management, Information

Technology, International Space Station, Program and Project Management, and Safety and Mission

Assurance.
2

2

However, the performance reported on four targets was not fully reliable because the

supporting data did not accurately support the results described.7 Reported performance for

some of the 188 targets not reviewed may also not be fully reliable for the same reasons.

Reporting performance results that are not fully reliable limits the usefulness of the Performance

Report to NASA, OMB, and the Congress. Although NASA had taken steps to validate

performance information that will go into the annual Performance Report, NASA could further

improve the process.

Background

Because NASA Centers implement many of the programs and activities that have GPRA

performance goals and targets, they were the source for much of the data used to measure and

evaluate actual performance. For most of the targets we reviewed, the Headquarters offices

responsible for the program or activity had collected the data from the Centers, developed a

written assessment of the actual performance, and submitted that assessment to the Chief

Financial Officer (CFO) for use in preparing the Performance Report. We evaluated those

assessments and the supporting information during the audit.

Management's Response and OIG Evaluation

NASA concurred with our recommendations. Management reviewed the performance

statements for the four performance targets and made the necessary corrections or clarifications

in the FY 2000 Performance Report. In addition, for each performance target, NASA included

a description of the methods used to verify and validate supporting data and identified the data

source. These actions help to substantiate reported actual performance and improve the

usefulness of the Performance Report. In addition, the CFO emphasized the need to develop

clear and measurable performance targets in the FY 2002 Performance Plan data call letter and

the FY 2003 performance metric development guidance. The guidance on the FY 2003

performance metrics also summarized the characteristics of good performance metrics. NASA

will continue to emphasize the need to disclose data limitations in future performance reporting.

Details on the status of the recommendations are in the finding section of the report.

[original signed by]

Roberta L. Gross

Enclosure

 

7

 For purposes of our audit and this report, the terms “reported performance” and “reported results” are the

same and refer to written self-assessments of actual results prepared by the responsible GPRA officials and

provided to the NASA Chief Financial Officer for the Performance Report.
3

Final Report on Audit of Validation and Verification of Selected NASA FY 2000 Performance

Data Related to the Government Performance and Results Act (GPRA)
FINAL REPORT

AUDIT OF VALIDATION AND VERIFICATION OF SELECTED

NASA FY 2000 PERFORMANCE DATA RELATED TO THE

GOVERNMENT PERFORMANCE AND RESULTS ACT (GPRA)
W March 30, 2001

TO: B/Acting Chief Financial Officer

FROM: W/Assistant Inspector General for Auditing

SUBJECT: Final Report on Audit of Validation and Verification of Selected NASA FY

2000 Performance Data Related to the Government Performance and Results

Act (GPRA)

Assignment Number A0100500

Final Report IG-01-020

The subject final report is provided for your information and use. Please refer to the Executive

Summary for the overall audit results. Our evaluation of your response is incorporated into the

body of the report. The corrective actions completed for the recommendations were

responsive. Management's actions are sufficient to close the recommendations for reporting

purposes.

If you have questions concerning the report, please contact Mr. Chester A. Sipsock, Program

Director, Environmental and Financial Management Audits, at (216) 433-8960, or Ms. Carol

A. St. Armand, Audit Program Manager, at (301) 286-7269. We appreciate the courtesies

extended to the audit staff. The report distribution is in Appendix E.

[original signed by]

Russell A. Rau

Enclosure

cc:

AE/Chief Engineer

AS/Chief Scientist

BF/Director, Financial Management Division

BR/Acting Director, Resources Analysis Division

JM/Director, Management Assessment Division
vi

2

M/Associate Administrator for Space Flight

R/Associate Administrator for Aerospace Technology

S/Associate Administrator for Space Science

U/Acting Associate Administrator for Biological and Physical Research

Y/Associate Administrator for Earth Science

Z/Acting Associate Administrator for Policy and Plans
Contents

Executive Summary, i

Introduction, 1

Finding and Recommendations, 3

Reliability of Data for Reported Results, 3

Appendix A - Objectives, Scope, and Methodology, 9

Appendix B - Summary of Prior Audit Coverage, 11

Appendix C - Performance Targets Reviewed in Detail, 12

Appendix D - Management’s Response, 15

Appendix E - Report Distribution, 20
NASA Office of Inspector General

IG-01-020

 A0100500 March 30, 2001

Validation and Verification of Selected

NASA FY 2000 Performance Data Related to the

Government Performance and Results Act (GPRA)

Executive Summary

Background. The NASA OIG has completed an audit of the accuracy and reliability of

performance data for selected GPRA performance targets that will be in the Agency's FY 2000

Performance Report. The Performance Report is an important document that NASA, the

Congress, and the OMB will use to assess the Agency's overall performance and make

decisions on programs and funding levels. The audit is a continuation of our oversight of

NASA's implementation of GPRA as described in our Results Act Review Plan.

Objectives. Our overall audit objective was to assess the quality of data supporting the

reported results that will be in the NASA FY 2000 Performance Report. We reviewed the

supporting data for 23 performance targets related to 6 critical areas: Procurement, Financial

Management, Information Technology, International Space Station, Program and Project

Management, and Safety and Mission Assurance.

Appendix A contains further details on the audit objectives, scope, and methodology.

Appendix C provides details on the 23 performance targets reviewed.

Results of Audit. We considered the supporting data and information on 19 (83 percent) of

the 23 performance targets reviewed to be adequate and did not identify any significant

problems with reported actual performance. However, the reported performance on four

performance targets is not fully reliable because the data reviewed did not accurately support

the described results. Given this finding and the results from previous GPRA audits, we surmise

that the reported performance for some of the 188 targets not reviewed may also not be fully

reliable for the same reasons. NASA could improve the accuracy of the FY 2000 Performance

Report by more effectively validating the supporting data and by developing clearer, more

specific targets. This would increase the Performance Report's value as a source of information

for making important program and funding decisions.

We commend NASA for a significant improvement in the reporting of actual performance.

Two factors contributed to this. First, FY 2000 was the second year that the Agency prepared

a Performance Report. NASA overcame many of the learning hurdles associated with

preparing the first report. Second, the CFO's data call for the FY 2000 Performance Report
ii

prescribed a performance reporting format.8 For each performance target, a responsible

GPRA official was to provide an assessment of actual performance, a description of data

sources for the supporting information, and documentation of the method of verification and the

process used for validation of results. We believe that the prescribed reporting format led to

improved narratives for the performance targets.

Recommendations. The responsible GPRA officials should review and correct information on

the four targets discussed in this report and verify and validate the supporting data for the other

FY 2000 targets not audited to ensure that all reported results in the Performance Report are

accurate and reliable; develop future GPRA targets that are clear and represent desired

performance; and fully disclose data limitations in future performance reports.

Management's Response. Management concurred with the recommendations. NASA

management has reviewed and corrected the performance statements for the four performance

targets discussed in this report and will include the revised statements in the Agency’s final

Performance Report. In correspondence to the Strategic Enterprises, Headquarters Offices,

and Centers on performance targets for FY's 2002 and 2003, management has emphasized the

need to develop clear and measurable performance targets. Management will continue to stress

the requirement to report data limitations in future Performance Reports. Beginning with the FY

2002 final Performance Plan, NASA will discuss anticipated data limitations. The complete text

of management's response is in Appendix D.

Evaluation of Response. Management's planned actions are responsive to the

recommendations. A detailed evaluation of management's comments is provided with each

recommendation in the body of the report.

 

8

 The CFO issued a data call letter for the FY 2000 Performance Report and FY 2000 Accountability Report to

the Officials-in-Charge of Headquarters and Functional Offices (September 26, 2000).
Introduction

During FY 2000, NASA conducted its programs and activities through four Strategic

Enterprises that constitute NASA's primary missions. The Strategic Enterprises are Space

Science, Earth Science, Human Exploration and Development of Space (HEDS), and

Aerospace Technology.9 Four Crosscutting Processes support the Strategic Enterprises and

enable them to perform their mission activities. The four Crosscutting Processes are Manage

Strategically, Provide Aerospace Products and Capabilities, Generate Knowledge, and

Communicate Knowledge. The Associate Administrator for each Strategic Enterprise is

responsible for GPRA performance within the respective Enterprise. For the four Crosscutting

Processes, the Associate Administrator for Policy and Plans is responsible for Manage

Strategically; the Chief Engineer is responsible for Provide Aerospace Products and

Capabilities; and the Chief Scientist is responsible for both the Generate Knowledge and

Communicate Knowledge Processes. NASA refers to the responsible GPRA official for each

Crosscutting Process as the GPRA Steward.

NASA's Office of Policy and Plans is responsible for developing and implementing the Strategic

Plan. The Associate Administrators for the Strategic Enterprises and the GPRA Stewards are

responsible for developing and implementing the annual Performance Plan and for reporting on

actual performance for the annual Performance Report. The CFO coordinates the performance

planning and reporting processes, collecting information from the Associate Administrators for

the Strategic Enterprises and the GPRA Stewards to prepare the annual Performance Plan and

Performance Report.

NASA Centers are responsible for implementing many of the programs and activities that have

GPRA performance goals and targets. Therefore, Center systems were the source for much of

the data used to measure and evaluate actual performance. The Associate Administrators for

the Strategic Enterprises and the GPRA Stewards collected the data from the Centers,

developed a written assessment of the actual performance, and submitted the assessment to the

CFO for use in preparing the Performance Report.

In its review of NASA's FY 2000 Performance Plan, the General Accounting Office (GAO)

stated that the Plan provides limited confidence that the Agency's performance information is

credible. The Plan identifies internal and external organizations that will evaluate performance,

provides expanded detail on such evaluations, and identifies specific internal and external

sources for data. However, the Plan does not include an explicit discussion of the procedures

the Agency will use to verify and validate performance data. In addition, the Plan does not

address possible limitations in internal and external sources of data.10

 

9

 On September 29, 2000, NASA established the Office of Biological and Physical Research, which formed a

separate and fifth Enterprise focusing on scientific research. The Associate Administrators for the Office of

Space Flight and the Office of Biological and Physical Research are each responsible for separate

performance targets in the HEDS Enterprise.

10 More details on GAO's review are in Appendix B.
2

We noted in an earlier OIG audit11 that the Agency lacked specific procedures for NASA

program offices to follow in verifying performance data and reported results. NASA

management took the position that most of the data used to measure GPRA-related

performance came from internal sources, thus additional procedures were not necessary to

ensure accuracy and reliability. Details of the prior audit are in Appendix B. In another audit of

the supporting data for selected performance targets in NASA's FY 1999 Performance

Report,12 we considered the reported performance on five targets reviewed as not fully reliable.

We found that the supporting data did not accurately support the results described.

 

11 Report Number IG-99-055, “NASA Implementation of the Government Performance Results Act,” was

issued September 28,1999. The report discusses NASA's efforts to develop and use performance measures

for determining progress toward achieving the performance goals and program outcomes described in its

annual performance plans and performance reports under the GPRA.

12 Report Number IG-00-020, “Validating FY 1999 Performance Data to Be Reported Under the Government

Performance Results Act,” was issued March 28, 2000. The report discusses the accuracy and reliability of

supporting data for selected performance targets. See Appendix B for details.
3

Finding and Recommendations

Reliability of Data for Reported Results

For 4 (17 percent) of 23 performance targets reviewed, responsible GPRA officials prepared

written assessments that did not accurately reflect supporting data and actual results. This

occurred because the officials did not consistently follow procedures in place to verify and

validate supporting data and the results. Also contributing to the problems were poor phrasing

of some targets. The planned reported performance on these four targets is, therefore, not fully

reliable, which may limit the usefulness of the information to NASA, OMB, and the Congress

for decisionmaking. Reported performance for some of the 188 targets not reviewed may also

not be fully reliable for the same reasons. Management attention is needed to address and

correct these problems before issuing future Performance Reports.

GPRA Requirements

GPRA require s an agency to prepare an annual Performance Report that compares actual

performance with the performance targets set out in the annual Performance Plan. When a

performance target is not met, the Performance Report should include an explanation for not

achieving a performance target and describe steps for meeting the goal in the future. For the

annual Performance Report to be useful, the data on the actual achievements on the Agency's

performance goals and targets and the comparisons of planned and actual performance must be

accurate.13 GPRA further requires the annual Performance Plan to include a description of the

means used to verify and validate measured values. Also, to have accurate measurements of

actual performance, it is important that the targets are described in the Plan in a manner that

ensures the planned achievements and how they are measured are clear.

Verifying and Validating Supporting Data and Results

NASA's FY 2000 Performance Plan described, as required by GPRA, the means by which the

Agency would verify its performance data. The primary means were reviews by internal and

external groups. Internally, standard monthly and quarterly project- and program-level reviews

occur at the Centers, at contractor installations, and at NASA Headquarters. Program

Management Councils14 assess program schedules, cost, and technical performance against

established programmatic commitments. External review processes include peer reviews by

outside scientific experts to ensure that science research proposals are selected strictly on the

 

13 NASA also includes selected performance information in an Accountability Report, which is issued as

part of the annual Financial Statements. We discussed with CFO staff our audit findings on specific targets

that were also going to be included in the 2000 Accountability Report so that the CFO staff could take

appropriate action.

14 The Program Management Council is the Senior Management Group, chaired by the Deputy

Administrator, responsible for reviewing and recommending approval of programs and for overseeing their

implementation according to Agency commitments, priorities, and policies.
4

merits of the planned research. Other external groups involved with verifying performance data

include the OIG, the GAO, and the NASA Advisory Council.15

For 19 of the 23 reviewed performance targets that will be in NASA’s FY 2000 Performance

Report, we did not find any significant problems with the actual performance the Associate

Administrators for the Strategic Enterprises and the GPRA Stewards reported. Except for

minor errors, the supporting data and manner in which the actual results were reported were

generally adequate. However, as evidenced by the four targets discussed below, there could be

further improvement in the overall process for validating and verifying all GPRA performance

data and reported results. These targets have data limitations or special circumstances that

should be discussed in the FY 2000 Performance Report.

Target 0H13: “Achieve 85% on-time, successful launches, excluding weather risk.”

In January 2000, the Administrator and the Associate Administrator for Space Flight, along with

the Space Shuttle Program Manager, believed that the wording of this target might be

interpreted as compromising safety. Consequently, the officials replaced the target originally

included in the FY 2000 Performance Plan with the following: “Achieve 100% on-orbit mission

success.” For the FY 2000 Performance Report, NASA plans to report that it failed to

achieve the original target but that it has achieved the desired performance for the replacement

target. We believe, however, that the replacement target is unclear.

The target, as replaced, relates to a program managed by the Office of Space Flight. NASA

plans to report that it achieved at least a 100-percent success rate on each of the four Space

Shuttle missions completed in FY 2000. In reviewing the data to support this target, we found

that the actual measurement was of accomplishment of major mission objectives. We believe

that the target, as written, is misleading because it could be assumed that all planned activities

for the mission were included in the reported target results.

The Shuttle Program develops objectives for each mission. Depending on the activities planned

for the mission, there could be numerous objectives. However, the program prioritizes these

objectives, and those that support the primary mission(s) for that flight are documented as

Major Mission Objectives that the Shuttle Program will use as performance incentive criteria in

the Shuttle Flight Operations Contract. The Office of Space Flight also uses the criteria to

judge the success of this performance target (0H13). The Shuttle Program, however, still

makes a determination as to whether the other mission objectives were met. They may or may

not be achieved. The Shuttle Program

 

15 The NASA Advisory Council and its committees are considered “internal” because they are chartered by

NASA, their members chosen by NASA, and they provide their advice and counsel directly to the NASA

Administrator.
5

also develops metrics for the success of the mission considering the success of all the payloads

and the success of the Shuttle in providing the promised support for the payload customer.

If NASA plans to use the replacement target in future performance plans, it should restate the

metric to match the data being measured. For example, “Achieve 100% of major mission

objectives.”

Target 0P2: “Ensure the availability of NASA’s spacecraft and ground facilities by

decreasing the FY99 unscheduled downtime.” This target was one of the crosscutting

performance targets in the FY 2000 Performance Plan that affects the process by which

NASA’s Strategic Enterprises and their Centers deliver systems (ground, aeronautics, and

space); technologies; data; and operational services (which includes space networks) to NASA

customers. The target relates to the Provide Aerospace Products and Capabilities Crosscutting

Process, which the Office of the Chief Engineer manages. NASA plans to report that the target

was achieved. In reviewing the target, we identified data limitations in the way performance

results were written. Therefore, we do not consider the reported assessment to be complete

and accurate.

Spacecraft. Concerning the spacecraft portion of target 0P2, the reported performance

data was limited to specific spacecraft and space networks. For example:

• The Office of Space Science submitted data showing downtime only for spacecraft at

Goddard Space Flight Center and the Jet Propulsion Laboratory. Spacecraft at the

Marshall Space Flight Center were not included in the reported data.

• The Office of Space Flight reported on two of three space networks: Space Network -

TDRS and the Deep Space Network. The Wide-Area Network was not included in the

reported data.

In both instances, the data submitted to the Office of the Chief Engineer clearly identified the

spacecraft and space networks that the performance data represented. However, the reported

performance results submitted for the NASA Performance Report did not indicate that the

reported unscheduled downtime did not include all NASA spacecraft and space networks.

Ground Facilities. NASA’s major facility managers were instructed to input downtime

data into a Web site by a specified date. Immediately after the deadline passed, the Office of

the Chief Engineer accessed the site to retrieve the data. The data showed that Kennedy Space

Center (Kennedy) and Wallops Flight Facility (Wallops) had not reported any data on

downtime for their ground facilities. According to the Office of the Chief Engineer, all ground

facilities were included in the reported downtime data. We contacted the performance target

point of contact for ground facilities at Kennedy and Wallops to determine whether they had

submitted any data. The supporting data they provided us showed that Kennedy had input data

subsequent to the cutoff date but that Wallops still had not. The Office of the Chief Engineer,

however, could not provide supporting data showing that the data originally retrieved from the

Web site and used to report performance against this target had been updated to include the

data eventually input into the Web site by Kennedy. Thus, based on the supporting data
6

received, we were unable to determine whether all facilities were included in the downtime

calculation. The Office of the Chief Engineer did not disclose the data limitations of the

supporting data for this performance target when that office submitted the performance

assessment to the CFO.

Target 0R10: “Complete NSTAR [NASA Solar Electrical Propulsion Technology

Application Readiness] Mission Profile (100% design life) ground testing for Deep

Space –1 (concurrent, identical firing of an NSTAR engine in a vacuum chamber with

the actual firing sequence of the in-flight propulsion system).” This target relates to the

expansion of space research and exploration under the Office of Aerospace Technology.

NASA plans to report that the target was achieved. In reviewing the target performance

results, we were concerned with how the target was written. In our opinion, the target

performance is unclear as to how NASA measured “design life” and documented the test

results.

The CFO format for performance reporting states that documentation of the target

accomplishments should include persuasive evidence of performance that is auditable and

supportable. Information should be included indicating where the data came from, who verified

it, the process that was used to verify it and how it was validated. The reported performance

for this target did not contain this information.

To validate supporting data and reported results on this target, we reviewed program

assessments performed by NASA’s Office of Aerospace Technology as well as assessments

performed independently by other Agency components. The Office of Aerospace Technology

performs an assessment of selected aerospace technology programs each month. All programs

are subjected to assessment at least once every 3 months. In addition, the Aerospace

Technology Committee under the NASA Advisory Council conducted an independent

assessment of the targets. The Aerospace Technology Committee concurred with the GPRA

assessment prepared by the Aerospace Technology Strategic Enterprise. However, we were

unable to obtain auditable documentation for target accomplishments. Therefore, we could not

verify whether the target was achieved.

We discussed our concern with Office of Aerospace Technology representatives during the

audit. They submitted a rewritten performance results paragraph for the NASA Performance

Report. The new paragraph states that “design life equated to the consumption of 87 kilograms

of xenon propellant” and that the success of the tests (repeated firings) made ion propulsion a

legitimate option for deep space solar system exploration missions. Even with the addition of

the information, the paragraph still does not address how NASA documented the test results.
7

Target 0Y44: “Focus EOCAP [Earth Observing Commercialization Applications

Program] joint commercial applications research to develop 20 new market commercial

products (e.g., oil spill containment software by EarthSat; map sheets products by

ERDAS, Inc.).” This target relates to a program managed by the Office of Earth Science

under the NASA Commercial Remote Sensing Program. NASA plans to report that it

achieved the target performance. In reviewing this target, we determined that all of the 20

products reported as new were not. Therefore, we do not consider the reported assessment to

be accurate.

To verify the reported data, we reviewed the Commercial Remote Sensing Program Web site

under EOCAP projects. We reviewed the Web page and product description of each

company cited in the target and performance assessment and noted the year each was funded.

We randomly selected 6 of the 20 companies to contact directly and determine whether their

product was released to the market and, if so, when. One of the six products was released

commercially in 1996, three in 1997, one in 1999, and one in 2000. Only one of the six

products selected was a new market product released in 2000.

The focus of the EOCAP is to broaden the acceptance and use of remote sensing technology in

the marketplace by combining market knowledge with technical capability to guide product

development based on customers’ needs. NASA shares technical, financial, and productdevelopment risks with private sector companies while providing access to facilities and

technical expertise. EOCAP projects are typically 3-year projects. There is a time lag between

funding and bringing a product to market. After a project is completed, the entity has 2 years to

report on it. The projects cited in the target performance assessment were funded as far back

as 1993. A 1993 product would be released commercially by 1996, and the entity would have

until 1998 to report on its achievement in the market place. We determined that the actual

performance described in the target assessment was inaccurate because most of the reported

EOCAP products were already on the market. Therefore, the reported performance on this

target is not accurate.

NASA is preparing to issue its second Performance Report. The Agency is still learning how to

effectively implement GPRA and measure performance under the Act’s requirements. Though

not required by GPRA, NASA plans to identify the data source and the methods used to verify

and validate the supporting data for each performance target in its FY 2000 Performance

Report. By providing this information, NASA is improving the usefulness of its Performance

Report to the Congress, OMB, and others. However, NASA’s GPRA performance

measurement process continues to be an area of concern as discussed in this report.

Management actions are needed to ensure that all the information in the annual Performance

Plans and Performance Reports is reliable and useful to decision makers.
8

Recommendations, Management's Response, and Evaluation of

Response

The Associate Administrators for the Strategic Enterprises and the GPRA Stewards

should:

1. Review the information on the four performance targets discussed in this report

as well as the supporting data for all other FY 2000 targets to ensure that results

reported in the FY 2000 Performance Report are accurate and reliable.

2. Develop clear, future GPRA targets that appropriately represent the desired

performance expected to be achieved.

3. Disclose fully all target data limitations in future performance reports when

reporting actual performance.

Management's Response. Concur. Management reviewed the performance statements for

the four performance targets and made the necessary corrections or clarifications in the FY

2000 Performance Report. In addition, for each performance target, NASA included a

description of the methods used to verify and validate supporting data and identified the data

source.

In the FY 2002 Performance Plan data call letter and the FY 2003 performance metric

development guidance sent to the Associate Administrators for the Strategic Enterprises and the

GPRA Stewards, the CFO emphasized the need to develop clear and measurable performance

targets. The guidance on the FY 2003 performance metrics also summarized the characteristics

of good performance metrics. To ensure that stakeholders and the public more fully appreciate

NASA's planned performance, the NASA Office of Public Affairs incorporated statements of

Public Benefit in the FY 2002 Performance Plan. The statements are intended to show how

NASA performance targets are relevant to the public and congressional decision makers.

The FY 2000 Performance Report data call letter also emphasized the requirement to report

data limitations in performance statements. NASA will continue to emphasize the need to

disclose data limitations in future performance reporting. The complete text of management's

response is in Appendix D.

Evaluation of Response. Management's actions are responsive to the recommendations.

The actions taken and planned in response to the recommendations show a strong management

commitment to GPRA and a willingness to ensure that all the information in the annual

Performance Plan and Performance Report are reliable and useful to decision makers.
9

Appendix A. Objectives, Scope, and Methodology

Objectives

The overall objective was to assess the quality of data supporting the reported results that will

be contained in the NASA FY 2000 Performance Report. The specific objectives were to

review and test selected performance targets to assess: (1) whether the data are appropriate for

the performance target and (2) whether the data is complete, accurate, consistent, and timely.

Scope and Methodology

The audit covered targets contained in NASA's FY 2000 Performance Plan. NASA's Plan

included 211 performance targets. To perform this audit, we concentrated on targets that fit

within six areas considered critical to the Agency: Procurement, Financial Management,

Information Technology, International Space Station, Program and Project Management, and

Safety and Mission Assurance. Further, we included only performance targets that NASA

intended, at the time of our audit, to report as being achieved or exceeded. We also included

targets that were similar to those we identified in our audit of FY 1999 targets as not having

reliable supporting data. We reviewed 23 performance targets that, in our opinion, met all of

the above criteria. Appendix C provides details on the performance targets we reviewed.

Although we did not use statistical sampling procedures, we considered the selected targets

reasonably representative of all the targets included in NASA's Performance Plan.

To accomplish our objectives we:

• Reviewed GPRA legislation, OMB guidance, and related documentation relative to

measuring and reporting performance results.

• Obtained and reviewed, for the selected performance targets, the measured data and

information supporting the results that were planned to be included in NASA's FY 2000

Performance Report.

• Interviewed NASA personnel and others who had a role either in collecting and

providing the data and information used to measure results or in summarizing and

reporting the results.

• Determined, through interviews and reviews of readily available studies or analyses,

whether there were known major problems with the systems or sources of the

performance data.
10

Appendix A

Management Controls Reviewed

We reviewed the following controls with respect to measuring and reporting performance:

• NASA FY 2000 Performance Plan

• NASA Strategic Management Handbook (February 2000)

• Office of the Chief Financial Officer, Office Work Instruction, “Performance Plan

Update & Reporting,” HOWI7410-B003

• OMB Circular A-11, “Preparing and Submitting Budget Estimates” (July 12, 1999, and

July 19, 2000)

• FY 2000 Performance Report and FY 2000 Accountability Report Data Call Letter

from the Chief Financial Officer to the Officials-in-Charge of Headquarters and

Functional Offices (September 26, 2000)

Management controls for validating and verifying the reliability of GPRA-related performance

data and the reported results are not adequate as discussed in the finding.

Audit Field Work

We conducted field work from November 2000 through February 2001 at NASA

Headquarters and Johnson Space Center. We performed the audit in accordance with

generally accepted government auditing standards.
11

Appendix B. Summary of Prior Audit Coverage

NASA Office of Inspector General

“NASA Implementation of the Government Performance Results Act,” Report

Number IG-99-055, September 28, 1999. The report states that NASA (1) had not made a

timely assessment of progress in achieving FY 1999 performance goals and (2) had not

established formal procedures to ensure that all the data and information used to evaluate

progress and report final results are accurate and reliable. The report contains three

recommendations to assist NASA in addressing and correcting these issues. Management

concurred with all recommendations.

“Validating FY 1999 Performance Data To Be Reported Under the Government

Performance Results Act,” Report Number IG-00-020, March 28, 2000. The OIG

reviewed the accuracy and reliability of performance data for 23 performance targets to be

reported in NASA's FY 1999 Performance Report. We concluded that the supporting data

and information on 18 of 23 performance targets reviewed were adequate, and we did not

identify any significant problems with reported actual performance for those targets. However,

the reported performance on five targets reviewed was not fully reliable because the supporting

data did not adequately support the results described. The report contains three

recommendations to improve the reliability of reported performance. Management concurred

with all recommendations.

General Accounting Office

“Observations on the National Aeronautics and Space Administration's Fiscal Year

2000 Performance Plan,” July 20, 1999, GAO-NSIAD-99-186R. Pursuant to a

congressional request, GAO reviewed NASA's plan with a focus on (1) assessing the

usefulness of the Agency's plan for decisionmaking and (2) identifying the degree of

improvement the Agency's FY 2000 Performance Plan represents over the FY 1999 Plan.

GAO determined that the Agency's plan should be useful to decisionmakers. It provides a

limited picture of intended performance across the Agency, a general discussion of strategies

and resources the Agency will use to achieve its goal, and limited confidence that performance

information will be credible. NASA's FY 2000 Plan represents a moderate improvement over

the FY 1999 Plan in that it indicates some degree of progress in addressing the weaknesses

identified in GAO's assessment of the FY 1999 Plan.
12

 Appendix C. Performance Targets Reviewed in Detail

Target

Number* Target Description as Stated in the FY 2000 Performance Plan

0C11 Support no less than 800 portable exhibit loans and send portable exhibits to a

minimum of 175 targeted events per year.

0GK13 To implement flight missions, the Earth Science Enterprise will successfully launch

one spacecraft and deliver two instruments for international launches, within 10% of

budget on average.

0H1 Support an expanded research program of approximately 935 investigations, an

increase of ~17% over FY99. Publish 100% of science research progress in the

annual OLMSA [Office of Life and Microgravity Science and Applications] Life

Sciences and Microgravity Research Program Task Bibliographies and make this

available on the Internet.

0H13 Achieve 85% on-time, successful launches, excluding weather risk.

0H26 Develop medical protocols and test the capability of the Crew Health Care System

as integrated in the ISS [International Space Station] U.S. Laboratory.

0H43 Reduce the space communications budget submit for FY00 by 30-35% from the

FY96 congressional budget submit.

0MS3 Reduce the number of Agency lost workdays (from occupational injury or illness)

by 5% from the FY94-96 3-year average.

0MS4 Cost 70% or more of available resources.

0MS5 Of funds available for PBC [Performance Based Contract], maintain PBC

obligations at 80% (funds available exclude grants, cooperative agreements, actions

<$100,000, SBIR [Small Business Innovation Research], STTR [Small Business

Technology Transfer], FFRDC's [Federally Funded Research and Development

Centers], intergovernmental agreements, and contracts with foreign governments or

international organizations.

0MS8 Achieve at least the congressionally mandated 8% goal for annual funding to small

disadvantaged businesses (including prime and subcontracts, small disadvantaged

businesses, HBCU's [Historically Black Colleges and Universities], other minority

institutions, and women-owned small businesses).

0P2 Ensure the availability of NASA's spacecraft and ground facilities by decreasing the

FY99 unscheduled downtime.

0P7 Increase the amount of leveraging of the technology budget with activities of other

organization, relative to the FY99 baseline that is established during the process

development

0R10 Complete NSTAR [NASA Solar Electrical Propulsion Technology Application

Readiness] Mission Profile (100% design life) ground testing for Deep Space-1

(concurrent, identical firing of an NSTAR engine in a vacuum chamber with the

actual firing sequence of the in-flight propulsion system).

* See the Legend at the end of the table.
13

Appendix C

0R14 Achieve a facility utilization customer satisfaction rating of 95 percent of

respondents at “5” or better and 80 percent at “8” or better, based on exit

interviews.

0R15 Transfer at least 12 new technologies and processes to industry during the fiscal

year.

0S6 Prepare the INTEGRAL Science Data Center (ISDC) for data archiving and

prepare instrument analysis software for the Spectrometer on INTEGRAL (SPI)

instrument within 10% of estimated cost.

0S16 NEAR [Near Earth Asteroid Rendezvous] will successfully orbit 433 Eros and

meet primary scientific objectives while not exceeding projected mission cost by

more than 10%.

0S49 Information systems R&T [Research and Technology] will demonstrate the search,

discovery, and fusion of multiple data products at a major science meeting.

Accomplish and document the infusion of five information systems R&T efforts into

flight projects or the broad research community. Space science data services shall

be acknowledged as enabling for two interdisciplinary collaborations.

0S67 Successful achievement of at least seven of the following eight objectives will be

made. (1) Each new Space Science mission will have a funded education and

outreach program. (2) By the end of FY00, 10% of all Space Science research

grants will have an associated education and outreach program under way. (3)

Twenty-six states will have Enterprise-funded education or outreach programs

planned or under way. (4) At least five research, mission development/operations,

or education programs will have been planned/undertaken in Historically Black

Colleges and Universities, Hispanic Serving Institutions, or Tribal Colleges, with at

least one project under way in each group. (5) At least three national and two

regional educational or outreach conferences will be supported with a significant

Space Science presence. (6) At least three exhibits or planetariums shows will be

on display. (7) An online directory providing enhanced access to major Space

Science-related products and programs will be operational by end of the fiscal year.

(8) A comprehensive approach to assessing the effectiveness and impact of the

Space Science education and outreach efforts will be under development, with a

pilot test of the evaluation initiated.

0Y9 Establish a benchmark for global and regional rainfall measurements by combining

TRMM [Tropical Rainfall Measuring Mission] measurements with measurements

from other sources. Create maps of the diurnal cycle of precipitation for the first

time. Combine the existing 10-year data set with TRMM measurements to validate

climate models and demonstrate the impact of rainfall on short-term weather

forecasting. Distribute through the Goddard DAAC [Distributed Active Archive

Center] for ease of access to science and operational users.

Appendix C
14

0Y10 Develop/improve methods to couple state-of-art-land surface and sea ice models to

a global coupled ocean-atmosphere model and use to predict regional climatic

consequences of Él Niño or La Niña occurrence in the tropical Pacific. Results of

research will be published in the open literature and provided to NOAA's [National

Oceanic and Atmospheric Administration] National Climate Prediction Center and

the U.S. Navy's Fleet Numeric Prediction Center. Ultimate goal: develop a

capability to significantly improve the prediction of seasonal-to-interannual climate

variations and their regional climate consequences. The main focus is on North

America.

0Y43 Implement at least five joint applications research projects/partnerships with State

and local governments in remote-sensing applications.

0Y44 Focus EOCAP [Earth Observing Commercialization Applications Program] joint

commercial applications research to develop 20 new market commercial products,

(e.g., oil spill containment software by EarthSat and map sheets products by

ERDAS, Inc.)

* Legend:

0 refers to FY 2000.

C refers to the Communicate Knowledge Crosscutting Process.

GK refers to the Generate Knowledge Crosscutting Process.

H refers to the Human Exploration and Development of Space Enterprise.

MS refers to the Manage Strategically Crosscutting Process.

P refers to the Provide Aerospace Products and Capabilities Crosscutting Process.

R refers to the Aerospace Technology Enterprise.

S refers to the Space Science Enterprise.

Y refers to the Earth Science Enterprise.
15

Appendix D. Management’s Response
16

Appendix D
17

Appendix D
18

Appendix D
19

Appendix D
20

Appendix E. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters

A/Administrator

AA/Chief of Staff

AI/Associate Deputy Administrator

AE/Chief Engineer

AS/Chief Scientist

B/Acting Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

BR/Acting Director, Resources Analysis Division

G/General Counsel

H/Associate Administrator for Procurement

HK/Director, Contract Management Division

HS/Director, Program Operations Division

J/Associate Administrator for Management Systems

JM/ Director, Management Assessment Division

L/Acting Associate Administrator for Legislative Affairs

M/Associate Administrator for Space Flight

Q/Associate Administrator for Safety and Mission Assurance

R/Associate Administrator for Aerospace Technology

S/Associate Administrator for Space Science

U/Acting Associate Administrator for Biological and Physical Research

Y/Associate Administrator for Earth Science

Z/Acting Associate Administrator for Policy and Plans

NASA Centers

Director, Ames Research Center

Director, Dryden Flight Research Center

Director, John H. Glenn Research Center at Lewis Field

Director, Goddard Space Flight Center

Director, Jet Propulsion Laboratory

Acting Director, Lyndon B. Johnson Space Center

Director, John F. Kennedy Space Center

Chief Counsel, John F. Kennedy Space Center

Director, Langley Research Center

Director, George C. Marshall Space Flight Center

Acting Director, John C. Stennis Space Center
21

Appendix E

Non-NASA Federal Organizations and Individuals

Assistant to the President for Science and Technology Policy

Deputy Associate Director, Energy and Science Division, Office of Management and

 Budget

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office

 of Management and Budget

Director, Acquisition and Sourcing Management Team, General Accounting Office

Professional Staff Member, Senate Subcommittee on Science, Technology, and Space

Chairman and Ranking Minority Member – Congressional Committees and

Subcommittees

Senate Committee on Appropriations

Senate Subcommittee on VA, HUD, and Independent Agencies

Senate Committee on Commerce, Science, and Transportation

Senate Subcommittee on Science, Technology, and Space

Senate Committee on Governmental Affairs

House Committee on Appropriations

House Subcommittee on VA, HUD, and Independent Agencies

House Committee on Government Reform and Oversight

House Subcommittee on Government Efficiency, Financial Management, and

 Intergovernmental Relations

House Subcommittee on National Security, Veterans Affairs, and International Relations

House Subcommittee on Technology and Procurement Policy

House Committee on Science

House Subcommittee on Space and Aeronautics, Committee on Science

Congressional Member

Honorable Pete Sessions, U.S. House of Representatives
NASA Assistant Inspector General for Auditing

Reader Survey

The NASA Office of Inspector General has a continuing interest in improving the usefulness of

our reports. We wish to make our reports responsive to our customers’ interests, consistent

with our statutory responsibility. Could you help us by completing our reader survey? For your

convenience, the questionnaire can be completed electronically through our homepage at

http://www.hq.nasa.gov/office/oig/hq/audits.html or can be mailed to the Assistant Inspector

General for Auditing; NASA Headquarters, Code W, Washington, DC 20546-0001.

Report Title: Validation and Verification of Selected NASA FY 2000 Performance

Data Related to the Government Performance and Results Act

(GRPA)

Report Number: Report Date: 

Circle the appropriate rating for the following statements.

Strongl

y

Agree

Agree Neutra

l

Disagre

e

Strongl

y

Disagre

e

N/A

1. The report was clear, readable, and logically

organized.

5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A

3. We effectively communicated the audit

objectives, scope, and methodology.

5 4 3 2 1 N/A

4. The report contained sufficient information to

support the finding(s) in a balanced and

objective manner.

5 4 3 2 1 N/A

Overall, how would you rate the report?

Excellent Fair

Very Good Poor

Good

If you have any additional comments or wish to elaborate on any of the

above responses, please write them here. Use additional paper if necessary.

 

 

 

 

 

 
How did you use the report? 

 

 

 

 

 

 

How could we improve our report? 

 

 

 

 

 

 

How would you identify yourself? (Select one)

Congressional Staff Media

NASA Employee Public Interest

Private Citizen Other: 

Government: Federal: State: Local: 

May we contact you about your comments?

Yes: ______ No: ______

Name: ____________________________

Telephone: ________________________

Thank you for your cooperation in completing this survey.
Major Contributors to the Report

Chester A. Sipsock, Program Director, Environmental and Financial Management Audits

Carol A. St. Armand, Audit Program Manager

Mary S. Anderson, Auditor

Ronald T. Callahan, Auditor

Nancy C. Cipolla, Report Process Manager

Annette Huffman, Program Assistant