IG-02-025

AUDIT 

REPORT 

VALIDATION AND VERIFICATION OF 

SELECTED NASA FISCAL YEAR 2001 

PERFORMANCE DATA RELATED TO THE 

GOVERNMENT PERFORMANCE AND 

RESULTS ACT 

September 27, 2002 

 National Aeronautics and 

Space Administration

OFFICE OF INSPECTOR GENERAL
Additional Copies 

To obtain additional copies of this report, contact the Assistant Inspector General for 

Audits at (202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General 

for Audits. Ideas and requests can also be mailed to: 

Assistant Inspector General for Audits 

 Code W

 NASA Headquarters 

Washington, DC 20546-0001 

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800) 

424-9183, (800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form; 

or write to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, 

Washington, DC 20026. The identity of each writer and caller can be kept confidential, 

upon request, to the extent permitted by law. 

Reader Survey 

Please complete the reader survey at the end of this report or at 

www.hq.nasa.gov/office/oig/hq/audits.html. 

Acronyms 

APG Annual Performance Goal 

CFO Chief Financial Officer 

DAAC Distributed Active Archive Centers 

EOCAP Earth Observing Commercialization Applications Program

ESIP Earth Science Information Partners 

EVA Extravehicular Activity 

FY Fiscal Year 

GAO General Accounting Office 

GPRA Government Performance and Results Act 

HEDS Human Exploration and Development of Space 

ISS International Space Station 

IT Information Technology 

OIG Office of Inspector General 

OMB Office of Management and Budget 

PAPAC Provide Aerospace Products and Capabilities 
September 27, 2002 

W 

TO: B/Deputy Chief Financial Officer for Financial Management 

FROM: W/Assistant Inspector General for Audits 

SUBJECT: Final Report on Audit of Validation and Verification of Selected NASA 

Fiscal Year 2001 Performance Data Related to the Government 

Performance and Results Act 

 Assignment Number A-02-013-00 

Report Number IG-02-025 

Enclosed please find the subject final report for your information and use. Please refer to 

the Executive Summary for the overall audit results. Our evaluation of your response has 

been incorporated into the body of the report. NASA's comments and planned corrective 

actions for the recommendations were responsive. The recommendations will remain 

open for reporting purposes until corrective actions are completed. Please notify us when 

actions have been completed on the recommendations. The final report distribution is in 

Appendix E. 

We appreciate the courtesies extended to the audit staff. If you have questions 

concerning the report, please contact Mr. Chester A. Sipsock, Program Director, 

Financial Audits, Management and Oversight, at (216) 433-8960, or 

Ms. Bonnie Armstrong, Auditor-in-Charge, at (321) 867-4073. 

[original signed by] 

Alan J. Lamoreaux 

Enclosure 
2 

cc: 

HQ/AI/Associate Deputy Administrator 

HQ/AB/Associate Deputy Administrator for Institutions and Asset Management 

HQ/AA/Chief of Staff 

HQ/AE/Chief Engineer 

HQ/AO/Acting Chief Information Officer 

HQ/B/Deputy Chief Financial Officer for Resources (Comptroller) 

HQ/BF/Director, Financial Management Division 

HQ/BR/Director, Resources Analysis Division 

HQ/G/General Counsel 

HQ/M/Associate Administrator for Space Flight 

HQ/R/Associate Administrator for Aerospace Technology 

HQ/S/Associate Administrator for Space Science 

HQ/U/Associate Administrator for Biological and Physical Research 

HQ/Y/Associate Administrator for Earth Science 

HQ/JM/Director, Management Assessment Division 
Contents 

Executive Summary, i 

Introduction, 1 

Finding and Recommendations, 3 

Data Reliability and Presentation, 3

Appendix A - Objectives, Scope, and Methodology, 12

Appendix B - Annual Performance Goals Reviewed in Detail, 15 

Appendix C - Summary of Prior Audit Coverage, 22 

Appendix D - Management's Response, 25 

Appendix E - Report Distribution, 26

 
NASA Office of Inspector General 

IG-02-025 September 27, 2002 

 A-02-013-00 

Validation and Verification of Selected NASA Fiscal Year 2001 

Performance Data Related to the Government 

Performance and Results Act 

Executive Summary 

Background. The NASA Office of Inspector General (OIG) has completed an audit of 

the accuracy and reliability of performance data for selected Government Performance 

and Results Act (GPRA) annual performance goals (APG's)

1

 in the Agency's fiscal year 

(FY) 2001 Performance Report. The Performance Report is an important document that 

NASA, the Congress, and the Office of Management and Budget (OMB) will use to 

assess the Agency's overall performance and make decisions on programs and funding 

levels. The NASA Chief Financial Officer (CFO) obtains and aggregates performance 

results from NASA organizations and prepares the Performance Report. The CFO 

requests input from Agency organizations with a data call letter that specifies the 

requirements and format for performance results. The audit is a continuation of our 

oversight of NASA's implementation of GPRA.2,3

 

Objectives. Our overall audit objective was to assess the quality of data supporting the 

reported results in the NASA FY 2001 Performance Report. We assessed the quality of 

data by examining supporting data for selected APG's for appropriateness, completeness, 

accuracy, consistency, and timeliness.4

 The Performance Report contains NASA's 

assessment of its actual performance against 88 APG's. We reviewed the supporting data 

for 19 APG's related to 7 of 14 major management challenges that the General 

Accounting Office (GAO) and the NASA OIG identified: environmental management, 

fiscal management, information security, information technology, program and project 

management, safety and mission assurance, and human capital management. Appendix 

A contains further details on the audit objectives, scope, and methodology. Appendix B 

provides details on the 19 APG's reviewed.

 

1

 Performance goal means a target level of performance expressed as a tangible, measurable objective 

against which actual achievement can be compared. A goal expressed as a quantitative standard, value, or

rate is a performance goal. 2

 An October 1998 letter signed by the House Majority Leader and Chairmen of the House Committee on 

Government Reform and Oversight; the House Subcommittee on Government Management, Information, 

and Technology; and the Results Caucus asked the NASA OIG to establish a GPRA review plan to assess 

Agency controls. In response to the request, the OIG included a plan in its Semiannual Reports for 

March 31, 1999, and 2000, and described the GPRA audit in its FY 2001 annual plan. 3

 The NASA OIG has issued reports on previous audits of NASA's implementation of GPRA. Details on 

the audits are in Appendix C. 4

 Timeliness refers to whether performance results occurred during FY 2001, that is, October 1, 2000, 

through September 30, 2001. 

 
Results of Audit. For 12 (63 percent) of the 19 APG's reviewed, we considered the 

supporting data and information to be adequate and did not identify any significant 

problems with reported actual performance. However, for seven APG's, we found that 

either the initially reported performance5

 was not fully reliable or the presentation of 

results was unclear. For example, three Enterprises6

 initially reported accomplishment of 

projects that supporting data showed either were not completed or were completed 

outside the performance period. In addition, NASA did not always disclose limitations in 

the supporting data that were needed to understand the basis for reported performance 

results. Based on the current finding and similar results from previous GPRA audits, it is 

possible that the reported performance for some of the 69 APG's not reviewed may also 

not be fully reliable or clearly presented for the same reasons. NASA could improve the 

accuracy of future Performance Reports by more effectively analyzing the supporting 

data and by clearly and precisely presenting results. Improved accuracy would increase 

the Performance Report’s value as a source of information to management for making 

important program and funding decisions. 

Recommendations. The Deputy CFO for Financial Management should emphasize in 

the data call letter for subsequent Performance Reports that reported performance results 

must accurately reflect supporting data and must be achieved during the subject fiscal 

year. Additionally, the Deputy CFO for Financial Management should emphasize that 

reported results be aligned with planned performance and that data limitations be 

disclosed. 

Management's Response. Management concurred with all the recommendations and 

has planned corrective actions that should improve the data call letter for the FY 2002 

Performance Report. The complete text of the response is in Appendix D. We consider 

management's comments responsive. 

 

5

 For purposes of our audit and this report, the term “initially reported performance” refers to written selfassessments of actual results prepared by the responsible GPRA officials and provided to the NASA Chief 

Financial Officer (in response to the data call letter) for the draft Performance Report. “Reported results” 

refer to assessments included in the published Performance Report. 6

 NASA's mission is accomplished through five Strategic Enterprises: Space Science, Earth Science, 

Biological and Physical Research, Human Exploration and Development of Space, and Aerospace 

Technology. 

ii 
Introduction 

Congress enacted the GPRA in 1993 to improve public confidence in the Federal 

Government by holding agencies accountable through setting program goals, measuring 

performance against those goals, and reporting publicly on progress. Each agency is 

required to prepare a Strategic Plan, an annual Performance Plan, and an annual 

Performance Report. NASA released its latest Strategic Plan in September 2000.7

 

NASA issued Performance Plans for FY's 1999 through 2003 and Performance Reports 

for FY's 1999 through 2001. During our audit, NASA prepared the Performance Report 

covering FY 2001. 

The Associate Administrators for the Enterprises and the Crosscutting Process 

Stewards8,9

 are responsible for developing and implementing the annual Performance 

Plan and for reporting on actual performance for the annual Performance Report. Four 

Crosscutting Processes that are common to each Enterprise provide key supporting 

functions that enable the Enterprises to perform their mission activities. The 

Crosscutting Process and organization responsible for developing and reporting 

performance results are shown below: 

 

Crosscutting Process Responsible Organization 

Manage Strategically Strategic Management and Planning office 

in the Office of the CFO 

Provide Aerospace Products and 

Capabilities (PAPAC) 

Chief Engineer 

Generate Knowledge Chief Scientist 

Communicate Knowledge Public Affairs 

The CFO coordinates the performance planning and reporting processes, collecting 

information (submitted in response to a data call letter) from the Associate 

Administrators for the Enterprises and the Crosscutting Process Stewards to prepare the 

annual Performance Plan and Performance Report. 

NASA Centers are responsible for implementing many of the programs and activities that 

have GPRA performance goals and indicators.10 Therefore, Center systems were the 

source for much of the data used to measure and evaluate actual performance. The 

 

7

 NASA also released Strategic Plans in 1995, 1996, and 1998. In 1999, NASA issued interim adjustments 

to the 1998 plan. 8

 Crosscutting Processes are critical processes underlying the activities of the Agency that NASA uses to 

develop and deliver products and services to its customers. These processes support systems that enable

each Strategic Enterprise to develop and deliver products and services to internal and external customers. 9

 NASA refers to the responsible GPRA official for each Crosscutting Process as the GPRA Steward. 10 Indicators are the particular values or characteristics used to measure output or outcome. At least one 

indicator was established for each APG. 
Associate Administrators for the Enterprises and the GPRA Stewards collected the data 

from the Centers, developed a written assessment of the actual performance, and 

submitted the assessment to the CFO for use in preparing the Performance Report. 

The GAO and the OIG have assessed NASA's past progress in implementing the GPRA. 

The GAO reported that NASA’s FY 2001 Performance Plan did not include an explicit 

discussion of procedures for verifying and validating performance data and did not 

address possible data limitation issues and problems. Additionally, prior OIG audits 

concluded that performance results were not fully reliable because the supporting data 

did not adequately confirm the results described. Details of the prior GAO and OIG 

audits are in Appendix C. 

2 
Finding and Recommendations 

Data Reliability and Presentation 

For 7 (37 percent) of 19 APG's reviewed, responsible GPRA officials prepared written 

assessments that did not accurately reflect supporting data or did not clearly present 

results. For example, three Enterprises initially reported accomplishment of projects that 

supporting data showed either were not completed or were completed outside the 

performance period. Inaccurate or unclear assessments occurred, in part, because 

individuals responsible for performance did not consistently verify and validate that the 

results accurately reflected supporting data. Confusion about when accomplishments 

were completed, inconsistency between planned and reported performance, and 

nondisclosure of data limitations also contributed to the condition. Because reported 

performance for the seven APG's was not fully reliable or clearly presented, the 

usefulness of the performance data to NASA, OMB, and the Congress for 

decisionmaking may have been limited. Based on our audit results, NASA’s reported 

performance for some of the 69 APG's we did not review may also not be fully reliable or 

clearly presented for the same reasons. 

Management attention is needed to address and correct these problems before issuing 

future Performance Reports. Of the seven APG’s that we identified as unreliable or not 

clearly presented, NASA confirmed achievement of one APG and revised three11 APG's 

prior to the printing of the Performance Report. 

Reporting Requirements 

GPRA Requirements. The GPRA requires an agency to prepare an annual Performance 

Report that compares actual performance with the APG's set out in the annual 

Performance Plan. When an APG is not achieved, the Performance Report should 

include an explanation for the lack of achievement and describe steps for meeting future 

goals. For the annual Performance Report to be useful, the data on the actual 

achievements of the agency's performance goals and the comparisons of planned and 

actual performance must be accurate. GPRA further requires the annual Performance 

Plan to include a description of the means used to verify and validate measured values. 

Additionally, to have accurate measurements of actual performance, it is important that 

the APG's are described in the Plan in a manner to ensure that the planned achievements 

and how they are measured are clear. The CFO issued the annual data call letter to 

NASA organizations requesting input for the FY 2001 Performance Report and required 

 

11 We provided our conclusions to NASA for five APG's (1H7, 1H11, 1MS4, 1R8, and 1Y16) prior to the 

printing of the Performance Report. Management addressed our concerns related to four of those APG's by

providing additional supporting data for one APG (1Y16) and by revising performance results for three 

APG's (1H7, 1H11, and 1R8). NASA agreed to revise performance results for APG 1MS4, but the 

revision was not included in the Performance Report. We completed our review of two APG's (1H18 and 

1Y2) subsequent to the print date for the Performance Report and could not notify NASA of our concerns 

in time to suggest revisions to the Performance Report. Please refer to Appendix A for further details. 

3
that performance results be understandable to a broad audience. The request also 

required Enterprises and Crosscutting Processes to discuss any data limitations that they 

had experienced when compiling performance results. 

OMB Guidance. OMB Circular A-11, "Preparation and Submission of Budget 

Estimates," describes requirements of the GPRA and guides agencies in preparing and 

submitting strategic plans, annual performance plans, and annual performance reports. 

Circular A-11, Section 232.4, “Comparing actual performance to the performance goal 

target levels,” requires that actual performance be reported as it occurred during the fiscal 

year covered by the Performance Report. Section 232.5, "Unavailability of actual 

performance information," requires that the annual performance report identify those 

performance goals for which actual performance information is missing, incomplete, or 

preliminary. Section 232.10, “Assessing the completeness and reliability of performance 

data,” considers performance data complete if actual performance is reported for every 

performance goal and indicator in the annual plan and if the agency identifies in the 

report any performance goals and indicators for which actual performance data are not 

available. 

NASA Policies and Procedures. NASA's FY 2001 Revised Final Annual Performance 

Plan12 described the means by which the Agency verifies and validates its performance 

data. Performance is evaluated at the Agency, Enterprise, functional office, program and 

project, and Crosscutting Process levels. Each level is responsible to execute 

requirements and to measure, evaluate, and report results. Program managers are 

responsible for data collection and reporting. NASA relies on the individuals responsible 

for performance to verify and validate results. For purposes of assessing overall 

performance, NASA asks Advisory Committees13 to evaluate accomplishments at the 

levels of the Enterprise and Crosscutting Process objectives and goals. The NASA 

Strategic Management Handbook explains that NASA uses regular management insight 

and review processes as well as external reviews to assess its performance. Internally, 

the Program Management Council14

 assesses program schedules, cost, and technical 

performance against established programmatic commitments, and the NASA Advisory 

Council15 provides advice on programs and issues. Externally, researchers 

 

12 An agency prepares at least two iterations of its annual plan -- an initial plan consistent with the 

agency's budget request to OMB and a final plan that is consistent with the President's budget. In addition, 

agencies may prepare a third iteration, called a revised final plan, which reflects congressional action on 

the agency's budget request. 13 Several Advisory Committees have been established under the NASA Advisory Council (see 

footnote 15) to advise NASA programs. Advisory Committees have been established for Space Flight, 

Aerospace Technology, Earth System Science and Applications, Biological and Physical Research, 

Minority Business Resource, and Space Science. 

14 The Program Management Council, chaired by the Associate Deputy Administrator, provides advice, 

counsel, and recommendations for consideration by the Administrator relating to planning, implementation, 

and management of all major Agency programs. 15 The NASA Advisory Council advises the NASA Administrator on Agency programs, policies, plans, 

and other matters pertinent to the Agency's responsibilities. 

4
and other organizations assess NASA's progress in meeting its annual performance goals. 

Other external groups involved with verifying and validating performance data include 

the OIG and GAO. 

Verifying and Validating Supporting Data and Results 

For 12 of the 19 APG's included in NASA's FY 2001 Performance Report, we did not 

find any significant problems with the actual performance reported by the Associate 

Administrators for the Enterprises and the GPRA Stewards. Except for minor 

differences, the supporting data and manner in which the actual results were reported 

were generally adequate. NASA could further improve the overall process for verifying 

and validating future GPRA performance data and reported results as evidenced by the 

seven APG's discussed below: 

APG 1H7: "Achieve 8 or fewer in-flight anomalies16 per mission." The literal 

interpretation of this APG is that NASA could not meet this goal if the Agency 

experienced more than eight anomalies on any Space Shuttle mission performed in 

FY 2001. Yet, the Human Exploration and Development of Space (HEDS) Enterprise 

initially reported that it had met this goal on the basis that an average of 4.57 anomalies 

occurred over a span of 7 missions. On 1 of the 7 missions, however, NASA experienced 

12 in-flight anomalies. We concluded that the HEDS Enterprise should have reported 

that NASA had not met this goal. We conveyed to HEDS officials our concern that 

reporting the average instead of the actual number of anomalies did not match planned 

performance. HEDS officials stated that the APG presented in the Performance Plan 

should have reflected average in-flight anomalies. The officials explained that due to the 

complexity of the multitude of systems onboard the Shuttle, missions sometimes 

experience more in-flight anomalies than the goal, and the intent was for the indicator to 

reflect the average, rather than actual, number of anomalies. NASA should have clearly 

specified in the Performance Plan and the Performance Report its intention to use an 

average as a measure of performance. 

To ensure clear presentation of results, we requested that the HEDS Enterprise explain in 

the Performance Report why it was appropriate to use the average number of anomalies. 

Prior to issuing the Agency’s final Performance Report, the Enterprise revised the 

initially reported results to explain that the Shuttle program has been managed to the 

average number of in-flight anomalies for years and to point out that reporting both 

individual and average mission results is consistent with past and future performance. 

Our review of supporting data and discussions with HEDS officials concluded that the 

goal was achieved and that the results presented in the published Performance Report had 

been adequately clarified. 

 

16 In-flight anomalies are deviations from expectations that occur during Space Shuttle missions. 

5
APG 1H11: "[S]uccessfully complete the majority of the ISS [International Space 

Station] planned on-orbit activities such as delivery of mass to orbit and enhanced 

functionality." To measure achievement of APG 1H11, the HEDS Enterprise 

established two indicators. The first indicator required launch and delivery of 180,000 

pounds of hardware and logistics to the ISS. The Enterprise exceeded the requirement by 

delivering 240,000 pounds of hardware and logistics. 

The second indicator required initiation and demonstration of ISS Extravehicular Activity 

(EVA) capability to support up to 30 EVA's annually. The second indicator was to be 

measured by completion of five EVA's from the ISS Airlock. The HEDS Enterprise 

initially intended to report that NASA had achieved this APG. HEDS officials explained 

that when the FY 2001 Performance Plan was developed, five EVA's were planned for 

the fiscal year. However, after development of the indicator, the ISS mission planning 

organization had scheduled only two EVA’s from the ISS during FY 2001. Our review 

determined that initially reported results did not accurately reflect supporting data, 

because only two EVA's were completed in FY 2001, while three were completed in 

FY 2002, within 6 weeks of the end of FY 2001. NASA’s intention of reporting this goal 

as being met is contrary to OMB Circular A-11, which requires that agencies report 

actual performance as it occurred during the fiscal year covered by the Performance 

Report. In response to our concerns, the HEDS Enterprise changed its assessment and 

reported that, although the goal was not achieved, progress was significant, and the goal 

would be achieved the following year. 

APG 1H18: "Demonstrate, in ground test, at least one technology that could reduce 

up to 25% of life support logistics over ISS baseline and release progress report for 

review on the Internet." To measure achievement of APG 1H18, the Biological and 

Physical Research Enterprise established two indicators. The first indicator just repeated 

the APG. The second indicator required performing a detailed calculation of life-support 

equivalent system mass17 index and placing the calculation on the Internet for review and 

comment. The second indicator explained that the equivalent system mass index is a 

measure of the performance of a life-support system that incorporates demonstrated 

technologies. The Enterprise reported that the goal was achieved. However, NASA did 

not place the progress report containing the calculation on the Internet until 

January 31, 2002, 4 months after the end of FY 2001. 

We concluded that reported performance results did not accurately reflect supporting data 

because the progress report was not completed during FY 2001. Although the APG did 

not specify when the progress report would be released, OMB guidance requires that 

actual performance be reported as it occurred during the fiscal year covered by the 

Performance Report. Therefore, to report the APG as achieved, the progress report 

would have to be released during FY 2001. The progress report was not released until 

FY 2002, and the FY 2001 Performance Report incorrectly states that APG 1H18 was 

achieved. We completed our review of this APG after the Performance Report was 

 

17 Equivalent system mass is the sum of the masses of life-support equipment and supplied commodities. 

6
submitted for printing. Therefore, the Biological and Physical Research Enterprise was 

not aware of our conclusion and did not have the opportunity to clarify reported 

performance results before NASA issued the Performance Report. 

APG 1MS4: "Improve Information Technology (IT) infrastructure service delivery 

to provide increased capability and efficiency while maintaining a customer rating 

of satisfactory, and enhance IT security through reduction of system vulnerabilities 

across all NASA Centers, emphasizing IT security awareness training for all NASA 

personnel by meeting 2 our of 2 performance indicators in this area." The NASA 

Chief Information Officer established two indicators to measure accomplishment of the 

APG. The first indicator required NASA to improve IT infrastructure service delivery to 

provide increased capability and efficiency while maintaining a customer rating of

"satisfactory" and holding costs per resource unit to the FY 1998 baseline. We reviewed 

the supporting data and agreed that NASA met the first indicator. 

The second indicator required NASA to enhance IT security through a reduction of 

system vulnerabilities across all NASA Centers and through emphasis on IT security 

awareness training for all Agency personnel. The NASA Chief Information Officer 

reported the indicator as achieved but did not disclose data limitations in the Performance 

Report. 

NASA used part of the supporting data for the second indicator in a report that the 

Agency submitted to OMB under the Government Information Security Reform Act 

(Security Act).18 NASA identified several limitations of the supporting data in the report. 

However, in its initially reported performance results, NASA had not planned to report 

any limitations with the supporting data. NASA intended to report that it had met the 

second indicator by identifying and reducing IT system vulnerabilities19 and by providing 

IT security awareness training to NASA civil service employees, civil service managers, 

and civil service system administrators. In a previous audit, we found that the supporting 

data was limited in scope.20 Specifically, we found the following: 

• The Centers did not perform consistent scans of Center IT systems for 

vulnerabilities, and NASA did not make complete use of all available scanning 

capability. As a result, the data from the vulnerability scans was limited in scope 

and did not accurately estimate the vulnerability of NASA's IT systems. 

• The supporting data for providing IT security awareness training to civil service 

system administrators did not include contractor system administrators, who 

 

18 The Security Act, Public Law 106-398, requires an agency to report to OMB the measures of 

performance used to ensure that agency officials are fulfilling their security responsibilities and a 

description of the actual level of agency performance in implementing its security requirements. 

19 A vulnerability is a weakness in an IT system that can be exploited to compromise or violate security 

processes or controls. If a system is vulnerable to a threat, that vulnerability represents a risk to the system. 20 The OIG issued Report Number IG-02-003, “Performance Management Related to Agencywide 

Information Technology Security Goals,” November 19, 2001. We performed the audit at the same time 

that NASA submitted one report required by the Security Act to OMB. 

7
comprise 79 percent of NASA's system administrator workforce. As a result, 

NASA was not ensuring that contractor system administrators received the same

training as their civil service counterparts. Untrained system administrators 

become an unnecessary weak link in NASA's IT security program. 

We discussed the discrepancy between the initially reported performance results and the 

Security Act report with NASA Chief Information Officer officials. They agreed that 

data limitations described in the Security Act report should also be described in the 

Performance Report and told us that they would revise the document. However, the 

revisions were not included in the published Performance Report.21 As a result, the 

Performance Report is inconsistent with the report that NASA submitted to OMB under 

the Security Act. 

APG 1R8: "Develop at least three new design tools, accomplish at least four 

demonstrations of advances in computation and communications, and complete the 

intelligent synthesis environment proof-of-concept systems capability build to 

technology readiness level 3: indicators include computer testbed demonstrations, 

real-time remote access of data, new design methods and an intelligent synthesis 

environment proof-of-concept system." To measure achievement of the goal, the 

Aerospace Technology Enterprise established 21 indicators.22 The Enterprise reported 

the goal as achieved. However, our audit concluded that supporting data did not confirm

initially reported performance, and results were not clearly presented. Specifically, 

supporting data suggested that initial performance results were not achieved during 

FY 2001, and performance results were not provided for each indicator. The problems 

occurred because individuals did not agree on the dates of accomplishment for projects 

and because of the complexity of the APG and its indicators.23 In response to our 

conclusions and notification to the Enterprise, it eliminated projects not completed during 

FY 2001 from the Performance Report and replaced them with other projects completed. 

Although the Enterprise achieved the APG, the presentation for APG 1R8 in the 

Performance Report may not be easily understood by a broad audience. 

We asked the Enterprise for supporting data to determine whether initially reported 

performance accurately reflected the data. Supporting data included quarterly status 

reports, publications, and presentations. The Enterprise provided us a list of publications 

and presentations indicating that three projects initially reported as achieved during 

FY 2001 may have been achieved in FY 2000. When we notified the Enterprise of our 

concern about the projects' completion dates, the Enterprise eliminated the projects from

its initially reported performance. Enterprise officials should verify and validate actual 

 

21 The Office of the Chief Information Officer revised its submission to the CFO in February 2002. 

However, the revision was not incorporated into a draft of the Performance Report due to an oversight. 

When that office realized that the revision had not been incorporated, the Performance Report had already

been submitted to the printer and could not be changed. 22 The 21 indicators are shown in Appendix B. 23 The Enterprise has simplified the APG in the FY 2003 Performance Plan by reducing the number of 

indicators from 21 to 4. 

8
performance to ensure that reported results accurately reflect supporting data and that 

only projects accomplished during the reporting period are reported as achieved. 

We had difficulty comparing initially reported performance to planned performance 

because the initially reported performance did not align with the APG or individual 

indicators described in the Performance Plan. For example, instead of results presented 

as three new design tools, four demonstrations of advances, and completion of the 

intelligent synthesis environment to match the APG stated in the Performance Plan, 

results were stated as four achievements, tools, and demonstrations; four presentations, 

development of tools and services, and establishment of methodology; and development 

of the intelligent synthesis environment. Additionally, reported results did not address 

the 21 indicators or specify whether individual indicators were achieved. We were able 

to correlate results with indicators only after several discussions with the Enterprise 

representative and after reviewing documentation that explained relationships between 

planned performance and results. Because reported results were not aligned with the 

APG and did not provide assessments for indicators, readers could not compare actual 

achievements to planned accomplishments. 

APG 1Y2: "Successfully disseminate Earth Science data to enable our science 

research and applications goals and objectives by meeting all performance 

indicators in this research area." The Earth Science Enterprise established six 

indicators to measure performance of the goal. The indicators require prompt data 

availability, decreased number of order errors, increased volume of data archived, 

increased number of customers, increased number of products, and increased customer 

satisfaction. The Enterprise reported that all six indicators were achieved, but did not 

disclose a data limitation in the Performance Report. 

The dissemination of Earth Science data is accomplished through the Earth Observing 

System Data and Information System. NASA-funded researchers provide the data to be 

disseminated. Two research organizations, the Earth Science Information Partners 

(ESIP) and Distributed Active Archive Centers (DAAC), provided statistics validating 

initially reported performance. Statistics from the DAAC appropriately included data for 

the period October 1, 2000, through September 30, 2001 (FY 2001). However, statistics 

from ESIP for the fourth quarter of FY 2001 (July 1 through September 30) were not 

available at the time NASA prepared the Performance Report. To provide 12 months of 

statistics for assessment, the Enterprise changed the period of performance for ESIP to 

include statistics from the third quarter of FY 2000. The Enterprise used FY 2000 

statistics to assess performance for four indicators requiring prompt data availability, 

increased volume of data archived, increased number of customers, and increased 

products. Earth Science representatives explained that similar results are experienced 

each quarter and that using data from the third quarter of FY 2000 instead of the fourth 

quarter of FY 2001 would have little effect on overall achievement. 

We concluded that statistical differences between quarters were insignificant and that the 

Earth Science Enterprise correctly reported the goal as achieved. Nonetheless, the 

Performance Report should have disclosed the data limitation, explaining that only 

preliminary statistical data was available for FY 2001 and that statistics from the third 

9
quarter of FY 2000 were substituted. We completed our review of this APG after NASA 

submitted the Performance Report to a contractor for printing. Consequently, the Earth 

Science Enterprise was not aware of our conclusion and did not have the opportunity to 

correct reported results before NASA issued the Performance Report. 

APG 1Y16: "Stimulate the development of a robust commercial remote sensing 

industry by meeting at least 4 of 5 performance indicators in this area." The Earth 

Science Enterprise established five indicators to measure achievement of the goal. The 

Enterprise reported that the goal was met because four of the five indicators had been 

achieved. In reviewing original supporting data, we could not confirm that one of the 

four indicators had been achieved. If one of the four indicators was not achieved, the 

goal should have been reported as not achieved, because only three of five indicators 

were met. We expressed our concerns to the Enterprise official, who provided additional 

supporting data showing that the indicator had been achieved and exceeded. 

The Earth Science Enterprise initially reported results for one indicator that did not 

accurately reflect supporting data. The indicator required development of 10 new market 

commercial products in joint commercial applications research projects. As evidence of 

achieving the indicator, the Enterprise provided a list of 10 products developed during 

FY 2001 and the names of company contacts who could confirm development of the 

products. Our confirmation request to one company contact showed that two products 

that the Enterprise initially reported as developed during FY 2001 were never completed. 

When we notified the Enterprise that we could not confirm two products, the Enterprise 

official provided a list of five additional products that, according to the Enterprise, were 

developed during FY 2001. We confirmed that, in fact, two of the products had been 

developed and could be used to replace the two products included in NASA’s initial 

submission that were inaccurately reported as developed. We did not attempt to confirm

development of the other three products. In this particular case, if 15 products had been 

developed, the indicator requiring 10 products may have been exceeded. However, 

because the Enterprise reported that only 10 products had been developed, the 

Performance Report may not accurately reflect actual performance for this APG. To 

ensure accurate reporting of performance results in future Performance Reports, 

Enterprise officials should carefully review supporting data to verify and validate actual 

performance prior to submitting assessments for inclusion in the Performance Report. 

Conclusion 

NASA continues to improve its performance planning and reporting process. The 

Agency could improve the accuracy and reliability of future Performance Reports by 

more effectively verifying and validating performance to ensure that reported results 

accurately reflect supporting data. Enterprises and Crosscutting Processes initially 

reported performance results that were not fully reliable or were not clearly presented for 

10
7 of the 19 APG's that we reviewed. Ensuring the reliability and clear presentation of

performance results would increase the Performance Reports' value as a decisionmaking 

tool. 

Recommendations, Management's Response, and Evaluation of 

Response 

The Deputy Chief Financial Officer for Financial Management should emphasize in 

the data call letter for the FY 2002 Performance Report and subsequent 

Performance Reports that responsible Enterprise and Crosscutting Process officials 

must: 

 1. Verify and validate actual performance to ensure that reported results 

accurately reflect supporting data. 

 2. Report fully successful accomplishment of APG's only when all required 

elements and indicators are accomplished within the subject fiscal year. 

 3. Align reported results appropriately with the planned performance stated in 

APG's. 

 4. Disclose data limitations in Performance Reports, and compare disclosures 

included in all Agency reports to ensure that data limitations are consistently 

disclosed. 

Management's Response. Concur. Management stated that all of the report's 

recommendations would be addressed in the FY 2002 and subsequent Performance 

Report data call letters. Management will provide us a copy of the FY 2002 data call 

letter when it is issued. 

Evaluation of Response. Management's planned actions are responsive to the 

recommendations. The recommendations are resolved but will remain undispositioned 

and open until agreed-to corrective actions are completed. 

11
Appendix A. Objectives, Scope, and Methodology 

Objectives

The overall objective was to assess the quality of data supporting the reported results in 

the NASA Fiscal Year (FY) 2001 Performance Report. The specific objectives were to 

review and test selected annual performance goals (APG’s) to assess whether the data 

were appropriate for the APG and whether they were complete, accurate, consistent, and 

timely. 

Scope and Methodology 

The audit covered APG's in NASA's FY 2001 Revised Final Annual Performance Plan. 

NASA's Plan included 88 APG's consisting of 334 indicators. To perform this audit, we 

concentrated on APG's within seven areas considered critical to the Agency: 

environmental management, fiscal management, information security, information 

technology, program and project management, safety and mission assurance, and human 

capital management. 

During FY 2001, NASA conducted its programs and activities through five Strategic 

Enterprises that accomplish NASA's mission: Space Science, Earth Science, Human 

Exploration and Development of Space (HEDS), Aerospace Technology, and Biological 

and Physical Research. Supporting the Strategic Enterprises are four Crosscutting 

Processes: Manage Strategically, Provide Aerospace Products and Capabilities 

(PAPAC), Generate Knowledge, and Communicate Knowledge. We covered the five 

Enterprises and four Crosscutting Processes by reviewing one or more APG's for each 

management organization. Further, we included only APG's that NASA intended, at the 

time of our audit, to report as being achieved or exceeded. We also included APG's that 

were similar to those we identified in our audits of FY's 1999 and 2000 APG's for which 

no reliable supporting data existed. We reviewed 19 APG's that fit within one of seven 

critical areas, were initially reported as achieved or exceeded, or were similar to those 

identified in prior audits. Of the 80 indicators comprising the 19 APG's, 66 indicators 

were reported as achieved. We reviewed 59 of those 66 indicators. Appendix B provides 

details on the APG's and respective indicators that we reviewed. Although we did not 

use statistical sampling procedures, we considered the selected APG's reasonably 

representative of all the APG's included in NASA's FY 2001 Revised Final Annual 

Performance Plan. 

To accomplish our objectives we did the following: 

• Reviewed Government Performance and Results Act (GPRA) legislation, Office of 

Management and Budget (OMB) guidance, NASA policy, and related documentation 

relative to measuring and reporting performance results. 

12
Appendix A 

• Obtained and reviewed, for the selected APG's, the measured data and information 

supporting the results that were included in NASA's FY 2001 Performance Report. 

• Interviewed NASA personnel and others who had a role either in collecting and 

providing the statistics and information used to measure results or in summarizing 

and reporting the results. 

• Determined, through interviews and reviews of readily available studies or analyses, 

whether there were known major problems with the systems or sources of the 

performance data. 

We did not test any systems to determine whether they accurately accumulated and 

reported their respective data. In addition, we could not assess whether supporting data 

was complete because the level of documentation we received allowed testing for validity 

only. That is, we could test backwards from data obtained to assess whether results were 

valid, but we could not test from original documentation forward to assess completeness. 

OMB Circular A-11, Section 232.10, “Assessing the completeness and reliability of 

performance data,” considers performance data complete if actual performance is 

reported for every performance goal and indicator in the annual plan and the agency 

identifies in the report any performance goals and indicators for which actual 

performance data are not available and notes that the data will be included in a 

subsequent annual report. Except as noted in the finding section, we determined that 

NASA reported actual performance for each APG sampled. As discussed in the finding 

section, the Aerospace Technology Enterprise did not provide results for each of the 21 

indicators established for APG 1R8. 

Our intention was to issue this audit report before NASA released its FY 2001 

Performance Report (about March 29, 2002) as we had done for audits of selected 

supporting data for the FY’s 1999 and 2000 Performance Reports. On 

December 19, 2001, management informed us that the printing schedule for the FY 2001 

Performance Report had been advanced to the first week of February 2002. NASA 

management explained that the date was changed to ensure that the finished product 

would be issued by the required date of March 29, 2002. We were unable to allocate 

additional resources to the audit to ensure that NASA had the full benefit of our findings 

prior to publication of the FY 2001 Performance Report. Nevertheless, to the extent 

possible, we kept management informed of our audit results. As a result, NASA was able 

to make certain changes to the FY 2001 Performance Report prior to its official release. 

Because the FY 2001 Performance Report was issued before our audit report, our 

recommendations address future events, not the FY 2001 Performance Report. 

13
Appendix A 

Management Controls Reviewed 

We reviewed the following controls with respect to measuring and reporting 

performance: 

• NASA FY 2001 Performance Plan, including revisions 

• NASA Strategic Plan (2000) 

• NASA Strategic Management Handbook (February 2000) 

• Office of the Chief Financial Officer, Office Work Instruction, “Performance Plan 

Update & Reporting,” HOWI7410-B003 

• OMB Circular A-11, “Preparing and Submitting Budget Estimates” 

(July 17, 2001, revised November 8, 2001) 

• FY 2001 Performance Report Data Call Letter from the Chief Financial Officer to 

the Officials-in-Charge of Headquarters and Functional Offices (August 30, 2001) 

Management controls for verifying and validating the reliability of GPRA-related 

performance data and the reported results were not adequate as evidenced by the 

conditions discussed in the finding. 

Audit Field Work 

We conducted field work from December 2001 through May 2002 at NASA 

Headquarters and obtained supporting documentation from the NASA Centers. 

14
Appendix B. Annual Performance Goals Reviewed in Detail 

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.)

1CK3 Ensure consistent, high-quality, external communication by meeting 2 of the 3 indicators 

for this annual performance goal. 

Indicators: 

• Increase new opportunities to transfer technology developed at NASA to private 

industry to 20,100. Opportunities will be made available to the public through 

the NASA Technology Tracking System (TechTracS) database and will be 

accessible through the Internet. 

• Produce two industry-specific editions "Aerospace Technology Innovations"

publication in FY 2001. 

• Provide publications that will communicate technologies available for 

commercial use or that have already been commercialized. Print 

subscriber/distribution metrics are: "Aerospace Technology Innovations" 

(12,500), "Spinoff " (51,000), and "Tech Briefs" (210,000). 

1G3 The Space Science Enterprise, the Earth Science Enterprise, and the OLMSA/HEDS 

[Office of Life and Microgravity Sciences and Applications] will use competitive merit 

review wherever possible to select performers for science and basis technology research. 

NASA will meet at least 2 out of 3 of the indicators for this annual performance goal. 

Indicators:

• NASA will use Announcements of Opportunity (AOs), NASA Research 

Announcements (NRAs), and Cooperative Agreement Notice solicitations to 

award 80 percent or more of science and basic research funds via merit 

competition in the Enterprises and Functional offices that fund scientific research. 

• NASA will meet the level of funding requested by the investigators in their 

proposals 80% of the time. 

• NASA will increase the number of investigators funded over the 1999 baseline. 

1G5 The Space Science Enterprise, the Earth Science Enterprise, and OLMSA/ HEDS will 

make science data obtained widely accessible as soon as possible after receipt and will 

maintain these data in open archives. NASA will meet the two indicators for this target. 

Indicators:

• The Space Science Enterprise, and the Earth Science Enterprise, will achieve their 

specific individual indicators for ensuring mission data maintenance and access. 

• OLMSA will continue the archival of their life sciences research publications. 

1H2 Complete initial next decade planning mission architecture studies and technology plans. 

Architecture studies support near-term technology investment decisions to create 

building blocks that may enable a range of long-term planning options for future 

missions of exploration. 

Indicator: 

• Complete initial next decade planning mission architecture studies. 

* See the Legend at the end of the table. 

15
Appendix B 

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.) 

1H5 Continue initial research on the International Space Station (ISS) by conducting 6 to 10 

investigations. 

Indicators: 

• Increase fundamental knowledge in biological and biomedical sciences and address 

critical questions in crew health and safety by conducting 6 to 10 ISS investigations. 

• Acquire unique data on colloidal self assembly as an essential first step in the 

synthesis of new materials from colloidal particles. 

• Measure the ISS acceleration environment, develop models to characterize the effects 

of that environment on ISS research, and disseminate those results to the ISS 

investigator community. 

1H7 The Office of Space Flight continues to invest in Space Shuttle operations. Investments 

include hardware production, ground processing, launch and landing operations, flight crew

operations, training, logistics, and sustaining engineering. The annual performance goal is to 

achieve 8 or fewer flight anomalies per mission. 

Indicator: 

• Achieve 8 or fewer in-flight anomalies per mission. 

1H11 Deployment of the ISS occurs with on-orbit assembly over several years. Successful and 

timely deployment is dependent on the Shuttle and other international launch vehicles, and 

the provision of some elements and services from international partners and participants. The 

performance target is to successfully complete the majority of the ISS planned on-orbit 

activities such as delivery of mass to orbit and enhanced functionality. 

Indicators: 

• Expansion of the capabilities of the ISS through launch and delivery of 180,000 lbs. of 

hardware and logistics to the ISS; and initiation and demonstration of ISS Extravehicular 

Activity (EVA) capability to support up to 30 EVAs annually from the U.S. Airlock. 

This will be measured by completion of a minimum of 5 EVAs from the ISS Airlock. 

1H18 Demonstrate, in ground test, at least one technology that could reduce up to 25% of life 

support logistics over ISS baseline and release report of progress for review on the Internet. 

Indicators: 

• Demonstrate, in ground test, technologies that could reduce up to 25% of life support 

logistics over ISS baseline and release report of progress for review on the Internet. 

• Perform detailed calculation of life support equivalent system mass index and place 

online for review and comment. Equivalent system mass index is a measure of the 

performance of a life support system incorporating demonstrated technologies. 

1H20 Increase the percentage of the space operations budget allocated to acquisition of 

communications and data services from the commercial sector to 15% in FY 2001. The 

space communications program will conduct tasks that enable commercialization and will 

minimize investment in government infrastructure for which commercial alternatives are 

being developed. 

Indicator: 

• Increase to 15% the space operations budget allocated to acquisition of 

communications and data services from the 10% FY 2000 annual performance goal. 

* See the Legend at the end of the table. 

16
Appendix B 

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.) 

1MS3 Renew Agency’s management systems, facilities, and human resources through updated 

use of automated systems, facilities revitalization, and personnel training by meeting 4 

out of 7 performance indicators in this area. 

Indicators:

• Cost at least 75% of the resources authority available to cost during the fiscal 

year. 

• Completing installation of the Budget and Core Accounting Integrated Financial 

Management System at NASA’s remaining field locations. 

• Maintain a diverse NASA workforce where women, minorities, and persons with 

disabilities are represented at levels equal to or greater than their FY 1999 levels, 

with a target of increasing representation of minorities by at least one percent per 

year, women by at least one percent per year, and persons with disabilities by at 

least .5 percent per year. 

• Increasing training opportunities in technology-based learning by 10%. 

• Increasing by 20% employee use of technology-based learning opportunities. 

• Using FY 01 budgeted funds for awarding construction contracts toward reducing 

the Agency’s estimated $1.4B facilities revitalization needs. 

• Implement 60% of the identified Environmental Compliance and Restoration 

(ECR) projects to reduce and manage the Agency’s $1.1B future unfunded 

environmental liability. 

1MS4 Improve information technology (IT) infrastructure service delivery to provide increased 

capability and efficiency while maintaining a customer rating of satisfactory, and 

enhance IT security through a reduction of system vulnerabilities across all NASA 

Centers, emphasizing IT security awareness training for all NASA personnel by meeting 

2 out of 2 performance indicators in this area. 

Indicators:

• Improve IT infrastructure service delivery to provide increased capability and 

efficiency while maintaining a customer rating of “satisfactory” and holding costs 

per resource unit to the FY 98 baseline. 

• Enhance IT security through reduction of system vulnerabilities across all NASA 

Centers and through emphasis on IT security awareness training for all NASA 

personnel. 

1P3 Ensure the availability of NASA’s spacecraft and major ground facilities by keeping the 

operating time lost due to unscheduled downtime to less than 10% of scheduled 

operating time. 

Indicator: 

• Each field center is reporting the operational downtime of the facilities identified 

for inclusion in the measure.

1P5 Dedicate 10 to 20 percent of the Agency's Research & Development budget to 

commercial partnerships. 

Indicator: 

• Each of the Enterprises are reporting the value of their contribution to commercial 

partnerships. 

* See the Legend at the end of the table. 

17
Appendix B 

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.) 

1R8 Develop at least three new design tools, accomplish at least four demonstrations of 

advances in computation and communications, and complete the intelligent synthesis 

environment proof-of-concept systems capability build to technology readiness level 3: 

indicators include computer testbed demonstrations, real-time remote access of data, new 

design methods and an intelligent synthesis environment proof-of-concept system.

Indicators: 

Aerospace Focused — High Performance Computing and Communications [HPCC]

• Develop software tools to reduce parallelization time from months to one week 

while maintaining 50% application performance compared with manual 

parallelization. 

• Develop tools to benchmark testbed performance in computing capability, 

database manipulation, and scheduling to evaluate alternate scheduling strategies 

and choose optimal approaches to reduce variability and improve predictability of 

turnaround time. 

• Develop automated quality of service data collection tool capable of measuring 2 

service classes and scalable to at least 5 nodes. 

• 3 relevant application codes parallelized; 3 data analysis codes parallelized; 

documented evaluation of parallelization tools. 

• 3X performance in an aerospace application through the integration of 

networking enhancements into application codes. 

• 3 applications interoperating on multiple Quality of Service (QoS) enabled 

networks; 50Mbps (aggregate internal) multicast; gigabit performance between 2 

NASA sites; 2 applications utilizing enhanced hybrid networking. 

• Improvement in aerospace applications: Complete combustor and compressor 

simulation in 3 hours each; high-fidelity space transportation vehicle analysis in 1 

week and optimization enabled; S&C [stability and controls] database generation 

for aerospace vehicles within 1 week; demonstration of improvements in 4 

NASA-sponsored design events. 

• Assess initial HPCC technology capabilities and customer impacts. 

• Demonstrate a near-term, state-of-the-art intelligent synthesis environment (ISE), 

user interface and infrastructure. 

• Demonstrate life-cycle simulation and ISE capabilities as specified by the 

prototype test applications (i.e., legacy engineering and analysis tools.) 

• Validate three prototype test applications. 

• Demonstrate ISE prototype measurement and assessment techniques. 

Aerospace Base Research and Technology (R&T) 

• Develop software tools for design of advanced computing systems. 

• Acquire and incorporate new large-scale computing systems and demonstrate 

seamless operations with heterogeneous distributed computing environment. 

• Demonstrate remote connectivity to high data-rate instruments and distributed 

real-time access to instrument data.

* See the Legend at the end of the table. 

18
Appendix B 

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.) 

1R8 

continued 

• Demonstrate an environment for aerospace hardware design that includes: remote 

connectivity and access to flight simulation data, computational simulation data and 

archival databases. 

• Demonstrate prototype cross-fidelity aerospace design system. 

• Establish experimental and analytical methodology for composite stringer pull-off 

failure prediction. 

• Figures of merit from static wind tunnel or CFD [computational fluid dynamics] 

results developed and assessed for use in predictions of uncommanded transonic 

lateral motions due to Abrupt Wing Stall. 

• Conduct turbulence modeling workshop to provide direction for turbulence modeling 

research to increase design confidence in flight regimes dominated by flow

separation. 

• Conduct assessment of OAT [Office of Aerospace Technology] program element 

impacts on goals of three pillars. 

1S4 Successfully develop and launch no less than one of two missions within 10% of budget and 

schedule. Missions are: Mars Odyssey ('01 Orbiter) and Genesis. 

Indicators: 

• Mars Odyssey Development: Deliver for launch; successful launch and check-out.

• Genesis Development: Deliver for launch; successful launch and check-out. 

• Rosetta Development: Deliver the flight units for the four U.S.-provided instruments 

or instrument subsystems to ESA [European Space Agency]. 

• Two Wide-angle Imaging Neutral-atom Spectrometer (TWINS) Development: 

Continue instrument development and deliver Flight Unit #1 for Integration and Test. 

• Comet Nucleus Tour (CONTOUR) Development: Successful Critical Design Review 

(CDR), meeting all program level requirements. 

• Discovery Program Future Missions: New mission selection. 

1S6 Perform innovative scientific research and technology development by meeting technology

development objectives for major projects, by achieving mission success in space physics 

rocket and balloon flights, and by making satisfactory research progress in related R & A

[Research and Analysis] and DA [Data Analysis] programs. Meet no fewer than 66% of the 

performance objectives for the following technology and research programs: Solar-B, 

STEREO [Solar Terrestrial Relations Observatory], Solar Probe, Future Solar Terrestrial 

Probes, Future Deep Space Technology, CISM [Center for Integrated Space Microelectronics], X-2000, Sounding Rockets, and Balloons. Achieve a "fully effective" (green) 

overall science achievement rating from the Space Science external advisory committee. 

Indicators:

• Solar-B Technology Development: Deliver engineering model of the optical telescope 

and X-ray telescope.

• STEREO Technology Development: Successfully complete Phase B effort, including 

Confirmation Review. 

• Solar Probe Technology Development: Begin Solar Probe prototype thermal shield 

fabrication.

• Future Solar Terrestrial Probes Technology Development: Complete preliminary

concept definitions for spacecraft systems and instruments for Magnetospheric 

Multiscale.

* See the Legend at the end of the table. 

19
Appendix B

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.)

1S6 

continued 

• Future Deep Space Technology Development: Deliver X-2000 Level 1-3 

requirements documents; define subsystem interfaces; demonstrate intermediate-level 

multi-functional structures (MFS); complete definition of system architecture; evaluate 

key risk areas and pass decision gates.

• CISM Technology Development: Demonstrate and deliver prototype advanced power 

transistor (0.35 micron Siliconon Insulator [SOI] Complementary Metallic Oxide 

Semiconductor [CMOS] [sic]: demonstrate Active Pixel Sensor with advanced 

processing capabilities on a single chip. 

• X-2000 Technology Development: Deliver engineering model and flight set of 

avionics.

• Sounding Rocket Flights: Achieve launch success rate of 80% for sounding rocket 

flights. 

• Balloon Flights: Achieve launch success rate of 80% for balloon flights. 

1S10 Investigate the composition, evolution, and resources of Mars, the Moon, and small bodies 

by successfully launching a Mars mission, by obtaining data from operational spacecraft, and 

by making satisfactory progress in related Research and Analysis (R&A) and Data Analysis 

(DA) programs. Meet no fewer than 75% of the performance objectives for Mars Odyssey

('01 Orbiter), Comet Nucleus Tour (CONTOUR), Mars Global Surveyor, and R&A. 

Achieve a "fully effective" (green) overall science achievement rating from the Space 

Science external advisory committee. 

Indicators:

• Mars Odyssey: Deliver for launch, within 10% of planned development budget and 

schedule; successful launch and check-out.

• CONTOUR Development: Successful Critical Design Review (CDR), to document 

that the design meets all program level requirements. 

• Mars Global Surveyor Operations: Complete primary mapping mission.

• Research and Analysis: Issue NASA Research Announcement (NRA) for Research 

Opportunities in Space Science (ROSS). 

1Y2 Successfully disseminate Earth Science data to enable our science research and applications 

goals and objectives by meeting all performance indicators in this research area. 

Indicators: 

• Make available data on prediction, land surface, and climate to users within 5 days. 

• Increase by 20% the volume of data archived compared to FY00 (annual performance 

goal = 442 terabytes). 

• Increase the number of distinct Earth Observing System Data and Information 

System (EOSDIS) customers by 20% compared to FY00 (annual performance goal = 

1.5 million). 

• Increase products delivered from the Distributed Active Archive Centers (DAAC's) 

by 10% compared to FY00 (annual performance goal = 5.4 million). 

• User satisfaction: Increase the number of favorable comments from DAAC and Earth

Science Information Partner (ESIP) users as recorded in the customer contact logs 

over FY00. Implement user satisfaction survey. 

• Decrease total percentage of order errors by 5% over FY00. 

* See the Legend at the end of the table. 

20
Appendix B 

APG * Description as Stated in the FY 2001 Revised Final Annual Performance Plan 

(Text in italics designates indicators that we did not review.) 

1Y16 Stimulate the development of a robust commercial remote sensing industry by meeting at 

least 4 of 5 performance indicators in this area. 

Indicators: 

• Develop ten new market commercial products (e.g., oil spill containment software 

by EarthSat and map sheet products by Earth Resources Data Analysis System

Inc.), in joint commercial applications research projects.

• Identify at least one new commercial source of science data as a result of the 

Scientific Data Purchase activities for Earth Science research and applications.

• Develop four new validated commercial information products as a result of 

verification and validation partnerships with the private sector and other users 

through the Mississippi State Commerce Initiative and the Space Act Agreement.

• Conduct Earth Observation Commercial Applications Program (EOCAP) 

Technology projects that result in ten prototype products that quantify the utility

of Hyperspectral and Synthetic Aperture Radar technologies and define future 

market requirements. 

• Increase the cost share leveraging with companies, academia and other 

government agencies within the EOCAP and Affiliated Research Center (ARC) 

programs by 10%.

Legend:

* 1 -- FY 2001. 

CK -- Communicate Knowledge Crosscutting Process. 

G -- Generate Knowledge Crosscutting Process. 

H -- Human Exploration and Development of Space Enterprise. 

MS -- Manage Strategically Crosscutting Process. 

P -- Provide Aerospace Products and Capabilities Crosscutting Process. 

R -- Aerospace Technology Enterprise. 

S -- Space Science Enterprise. 

Y -- Earth Science Enterprise. 

21
 Appendix C. Summary of Prior Audit Coverage 

NASA Office of Inspector General (OIG) 

Report to the Chairman, Committee on Government Reform, United States House 

of Representatives, Report Number IG-01-028, May 18, 2001. In response to a 

congressional request, the OIG determined the 10 most significant performance measures 

in NASA’s fiscal year (FY) 2000 Performance Report; assessed whether each was a 

useful indicator of performance; and determined the validity and accuracy of 

performance results reported by NASA for each measure. The OIG concluded that all 10 

measures had some usefulness as indicators of performance in support of NASA’s 

mission but that the Agency could more precisely word many of the measures to better 

demonstrate NASA’s actual performance in meeting stated goals. 

“Validation and Verification of Selected NASA FY 2000 Performance Data Related 

to the Government Performance and Results Act (GPRA),” Report Number 

IG-01-020, March 30, 2001. The OIG reviewed the accuracy and reliability of 

performance data for 23 performance targets to be reported in NASA's FY 2000 

Performance Report. We concluded that the supporting data and information on 19 of 23 

performance targets reviewed were adequate, and we did not identify any significant 

problems with reported actual performance for those targets. However, the reported 

performance on four targets reviewed was not fully reliable because the supporting data 

did not adequately support the results described. The report contains three 

recommendations to improve the reliability of reported performance. Management 

concurred with all the recommendations and implemented corrective actions. 

Report to the Chairman, Committee on Governmental Affairs, United States 

Senate, November 15, 2000. In response to a congressional request, the OIG conducted 

a review of NASA’s FY 1999 Performance Report to determine whether the report 

effectively addressed the key management challenges faced by the Agency. The specific 

objectives were to determine whether the FY 1999 Performance Report contained goals 

and measures that directly relate to key management challenges, evaluate NASA’s 

performance under the relevant goals and objectives, evaluate the validity and reliability 

of the data on which NASA based its performance, and evaluate NASA’s strategies to 

meet performance goals not attained during FY 1999. The OIG reported that NASA's 

FY 1999 Performance Report and FY 2001 Performance Plan described goals and 

measures for 8 of the 10 management challenges identified by the General Accounting 

Office (GAO) and the OIG, but did not provide enough specific information for 4 of the 

10 management challenges. The OIG considered information technology security to be a 

material control weakness and expressed concerns regarding future reusable launch 

vehicles; commercialization of launch services; environmental management; and 

balancing risk, performance, and cost in its programs and projects. 

22
Appendix C 

“Validating FY 1999 Performance Data To Be Reported Under the Government 

Performance and Results Act (GPRA),” Report Number IG-00-020, March 28, 

2000. The OIG reviewed the accuracy and reliability of performance data for 23 

performance targets to be reported in NASA’s FY 1999 Performance Report. We

concluded that the supporting data and information on 18 of 23 performance targets 

reviewed were adequate, and we did not identify any significant problems with reported 

actual performance for those targets. However, the reported performance on five targets 

reviewed was not fully reliable because the supporting data did not adequately support 

the results described. The report contains three recommendations to improve the 

reliability of reported performance. Management concurred with all the 

recommendations and implemented corrective actions. 

“NASA Implementation of the Government Performance and Results Act,” Report 

Number IG-99-055, September 28, 1999. The report states that NASA (1) had not 

made a timely assessment of progress in achieving FY 1999 performance goals and (2) 

had not established formal procedures to ensure that all the data and information used to 

evaluate progress and report final results are accurate and reliable. The report contains 

three recommendations to track progress, take timely corrective actions, and verify and 

validate supporting data. Management concurred with all the recommendations and 

implemented corrective actions. 

General Accounting Office (GAO) 

“NASA - Status of Plans for Achieving Key Outcomes and Addressing Major 

Management Challenges,” GAO-02-184, November 27, 2001. In response to a 

congressional request, GAO reviewed NASA's FY 2002 Performance Plan to assess 

planned performance for three key outcomes: (1) expand scientific knowledge of the 

Earth system, (2) deploy and operate the International Space Station safely and costeffectively, and (3) expand the commercial development of space. GAO found that 

NASA had improved its FY 2002 performance plan and responded to recommendations 

or suggestions by GAO and others to make its plan more useful. GAO reported that 

NASA's annual performance goals for its outcomes generally appeared to be objective 

and help to measure progress toward the outcomes. Yet the plan still did not explain the 

reasons for changes in performance goals. GAO concluded that not having these 

explanations could hinder the ability to assess NASA's performance over time. 

 

“NASA - Status of Achieving Key Outcomes and Addressing Major Management 

Challenges,” GAO-01-868, July 31, 2001. In response to a congressional request, GAO 

reviewed NASA's FY 2000 Performance Report to assess progress in achieving three key 

outcomes: (1) expand scientific knowledge of the Earth system, (2) deploy and operate 

the International Space Station safely and cost effectively, and (3) expand the 

commercial 

23
Appendix C 

development of space. GAO found that NASA reported mixed progress in achieving the 

key outcomes and that NASA's strategies for achieving unmet performance targets for 

these outcomes were generally clear and reasonable. 

“Observations on the National Aeronautics and Space Administration's Fiscal Year 

1999 Performance Report and FY 2001 Performance Plan,” GAO-NSIAD-00-192R, 

June 30, 2000. In response to a congressional request, GAO reviewed NASA's report 

and plan with a focus on three key outcomes: (1) expand scientific knowledge of the 

Earth system, (2) deploy and operate the International Space Station safely and cost 

effectively, and (3) expand the commercial development of space. GAO determined that 

NASA's FY 1999 performance objectives and targets were generally objective and 

measurable, but NASA continued quantifying output measures instead of outcomes. 

Additionally, the GAO reported that NASA did not provide assurance that performance 

information was credible and expressed concern about how NASA planned to use 

indicators in assessing whether goals were met. The GAO's review of NASA’s FY 2001 

Performance Plan concluded that the plan did not include an explicit discussion of 

procedures for verifying and validating performance data and does not address possible 

data limitation issues and problems. 

“Observations on the National Aeronautics and Space Administration's Fiscal Year 

2000 Performance Plan,” GAO-NSIAD-99-186R, July 20, 1999. In response to a 

congressional request, the GAO reviewed NASA's plan with a focus on (1) assessing the 

usefulness of the Agency's plan for decisionmaking and (2) identifying the degree of 

improvement the Agency's FY 2000 Performance Plan represented over the FY 1999 

Plan. GAO determined that the Agency's plan should be useful to decision makers. It 

provides a limited picture of intended performance across the Agency, a general 

discussion of strategies and resources the Agency will use to achieve its goal, and limited 

confidence that performance information will be credible. NASA's FY 2000 Plan 

represented a moderate improvement over the FY 1999 Plan in that it indicates some

degree of progress in addressing the weaknesses identified in GAO's assessment of the 

FY 1999 Plan. 

24
Appendix D. Management's Response 

25
Appendix E. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters 

HQ/A/Administrator 

HQ/AI/Associate Deputy Administrator 

HQ/AB/Associate Deputy Administrator for Institutions and Asset Management 

HQ/AA/Chief of Staff 

HQ/AE/Chief Engineer 

HQ/AO/Acting Chief Information Officer 

HQ/B/Deputy Chief Financial Officer for Financial Management 

HQ/B/Deputy Chief Financial Officer for Resources (Comptroller) 

HQ/BF/Director, Financial Management Division 

HQ/BR/Director, Resources Analysis Division 

HQ/C/Acting Director for Headquarters Operations 

HQ/G/General Counsel 

HQ/H/Assistant Administrator for Procurement 

HQ/HK/Director, Contract Management Division 

HQ/HS/Director, Program Operations Division 

HQ/J/Assistant Administrator for Management Systems 

HQ/JM/Director, Management Assessment Division 

HQ/L/Assistant Administrator for Legislative Affairs 

HQ/M/Associate Administrator for Space Flight 

HQ/P/Assistant Administrator for Public Affairs 

HQ/Q/Associate Administrator for Safety and Mission Assurance 

HQ/R/Associate Administrator for Aerospace Technology 

HQ/S/Associate Administrator for Space Science 

HQ/U/Associate Administrator for Biological and Physical Research 

HQ/X/Assistant Administrator for Security Management and Safeguards 

HQ/Y/Associate Administrator for Earth Science 

NASA Advisory Officials 

Chair, NASA Advisory Council 

26
Appendix E 

NASA Centers 

ARC/D/Director, Ames Research Center 

DFRC/X/Director, Dryden Flight Research Center 

GRC/0100/Director, John H. Glenn Research Center at Lewis Field 

GSFC/100/Director, Goddard Space Flight Center 

JPL/1000/Director, Jet Propulsion Laboratory 

JSC/AA/Director, Lyndon B. Johnson Space Center 

KSC/AA/Director, John F. Kennedy Space Center 

KSC/CC/Chief Counsel, John F. Kennedy Space Center 

LaRC/106/Acting Director, Langley Research Center 

MSFC/DA01/Director, George C. Marshall Space Flight Center 

SSC/AA00/Director, John C. Stennis Space Center 

Non-NASA Federal Organizations and Individuals 

Assistant to the President for Science and Technology Policy 

Deputy Associate Director, Energy and Science Division, Office of Management and 

 Budget 

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office 

 of Management and Budget 

Managing Director, Acquisition and Sourcing Management Team, General Accounting 

 Office 

Senior Professional Staff Member, Senate Subcommittee on Science, Technology, and 

 Space 

Chairman and Ranking Minority Member – Congressional Committees and 

Subcommittees 

Senate Committee on Appropriations 

Senate Subcommittee on VA, HUD, and Independent Agencies 

Senate Committee on Commerce, Science, and Transportation 

Senate Subcommittee on Science, Technology, and Space 

Senate Committee on Governmental Affairs 

House Committee on Appropriations 

House Subcommittee on VA, HUD, and Independent Agencies 

House Committee on Government Reform

House Subcommittee on Government Efficiency, Financial Management, and 

 Intergovernmental Relations 

House Subcommittee on Technology and Procurement Policy 

House Committee on Science 

House Subcommittee on Space and Aeronautics 

Congressional Member 

Honorable Pete Sessions, U.S. House of Representatives

27
NASA Assistant Inspector General for Audits 

Reader Survey 

The NASA Office of Inspector General has a continuing interest in improving the 

usefulness of our reports. We wish to make our reports responsive to our customers’ 

interests, consistent with our statutory responsibility. Could you help us by completing 

our reader survey? For your convenience, the questionnaire can be completed 

electronically through our homepage at http://www.hq.nasa.gov/office/oig/hq/audits.html

or can be mailed to the Assistant Inspector General for Audits; NASA Headquarters, 

Code W, Washington, DC 20546-0001. 

Report Title: Validation and Verification of Selected NASA Fiscal Year 2002 

Performance Data Related to the Government Performance and Results Act

Report Number: Report Date: 

Circle the appropriate rating for the following statements. 

Strongly 

Agree Agree Neutral Disagree 

Strongly 

Disagree N/A 

1. The report was clear, readable, and logically 

organized. 

5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A 

3. We effectively communicated the audit 

objectives, scope, and methodology. 

5 4 3 2 1 N/A 

4. The report contained sufficient information to

support the finding(s) in a balanced and 

objective manner. 

5 4 3 2 1 N/A 

Overall, how would you rate the report? 

# Excellent # Fair

# Very Good # Poor

# Good

If you have any additional comments or wish to elaborate on any of the above 

responses, please write them here. Use additional paper if necessary. 
How did you use the report?

How could we improve our report?

How would you identify yourself? (Select one) 

# Congressional Staff # Media 

# NASA Employee # Public Interest 

# Private Citizen # Other:

# Government: Federal: State: Local:

May we contact you about your comments? 

Yes: ______ No: ______ 

Name: ____________________________ 

Telephone: ________________________ 

___ 

Thank you for your cooperation in completing this survey.
Major Contributors to the Report 

Chester A. Sipsock, Program Director, Financial Audits, Management and Oversight 

Carol St. Armand, Program Manager 

Bonnie Armstrong, Auditor-in-Charge 

William R. Lester, Auditor 

Nancy Cipolla, Report Process Manager 

Annette Huffman, Program Assistant