IG-03-002

AUDIT 

REPORT EXPENDABLE LAUNCH VEHICLE 

PERFORMANCE MEASURES 

October 16, 2002

 National Aeronautics and 

Space Administration

OFFICE OF INSPECTOR GENERAL 
Additional Copies 

To obtain additional copies of this report, contact the Assistant Inspector General for 

Audits at (202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General 

for Audits. Ideas and requests can also be mailed to: 

 Assistant Inspector General for Audits 

 Code W 

 NASA Headquarters 

 Washington, DC 20546-0001 

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800) 

424-9183, (800) 535-8134 (TDD), or at 

www.hq.nasa.gov/office/oig/hq/hotline.html#form; or write to the NASA Inspector 

General, P.O. Box 23089, L’Enfant Plaza Station, Washington, DC 20026. The identity 

of each writer and caller can be kept confidential, upon request, to the extent permitted 

by law. 

Reader Survey 

Please complete the reader survey at the end of this report or at 

www.hq.nasa.gov/office/oig/hq/audits.html. 

Acronyms 

ELV Expendable Launch Vehicle 

FY Fiscal Year 

GAO General Accounting Office 

GPRA Government Performance and Results Act of 1993 

HEDS Human Exploration and Development of Space 

NPD NASA Policy Directive 

OMB Office of Management and Budget 

QuikTOMS Quik Total Ozone Mapping Spectrometer 
October 16, 2002 

W 

TO: M/Associate Administrator for Space Flight 

FROM: W/Assistant Inspector General for Audits 

SUBJECT: Final Report on Audit of Expendable Launch Vehicle Performance 

Measures 

 Assignment Number A-02-002-00 

 Report Number IG-03-002 

Enclosed please find the subject final report. Please refer to the Executive Summary for 

the overall audit results. Our evaluation of your response has been incorporated into the 

body of the report. We consider management’s proposed, corrective actions responsive 

for the recommendation. The recommendation will remain open for reporting purposes 

until corrective actions are complete. Please notify us when actions have been completed 

on the recommendation. The final report distribution is in Appendix G. 

We appreciate the courtesies extended to the audit staff. If you have questions 

concerning the report or would like to schedule an exit conference, please contact 

Mr. Daniel Samoviski, Program Director, Earth/Space Science Audits, at (301) 286-

6890; Ms. Nora Thompson, Program Manager, at (757) 864-3268; or Mr. Jim Richards, 

Auditor-in-Charge, at (321) 867-4841. 

[original signed by] 

Alan J. Lamoreaux 

Enclosure 
2 

cc: 

HQ/B/Acting Deputy Chief Financial Officer 

HQ/B/Comptroller 

HQ/BF/Director, Financial Management Division 

HQ/G/General Counsel 

HQ/M-5/Assistant Associate Administrator for Launch Services

HQ/JM/Director, Management Assessment Division 

KSC/AA/Director, John F. Kennedy Space Center 

KSC/VA/Manager, ELV and Payload Carriers Program, Kennedy Space Center 
Contents 

Executive Summary, i 

Introduction, 1 

Finding and Recommendation, 2

Performance Indicators in NASA Performance Plan, 2

Appendix A - Objectives, Scope, and Methodology, 7

Appendix B - ELV Program Performance Goals and Indicators Excerpted 

From the NASA FY 2003 Performance Plan, 10

Appendix C - Payload/Instrument Risk Categories in NPD 8610.7, 11

Appendix D - Risk Mitigation Requirements in NPD 8610.7, 13 

Appendix E - Verification and Validation of Performance Data, 14 

Appendix F – Management’s Response, 15 

Appendix G - Report Distribution, 16 
NASA Office of Inspector General 

IG-03-002 October 16, 2002 

 A-02-002-00 

Expendable Launch Vehicle 

Performance Measures 

Executive Summary 

Background. NASA's fiscal year (FY) 2003 budget for Expendable Launch Vehicle 

(ELV) services1

 is $338 million,2 including $36 million for technical management and 

acquisition services. The term expendable applies to launch vehicles that are used once 

and not recovered for reuse.3

 In February 2002, the NASA Administrator4 stated 

"NASA's ELV Program continues to afford the Agency a high level of launch success, 

with an impressive flight record of 98 percent." Within the Human Exploration and 

Development of Space (HEDS) Enterprise,5 the Office of Space Flight has overall 

responsibility for the ELV Program. Customers of the ELV Program include NASA 

Enterprises, such as the HEDS, Earth Science and Space Science Enterprises, and other 

Federal agencies such as the National Oceanic and Atmospheric Administration. Launch 

sites include Cape Canaveral Air Force Station, Florida; Vandenberg Air Force Base, 

California; Wallops Flight Facility, Virginia; Kwajalein Island, Republic of the Marshall 

Islands; and Kodiak, Alaska. 

The NASA FY 2003 Annual Performance Plan (Performance Plan)6

 includes two 

performance goals related to the ELV Program and one performance indicator for each 

 

1

Since 1987, NASA has relied on commercially procured ELV's to meet its requirements for scientific, 

Earth sensing, and planetary missions that do not require the Space Shuttle's unique capabilities. Before 

1987, NASA relied primarily on the Space Shuttle for such missions and rarely used ELV's. 2

The $338 million FY 2003 budget request for the ELV Program consists of (a) $36 million for ELV 

mission support (technical management and acquisition services); (b) $148 million for launches by the 

Space Science Enterprise (see footnote 5); (c) $53 million for launches by the Earth Science Enterprise; 

(d) $7 million for launch services needed by the Tracking and Data Relay Satellite System; and (e) $94 

million for launch services that NASA provides to other Federal agencies (for example, for weather 

satellites launched by the National Oceanic and Atmospheric Administration). The Federal agencies 

reimburse NASA for the $94 million in launch services the Agency provides them. The auditee provided 

the FY 2003 ELV budget information. 

3

The Space Shuttle is NASA's only reusable launch vehicle currently in operation. 4

The NASA Administrator's comment was part of his prepared statement to the House Committee on 

Science, on February 27, 2002. The 98-percent success rate was based on 55 successful missions of 56 

missions launched since 1987. 

5

NASA established five Enterprises to function in primary business areas for implementing NASA's 

mission and serving customers. Each Enterprise has a unique set of strategic goals, objectives, and 

implementing strategies that address the requirements of the Agency's primary customers. The five NASA 

Enterprises are: Space Science, Earth Science, HEDS, Aerospace Technology, and Biological and Physical 

Research. 

6

NASA included performance goals for the ELV Program beginning with the NASA FY 2002 Annual 

Performance Plan. 
goal (see Appendix B). The Performance Plan and ELV goals and indicators must meet 

requirements of the Government Performance and Results Act of 1993 (GPRA)7 and 

Office of Management and Budget (OMB) Circular A-11, "Preparing and Submitting 

Budget Estimates." Goals and indicators must be defined either in an objective and 

quantifiable manner or as precise, descriptive statements that allow an accurate, 

independent determination to be made of actual performance. In addition to the goals 

and indicators in the Performance Plan, Kennedy Space Center (Kennedy) has developed 

several metrics to evaluate performance of operations and organizations that provide 

launch services for the ELV Program. 

NASA Policy Directive (NPD) 8610.7, "Launch Services Risk Mitigation Policy for 

NASA-Owned or NASA-Sponsored Payloads," dated February 4, 1999, contains the risk 

mitigation policy for launches of NASA-owned or -sponsored payloads. NASA payloads 

must use reasonable commercial sources of launch services while ensuring vehicles are 

not exposed to excessive risk. To control mission risk, management must categorize 

payloads against specific criteria in NPD 8610.7 and select launch vehicles based on the 

payload risk category (see Appendix C). Additionally, management must follow other 

procedures in NPD 8610.7, relating primarily to procurement, that ensure the 

implementation of risk mitigation policy (see Appendix D). 

Objectives. Our overall audit objective was to evaluate the ELV Program's performance 

measurement system and compliance with NASA risk mitigation policy for launch 

vehicles. Specifically, we determined whether NASA did the following: 

• Established a performance measurement system8

 for the ELV Program that 

complied with Federal requirements and other appropriate criteria. 

• Managed ELV launch services in compliance with Agency risk mitigation policy. 

Specifically, we determined whether NASA categorized payload and instrument 

risk and selected launch vehicles based on the risk category. We also determined 

whether NASA complied with other relevant provisions of NASA's risk 

mitigation policy, including applicable Federal and NASA clauses in contracts for 

launch services.

Details on the audit objectives, scope, and methodology are in Appendix A.

Results of Audit. Generally, NASA established an adequate performance measurement 

system that complied with Federal requirements. Additionally, the four Kennedy metrics 

that we reviewed met applicable GPRA and OMB requirements. Further, planned 

procedures to verify and validate performance data for the ELV Program (see 

 

7

Public Law 103-62, as amended, contains the GPRA legislation. 8

For purposes of this audit, the performance management system includes (1) NASA's system for 

measuring ELV Program performance as stated in the NASA Annual Performance Plan and (2) the ELV 

Program Office system for measuring performance. 

ii
Appendix E) appeared to meet requirements.9

 Finally, the ELV Program Office met 

NPD 8610.7 risk mitigation requirements that we reviewed for the nine launches 

completed in or scheduled for 2001 through 2003. 

However, NASA needs to clarify one of two ELV performance goals and indicators in 

the Agency's FY 2003 Performance Plan to better meet GPRA and OMB requirements. 

Specifically, the Office of Space Flight had not fully defined two key terms or provided 

the basis for the prescribed 95-percent launch success rate used in one of two 

performance goals and indicators in the FY 2003 Performance Plan. Fully defining the 

terms and explaining the basis for the 95-percent rate will provide improved performance 

information. With improved performance information, external stakeholders, such as the 

Congress and the public, will be better able to determine the relative effectiveness of the 

ELV Program and have confidence that the program's results justify the $36 million spent 

for technical management and acquisition services for the $338 million ELV Program. 

Recommendation. NASA should explain the ELV Program performance indicator in 

the FY 2003 NASA Performance Plan or Performance Report to clarify the meaning of 

the two key terms and the basis for the prescribed 95-percent launch success rate. 

Management’s Response. NASA concurred with the recommendation and will take 

corrective action starting with the FY 2002 NASA Performance Report, which NASA 

expects to release in late-January 2003. The complete text of management’s response is 

in Appendix F. 

 

9

NASA's FY 2002 Performance Plan explains NASA's planned verification and validation procedures for 

FY 2002 performance data. However, we could not test the adequacy of procedures because they will not 

be performed until after the end of FY 2002. 

iii
Introduction 

ELV Program Management. The NASA Office of Space Flight has responsibility for 

the ELV Program. Within the Office of Space Flight, the Associate Administrator has 

tasked the Assistant Associate Administrator for Launch Services to develop specific 

objectives, strategies, and requirements for the ELV Program and to serve as an internal 

and external advocate for the program. In addition, the ELV Flight Planning Board is the 

Agency-level forum for approval of new missions, vehicle configuration changes, and 

launch manifest changes. The Board includes representatives from four NASA 

Enterprises--HEDS, Earth Science, Space Science, and Aerospace Technology. Kennedy 

serves as the lead Center for ELV launch services through the ELV Program Office. The 

ELV Program Office is responsible for technical management and acquisition of launch 

services for all NASA-managed launches. 

Performance Measurement. Performance measurement involves a process of planning 

a goal, establishing an objective measure of actual performance, recording performance, 

evaluating actual performance against the planned goal, and reporting results. GPRA, the 

OMB, the General Accounting Office (GAO), and NASA use slightly different terms for 

performance measures. GPRA and OMB define a performance measure as a 

performance goal or performance indicator. GPRA states that a performance goal means 

a target level of performance expressed as a tangible, measurable objective, against 

which actual achievement can be compared. A performance indicator means a particular 

value or characteristic used to measure output or outcome. The GAO considers 

performance measures as synonymous with performance indicators. For performance 

measurement, the NASA FY 2003 Performance Plan uses annual performance goals and 

indicators. 

Performance Goals and Indicators. The FY 2003 Performance Plan includes two ELV 

Program performance goals and indicators (see Appendix B). NASA will use the goals 

and indicators to report on performance of the ELV Program in the FY 2003 Performance 

Report. Additionally, Kennedy has developed several performance metrics applicable to 

the ELV Program, including metrics for mission success and customer satisfaction. 

NASA Risk Mitigation Policy. NPD 8610.7 states that launch services acquired for 

deployment of NASA-owned, NASA-sponsored payloads must take advantage of all 

reasonable sources of U.S. commercial launch services and at the same time, ensure that 

taxpayer-funded spacecraft are not exposed to excessive risk. The policy states that the 

NASA launch services acquisition strategy shall seek to balance mission risk with the 

demonstrated flight history and maturity of the launch vehicle. The policy provides 

criteria for categorizing payloads, launch vehicle requirements for each category, and 

controls to mitigate risk. Appendix C provides a description of categories and control 

requirements, and Appendix D lists the other risk mitigation requirements we reviewed. 
Finding and Recommendation 

Performance Indicators in NASA Performance Plan 

NASA needs to clarify one of the two ELV Program performance goals and indicators in 

the NASA FY 2003 Performance Plan or Performance Report to better meet GPRA and 

OMB requirements. Specifically, the Office of Space Flight has not fully defined two 

key terms or the basis for the 95-percent success rate. The condition occurred because 

the Office of Space Flight viewed the NASA Performance Plan as a high-level, succinct 

document that did not require definition of every key term or the basis of the success rate. 

Without improved performance information, external stakeholders, such as the Congress 

and the public, may not be readily able to determine the relative effectiveness of the ELV 

Program and may not have confidence that the program's results justify the $36 million 

that will be spent in FY 2003 for technical management and acquisition services for the 

$338 million ELV Program. 

Federal Requirements for Performance Measurement

Both the GPRA and OMB Circular A-11 require management to define performance 

goals and indicators in an objective manner to ensure that users of performance 

information can accurately and independently determine actual performance.10 The GAO 

stated that performance goals must specifically define the results the agency expects to 

achieve for program activities in the agency's budget.11 Management must establish a 

performance goal against an agency baseline, historical trend data, or benchmark from

another organization performing similar activities. 

The GPRA and OMB Circular A-11 state that the objective of performance measurement 

is to improve three areas. First, performance measurement will improve congressional 

decision making because it provides decision makers objective information on how 

Federal programs are actually performing. Second, performance measurement will 

improve public confidence in the capability of the Federal government because reporting 

actual performance holds Federal agencies accountable for results. Third, performance 

measurement will improve service quality, because Federal managers must plan for 

results and will have the information they need to achieve results. 

The President's Management Agenda (Agenda) states that by FY 2003, Federal agencies 

must formally integrate performance review with budget decisions as part of the 

 

10The requirements are in GPRA, Section 4, and OMB Circular A-11, "Preparing and Submitting Budget 

Estimates," July 17, 2001, Sec. 220.9, "General Guidelines on Developing Performance Goals and 

Indicators." 11GAO guidance on defining performance goals and objectives is in the following: GAO/GGD-10.1-20, 

"The Results Act: An Evaluator's Guide to Assessing Agency Annual Performance Plans," April 1998, and 

GAO/GGD/AIMD 10.1-18, "Agencies' Annual Performance Plans Under the Results Act: An Assessment

Guide to Facilitate Congressional Decision-making," February 1998. 

2
management initiative called Budget and Performance Integration.12 Budget and 

Performance Integration connects goals with resources. The intent of Budget and 

Performance Integration is to use objective measures that will identify Government 

programs that are not succeeding or programs that should be reinvented, redirected, or 

retired. It should allow the Congress, the public, and agency management to calculate 

cost-effectiveness, compare different approaches to similar goals, and compare the 

program to other programs with similar goals. 

The FY 2003 Performance Plan 

For one of the performance goals and indicators for the ELV Program, the NASA FY 

2003 Performance Plan did not fully explain the terms “success” and “running average” 

and did not provide the basis for the 95-percent success rate as shown below: 

Strategic Goal 1: Explore the Space Frontier 

Performance Goal and Indicator 3H03: Provide reliable launch services for 

approved missions. 

• NASA success rate at or above a running average of 95 percent for missions 

noted on the Flight Planning Board manifest and launched pursuant to 

commercial launch service contracts. 

The Performance Plan lacked specific detail on the terms success and running average 

and on the bases for the 95-percent rate as discussed below: 

• The Performance Plan did not specifically define success or state the criteria the 

Office of Space Flight would use to determine launch success. Launch service 

contracts include the criteria management will use to determine whether a launch 

is a success or failure, but the Performance Plan did not reflect the contract 

criteria. 

• The Performance Plan did not state the timeframe for calculating the running 

average success rate. The Office of Space Flight explained that the running 

average applied to the period 198713 to the present rather than to the fiscal year. 

Also, the Performance Plan did not fully explain that the running average would 

not include launches of NASA secondary payloads.14 For example, the launch of 

the Quik Total Ozone Mapping Spectrometer (QuikTOMS) spacecraft failed on 

 

12The President's Management Agenda includes the requirement under the section titled, "Budget and 

Performance Integration." The President's Management Agenda states, "Agency performance measures 

tend to be ill defined and not properly integrated into agency budget submissions and the management and 

operations of agencies. . . ." 

13The ELV Program used a 1987 start date because that was the year NASA changed from procuring ELV 

vehicles to procuring ELV services. 14NASA secondary payloads on commercial launches receive much less NASA oversight and insight than 

NASA payloads launched pursuant to NASA launch service contracts. 

3
September 21, 2001. The ELV Program running average will exclude the launch, 

because the QuikTOMS spacecraft was a secondary payload on a commercially 

sponsored launch.

• The Performance Plan did not explain the basis for the prescribed 95-percent 

success rate. The Office of Space Flight explained that the 95-percent rate was 

based on the goal to achieve a demonstrated reliability rate at or better than the 

standard ELV design reliability.15 The Performance Plan did not indicate why the 

95-percent rate was an acceptable indicator of successful performance or how the 

rate compared to other similar organizations.

High Level Document 

The performance goal and indicator did not include fully defined key terms because the 

Office of Space Flight viewed the NASA Performance Plan as a high-level, succinct 

document that did not require the definition of every key term or number. NASA 

personnel sufficiently understood the terms success and running average and the basis of 

the 95-percent success rate. The Office of Space Flight believed that other stakeholders, 

such as congressional oversight committees and the public, were familiar with NASA 

missions and would understand the meaning of these terms without additional 

explanations. 

Better Performance Reporting for Stakeholders 

Improved performance information will allow external stakeholders, such as the 

Congress and the public, to better determine the relative effectiveness of the ELV 

Program and to better evaluate whether the program's results justify the $36 million spent 

annually for technical management and acquisition services for the program: 

• The Congress and the public will know how NASA management will evaluate 

and report a launch that is not completely successful (for example, a mission 

launched into exact orbit after using extra propellant). Including this information 

in the Performance Plan helps the Congress and the public to better evaluate 

reports of successful performance. 

• The Congress and the public will more clearly understand the calculation of the 

running average success rate. For example, external stakeholders can then 

understand why the ELV Program might have a high success rate during the same

year a launch failed. Without properly explaining how the running average 

 

15NPD 8610.7, "Launch Services Risk Mitigation Policy for NASA-Owned or NASA-Sponsored 

Payloads,” requires 95-percent reliability for launch vehicles used to launch NASA payloads. Launch

vehicles with established launch history must demonstrate the reliability, while launch vehicles with

limited history must show a 95-percent predicted design reliability. See Appendix C for further 

information on NPD 8610.7 reliability requirements. 

4
success rate is calculated, at least three different calculations (and resulting rates) 

are possible as shown below for fiscal year 2001:16 

Running Average Success Rate

Description Rate (percent) 

Years 1987 through FY 2001 

(Excluding secondary payloads) 

98.21

FY 2001 

(Excluding secondary payloads) 

100.02

FY 2001 

(Including secondary payloads) 

87.53

1

The 98.2-percent rate resulted from the ratio of 54 successful launches out of a total of 55 launches. 

The success rate for 1987 to December 2001 was 98.2-percent, or 55 successful launches out of a 

total of 56 launches, because there was one additional successful launch during the additional 3-

month period, October through December 1987. 2

The 100-percent rate resulted from the ratio of seven successful launches out of a total of seven 

launches. The calculation excluded the failed launch of the QuikTOMS, a secondary payload on a 

non-NASA launch, because it was a secondary payload. 

3

The 87.5-percent rate resulted from the ratio of seven successful launches out of a total of eight

launches. The one launch failure was QuikTOMS, a secondary payload on a non-NASA launch. 

• The Congress and the public will obtain insight into how the 95-percent success 

rate compares to NASA's historical launch performance or to launch performance 

by comparable launch operations at Government and private organizations. 

Insight into the rate will enable the Congress and the public to determine whether 

meeting the 95-percent success rate is an acceptable level of performance. 

Conclusion 

GPRA and the President's Management Agenda intend that performance information for 

the ELV Program give the Congress and the public insight for decisions and confidence 

in the results achieved for the $36 million FY 2003 budget for technical management and 

acquisition services. GPRA and OMB require agencies to state performance goals and 

indicators in an objective form to accomplish this intended purpose. In Strategic Goal 1 

and Performance Measure 3H03, the Office of Space Flight did not clearly define the 

terms success and running average and did not provide the basis of the 95-percent 

success rate. With additional explanations, external users of performance information, 

such as the Congress and the public, can better understand reports of actual performance 

and can better evaluate actual performance against the $36 million spent for the $338 

million program. 

 

16NASA will not report actual performance under the FY 2002 performance goals and indicators until after 

September 30, 2002. 

5
Recommendation, Management’s Response, and Evaluation of 

Management’s Response 

The Associate Administrator for Space Flight should direct the Assistant Associate 

Administrator for Launch Services to explain the ELV Program performance goal 

and indicator in the FY 2003 NASA Performance Plan or Performance Report to 

clarify the 

• meaning of the key terms success and running average and 

• the basis for the prescribed 95-percent rate established for mission success. 

Management’s Response. Concur. NASA will explain the ELV Program performance 

indicator in the FY 2003 NASA Performance Plan or Performance Report to clarify the 

two key terms and the basis for the prescribed 95-percent rate established for mission 

success. In addition, NASA’s FY 2002 Performance Report will include a description of 

how the noted performance indicator was achieved, including clarification of noted 

terms. It is expected that the FY 2002 NASA Performance Report will be released in 

late-January 2003 (see Appendix F). 

Evaluation of Management’s Response. Management’s planned corrective action is 

responsive to the recommendation. The recommendation is resolved but will remain 

undispositioned and open for reporting purposes until the corrective action is completed. 

6
Appendix A. Objectives, Scope, and Methodology 

Objectives

Our overall objective was to evaluate the Expendable Launch Vehicle (ELV) Program's

performance measurement system and compliance with NASA risk mitigation policy for 

launch vehicles. Specifically, we determined whether NASA did the following: 

• Established a performance measurement system for the ELV Program that 

complied with Federal requirements and other appropriate criteria. 

• Managed ELV launch services in compliance with Agency risk mitigation policy. 

Specifically, we determined whether NASA categorized payload and instrument 

risk and selected launch vehicles based on the risk category. Also, we determined 

whether NASA complied with other relevant provisions of NASA's risk 

mitigation policy, including applicable Federal and NASA clauses in contracts for 

launch services.

Scope and Methodology 

For the first objective, we interviewed the NASA Assistant Associate Administrator for 

Launch Services and other key ELV Program officials at Kennedy Space Center. We

reviewed laws, policy, procedures, and guidelines applicable to performance measures.17 

We also reviewed the ELV Program Management Plan, the ELV Program Commitment 

Agreement, ELV Launch Services Project Plan, and other key documents for information 

on performance measures. We evaluated (1) the two ELV Program performance 

measures (that is, annual performance goals and indicators) appearing in the Agency's

fiscal year 2002 and 2003 Annual Performance Plans and (2) the four performance 

metrics Kennedy Space Center (Kennedy) uses to measure ELV Program performance. 

The two performance measures are in Appendix B, and the four Kennedy metrics we 

evaluated were Mission Success, On-Time NASA launches, On-Time Processing, and 

External Customer Satisfaction. We also reviewed benchmarks NASA used for 

evaluating the ELV Program's performance; procedures NASA management will use to 

gather, validate, and report performance data for FY 2002 and subsequent years; and 

support for Kennedy performance metrics. 

For the second objective, we interviewed the NASA Assistant Associate Administrator 

for Launch Services, the Kennedy ELV Program Deputy Program Manager, and other 

key ELV Program and procurement officials at Kennedy. We reviewed NASA Policy 

Directive (NPD) 8610.7, "Launch Services Risk Mitigation Policy for NASA-Owned or 

 

17Specifically, we reviewed the Government Performance and Results Act (GPRA), Office of Management

and Budget Circular A-11 ("Preparing and Submitting Budget Estimates") sections relating to performance 

plans and reports, and two General Accounting Office guides for preparing and evaluating Federal

agencies' annual performance plans. 

7
Appendix A 

NASA-Sponsored Payloads"; the ELV Program Management Plan; the ELV Program

Commitment Agreement; the ELV Launch Services Project Plan; and other key 

management documents. The scope included a universe of 27 ELV payloads scheduled 

for launch from 2001 to 2003. From the universe of 27 launches, we selected 918 for 

further review. For the nine selected launches, we asked the Assistant Associate 

Administrator for Launch Services to provide information on the cost, complexity, and 

risk category of each payload and the risk category of the launch vehicle selected for 

each. We used that information to evaluate NASA compliance with NPD 8610.7 launch 

vehicle selection requirements. We did not evaluate whether the subject launch vehicles 

met the technical certification requirements for their respective risk categories. See 

Appendix C for further information on the NPD 8610.7 risk categories and suitable 

launch vehicles. We also reviewed other risk mitigation controls contained in NPD 

8610.7: (1) NASA contract insight provisions, (2) NASA Spacecraft Announcements of 

Opportunities, 

(3) the launch service task order (solicitations) process, and (4) records of vehicle history 

and reliability statistics. See Appendix D for further information on these NPD 8610.7 

requirements. 

We did not test computer-processed data, because we did not rely on the data to achieve 

our objectives. 

Management Controls Reviewed 

As part of our evaluation of performance measurement, we reviewed (1) management 

controls over the development of performance goals and indicators included in the 

FY 2003 Performance Plan to ensure their compliance with the Government Performance 

and Results Act and Office of Management and Budget requirements; (2) procedures 

management will use to validate, verify, and report performance data for the FY 2002 

Performance Plan; and (3) metrics being developed to evaluate the performance of 

operations and organizations that provide launch services. Controls were adequate 

except for the matters discussed in the finding section of the report. 

As part of our evaluation of risk mitigation, we reviewed compliance with risk mitigation 

controls on launches covered by NPD 8610.7 and on two launches not covered by NPD 

8610.7 that experienced problems. For the seven launches we reviewed, procedures and 

practices met control requirements outlined in NPD 8610.7. 

 

18The nine launches we reviewed consisted of (1) seven launches procured under the Small ELV Services 

and NASA Launch Services contracts, to which NPD 8610.7 applies, and (2) two other launches that

experienced problems. The two problem launches included the Quik Total Ozone Mapping Spectrometer 

mission's launch, which failed, and the Kodiak Star mission's launch, which was delayed several times but

was eventually successful. 

8
Appendix A 

Audit Field Work 

We conducted field work from November 2001 through May 2002 at NASA 

Headquarters and Kennedy. We performed the audit in accordance with generally 

accepted Government auditing standards.

9
Appendix B. ELV Program Performance Goals and Indicators 

Excerpted From the NASA FY 2003 Performance Plan 

Human Exploration and Development of Space (HEDS) Enterprise 

Access to Space (Expendable Launch Vehicles and Payloads) 

NASA Strategic Goals and Objectives Annual Performance Goals 

and Indicators 

Goal 1 - Explore the Space Frontier 

Objective: Enable human exploration through 

collaborative robotic missions. 

Public Benefit: A better understanding (at the 

earliest possible dates) of the space and planetary 

environments to which human explorers will one 

day travel will make possible a more focused, 

more effective and lower cost investment to 

develop the technologies needed for future human 

and robotic exploration and development of space. 

This knowledge and understanding will also make 

possible reduced risks to the health and safety of 

future astronauts. Overall, pursuing collaborative 

robotic missions will result in future human and 

robotic exploration missions with lower costs and 

greater benefits that would be otherwise 

achievable. HEDS supports this strategic 

objective by working collaboratively with other 

enterprises on advanced planning activities and 

providing launch services supporting NASA 

sponsored missions including robotic spacecraft 

missions. 

3H03: Provide reliable launch 

services for approved missions. 

• NASA success rate at or 

above a running average of 

95 percent for missions 

noted on the Flight Planning 

Board manifest and 

launched pursuant to 

commercial launch service 

contracts. 

Goal 3 - Enable the Commercial Development of

Space 

Objective: Improve the accessibility of space to 

meet the needs of commercial research and 

development. 

Public Benefit: New commercially developed 

launch services will be able to compete for NASA 

launches when they meet NASA’s risk mitigation 

policy. 

3H17: Establish mechanisms to 

enable NASA access to the use of 

U.S. commercially developed 

launch systems. 

• Assure that NASA launch 

service contracts include 

annual on-ramps for newly 

developed commercial 

launch services as they meet 

NASA's risk mitigation 

policy. 

10
Appendix C. Payload/Instrument Risk Categories in NPD 8610.7 

To balance mission risk with launch vehicle demonstrated history and maturity, NASA 

Policy Directive 8610.7, "Launch Services Risk Mitigation Policy for NASA-Owned or 

NASA-Sponsored Payloads," divides payloads and instruments into the following three 

risk categories and describes suitable launch vehicles for each category. 

Risk Category 1 

• Payloads/Instruments - Applies to payloads and instruments that are nonmission critical, require simple interface and mission design, and/or are low cost 

(for example, university research experiments or instrumentation). 

• Suitable Launch Vehicle - Applies to a launch vehicle such as (1) a new launch 

vehicle with no previous flight history or (2) the first flight of a new configuration 

of a flight-proven launch vehicle incorporating a major system upgrade. Launch 

vehicle must achieve 95-percent predicted design reliability. 

Risk Category 2 

• Payloads/Instruments - Applies to missions that are collectively mission critical 

to the implementation of NASA's Strategic Plan, require moderate complexity of 

mission design and integration, and/or are of moderate cost. 

• Suitable Launch Vehicle - Mission may launch on (1) NASA-acquired launch 

services from qualified suppliers with at least one fully successful launch of a 

common vehicle or (2) Kennedy Space Center (Kennedy) verification that the 

demonstrated common vehicle configuration met predicted vehicle and 

performance parameters. Launch vehicle must achieve 95-percent predicted 

design reliability. 

Risk Category 3 

• Payloads/Instruments - Applies to missions that are individually mission critical 

to implementation of NASA's Strategic Plan, require complex mission interface or 

design, and/or are of high cost. 

 

• Suitable Launch Vehicle - Missions may launch on (1) NASA-acquired launch 

services from qualified suppliers with a demonstrated flight record consisting of a 

series of consecutive successful launches of a common vehicle configuration (that 

is, 95-percent reliability at 50-percent confidence level); (2) Kennedy verification 

that the common vehicle configuration has been verified to meet predicted vehicle 

11
Appendix C 

and performance parameters; or (3) an on-orbit services contract utilizing services 

from a qualified supplier that will be considered on a case-by-case basis for analysis 

between the mission Enterprise Associate Administrator and the Office of Space 

Flight. To prove the 95-percent reliability at a 50-percent confidence level, the 

launch vehicle must achieve a series of 14 or more consecutive successful flights of 

common vehicle configuration.

12
Appendix D. Risk Mitigation Requirements in NPD 8610.7 

Our review included the following NASA Policy Directive 8610.7, "Launch Services 

Risk Mitigation Policy for NASA-Owned or NASA-Sponsored Payloads," risk mitigation 

requirements: 

Section 1.c.(5)

Any NASA contracts which include the launch services as an 

integrated mission service (e.g., on-orbit or turnkey service) for a 

NASA-funded mission shall include a provision for NASA insight into

launch contractor systems engineering, processes and process control

to ensure quality and reliability of launch services, and consistency 

with this policy. 

Section 5.b.

The Enterprise Associate Administrators are responsible for assuring 

that all spacecraft Announcement of Opportunities and Request for 

Proposals are coordinated with the Office of Space Flight for 

consistency with this policy prior to publication.

Section 5.c.

The Kennedy Space Center Director is responsible for ensuring that all 

launch services solicitations for NASA-owned or NASA-sponsored 

payloads are consistent with this policy and coordinated with the 

Office of Space Flight (OSF). 

Section 7.a.

OSF will maintain a record of vehicle flight history and reliability 

statistics for all U.S. and foreign launch vehicle suppliers. 

13
Appendix E. Verification and Validation of Performance Data 

The FY 2003 Performance Plan describes the following verification and validation 

procedures: 

Verification and Validation 

Internal Assessment 

Interim evaluation and monitoring of performance targets will be

conducted -- as required -- as an element of regular meetings of the 

Office of Space Flight and HEDS [Human Exploration and 

Development of Space] Management Boards. 

Final data collection, reporting and verification for inclusion in

NASA's Annual Performance Report will rely on several different 

processes depending on the particular Annual Performance Goal. 

Wherever possible, a specific tangible product has been identified in

the indicator for individual performance goals to strengthen the 

validation process. 

For many HEDS performance goals, (e.g. Space Shuttle in-flight 

anomalies, International Space Station assembly milestones) 

verification of performance is straightforward and progress is

monitored through regular management channels and reports. 

External Assessment 

To assist in evaluating those performance goals that are more difficult 

to associate with specific tangible products, HEDS will employ an 

annual external assessment process. Past external assessors have 

included the: NASA Advisory Council, Space Flight Advisory 

Committee, General Accounting Office, NASA's Office of Inspector 

General, and National Research Council.

. 

14
Appendix F. Management’s Response 

15
Appendix G. Report Distribution

Note: This list shows the distribution of the final report only. The list does not apply to 

the draft report. 

National Aeronautics and Space Administration (NASA) Headquarters 

HQ/A/Administrator 

HQ/AI/Associate Deputy Administrator 

HQ/AA/Chief of Staff 

HQ/B/Acting Deputy Chief Financial Officer 

HQ/B/Comptroller 

HQ/BF/Director, Financial Management Division 

HQ/G/General Counsel 

HQ/H/Assistant Administrator for Procurement 

HQ/HK/Director, Contract Management Division 

HQ/HS/Director, Program Operations Division 

HQ/J/Assistant Administrator for Management Systems 

HQ/JM/Director, Management Assessment Division 

HQ/L/Assistant Administrator for Legislative Affairs 

HQ/M/Associate Administrator for Space Flight 

HQ/S/Associate Administrator for Space Science 

HQ/Y/Associate Administrator for Earth Science 

NASA Centers 

KSC/AA/Director, John F. Kennedy Space Center 

KSC/CC/Chief Counsel, John F. Kennedy Space Center 

Non-NASA Federal Organizations and Individuals 

Assistant to the President for Science and Technology Policy 

Deputy Associate Director, Energy and Science Division, Office of Management and 

 Budget 

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office 

 of Management and Budget 

Managing Director, Acquisition and Sourcing Management Team, General Accounting 

 Office 

Senior Professional Staff Member, Senate Subcommittee on Science, Technology, and 

 Space 

16
Appendix G 

Chairman and Ranking Minority Member – Congressional Committees and 

Subcommittees 

Senate Committee on Appropriations 

Senate Subcommittee on VA, HUD, and Independent Agencies 

Senate Committee on Commerce, Science, and Transportation 

Senate Subcommittee on Science, Technology, and Space 

Senate Committee on Governmental Affairs 

House Committee on Appropriations 

House Subcommittee on VA, HUD, and Independent Agencies 

House Committee on Government Reform

House Subcommittee on Government Efficiency, Financial Management, and 

 Intergovernmental Relations 

House Subcommittee on Technology and Procurement Policy 

House Committee on Science 

House Subcommittee on Space and Aeronautics 

Congressional Member 

Honorable Pete Sessions, U.S. House of Representatives 

17
NASA Assistant Inspector General for Audits 

Reader Survey 

The NASA Office of Inspector General has a continuing interest in improving the 

usefulness of our reports. We wish to make our reports responsive to our customers’ 

interests, consistent with our statutory responsibility. Could you help us by completing 

our reader survey? For your convenience, the questionnaire can be completed 

electronically through our homepage at http://www.hq.nasa.gov/office/oig/hq/audits.html

or can be mailed to the Assistant Inspector General for Audits; NASA Headquarters, 

Code W, Washington, DC 20546-0001. 

Report Title: Expendable Launch Vehicle Performance Measures

Report Number: Report Date: 

Circle the appropriate rating for the following statements. 

Strongly 

Agree Agree Neutral Disagree 

Strongly 

Disagree N/A 

1. The report was clear, readable, and logically 

organized. 

5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A 

3. We effectively communicated the audit 

objectives, scope, and methodology. 

5 4 3 2 1 N/A 

4. The report contained sufficient information to

support the finding(s) in a balanced and 

objective manner. 

5 4 3 2 1 N/A 

Overall, how would you rate the report? 

# Excellent # Fair

# Very Good # Poor

# Good

If you have any additional comments or wish to elaborate on any of the above 

responses, please write them here. Use additional paper if necessary. 

18
How did you use the report?

How could we improve our report?

How would you identify yourself? (Select one) 

# Congressional Staff # Media 

# NASA Employee # Public Interest 

# Private Citizen # Other:

# Government: Federal: State: Local:

May we contact you about your comments? 

Yes: ______ No: ______ 

Name: ____________________________ 

Telephone:_________________________

___________________

Thank you for your cooperation in completing this survey. 

. 

19
Major Contributors to the Report 

Daniel J. Samoviski, Program Director, Earth/Space Science Audits 

Nora E. Thompson, Program Manager 

James E. Richards, Auditor-in-Charge 

Nancy C. Cipolla, Report Process Manager 

Iris T. Purcarey, Program Assistant