IG-00-029

AUDIT

REPORT X-34 TECHNOLOGY DEMONSTRATOR

March 30, 2000

National Aeronautics and

Space Administration

OFFICE OF INSPECTOR GENERAL
Additional Copies

To obtain additional copies of this report, contact the Assistant Inspector General for Auditing at

(202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General for

Auditing. Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing

Code W

NASA Headquarters

Washington, DC 20546-0001

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800) 424-9183,

(800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form; or write to the

NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, Washington, DC 20026. The

identity of each writer and caller can be kept confidential, upon request, to the extent permitted

by law.

Reader Survey

Please complete the reader survey at the end of this report or at

http://www.hq.nasa.gov/office/oig/hq/audits.html.

______________________________________________________________________________________________

Acronyms

FY Fiscal Year

GAO General Accounting Office

GPRA Government Performance and Results Act

ISTP Integrated Space Transportation Plan

NPD NASA Policy Directive

NPG NASA Policy Guidance

OIG Office of Inspector General

OMB Office of Management and Budget

RLV Reusable Launch Vehicle

TRL Technology Readiness Level
W March 30, 2000

TO: A/Administrator

FROM: W/Inspector General

SUBJECT: INFORMATION: Audit of the X-34 Technology Demonstrator

Report Number IG-00-029

The NASA Office of Inspector General has completed an audit of the X-34 Technology

Demonstrator, one of NASA's faster, better, cheaper projects. The audit was completed prior to

the current X-34 replanning effort, which NASA initiated in early 2000 to improve X-34 risk

mitigation. We reviewed the status of the X-34 Technology Demonstrator Project in meeting

technology requirements for the next-generation Reusable Launch Vehicle (RLV), and, as

necessary, reviewed strategic planning for Space Transportation and the role of the X-34. We

concluded that the Agency needs to improve management of Space Transportation technologies

at the Agency, Enterprise, Center, and program/project levels. Specifically, NASA has not

adequately performed strategic planning for the Space Transportation mission and needs to better

state the next-generation RLV technology requirements. The Office of Aero-Space Technology

has not issued an Enterprise Strategic Plan for the Space Transportation mission. Also, the

Marshall Space Flight Center (Marshall) has not established mission-specific requirements for

each of the 27 currently planned X-34 flights, nor properly documented, with a cost/benefit

analysis, the requirement for these flights or the numerous changes to the proposed flight test

program. Finally, basic program documentation for the X-34 Project Plan and the FutureX/Pathfinder Program has not been finalized and approved in a timely manner.

Without adequate strategic planning, NASA has less assurance that the technologies resulting

from the X-34 Project will effectively support RLV needs. This deficiency also may extend the

time required to develop needed technology, delaying realization of the launch savings

anticipated from the next-generation RLV. Unless flight test requirements are properly

documented there is no assurance the appropriate number of test flights are planned or that the

test program content is the most cost-effective way to meet project objectives. Also, delays in

preparing program documentation can adversely impact Agency management of technologies

needed to reduce the cost of access to space and delay establishing approved program/project

objectives and concepts to be used to manage these high-priority, costly programs.
2

Background

The next-generation RLV concept is an attempt to reduce the cost of access to space. The

original RLV was the Space Shuttle. NASA is now looking towards a second-generation RLV to

reduce launch costs. The X-34 Project is one of the three original “stepping stones” (DC-XA,

X-34, and X-33) in Marshall’s RLV Program and is the first in a series of planned Pathfinder1

class technology demonstrators managed by Marshall’s new Future-X/ Pathfinder Program

Office. Marshall manages the X-34 as part of its responsibility as the Lead Center for Space

Transportation. The NASA Headquarters Office of Aero-Space Technology manages the overall

Agency Space Transportation mission. The estimated cost of the present X-34 Project (including

the $18.6 million cost of the Fastrac engine2 and approximately $2 million in experiments) totals

about $186 million as of November 1999, including the option exercised in January 1999 for 25

additional flights. Project costs may increase further as a result of the current replanning effort.

During our review, NASA continued to work on a Space Transportation Architecture Study,

including the latest phase, the Integrated Space Transportation Plan. Study results should be

incorporated into the Aero-Space Technology Enterprise Strategic Plan. This study is being done

in response to 1996 and 1997 Office of Management and Budget concerns regarding how NASA

planned to reduce launch costs. Enterprise officials told us that during fiscal year 2000, they will

perform extensive Space Transportation systems analyses to continue establishing the technology

requirements for the next-generation RLV.

Recommendations

We recommended that the Agency correct deficiencies in the Agency Strategic Plan and issue an

Enterprise Strategic Plan and Lead Center Implementation Plan addressing space transportation

that defines technology requirements, establishes an implementing strategy including technology

metrics, and implements an appropriate cost/benefit analysis process. We recommended that

Marshall revalidate X-34 flight test requirements to include the number and type of flights and

the number of engines, delete any unneeded flights or engines, cancel plans for the expanded

flight test program, and properly document X-34 management decisions. We also recommended

that NASA place added emphasis on (1) compliance with Agency program documentation

requirements to include promptly completing X-34 and Future-X/Pathfinder documentation and

(2) on establishment of internal control procedures to ensure the timely preparation of future

program documentation.

 

1 Pathfinder class technology demonstrators are technology-focused flight vehicle projects within the FutureX/Pathfinder Program and generally cost less than $100 million each.

2

 Marshall is developing the Fastrac engine in-house, with contractor support, using currently available technology.

Initiated in 1996, separately from the X-34 Project, Fastrac is to reduce launch costs by providing a low-cost booster

for small payloads. Fastrac will be the main propulsion for the X-34 vehicle.
3

Management's Response and OIG Evaluation

Management concurred with all 16 recommendations, agreeing to implement recommended

actions, which should significantly improve the overall effectiveness of Agency management of

Space Transportation programs and projects. The actions should also help ensure that Agency

and Enterprise Strategic Plans comply with Agency directives and more effectively address

required technologies, that flight programs cost-effectively meet X-34 needs, and that basic

program documentation is promptly finalized and approved. A summary of the status of all the

recommendations is in the Executive Summary of the report.

[Original signed by]

Roberta L. Gross

Enclosure

 Final Report on the Audit of the X-34 Technology Demonstrator
FINAL REPORT

AUDIT OF THE X-34 TECHNOLOGY DEMONSTRATOR
W March 30, 2000

TO: R/Associate Administrator for Aero-Space Technology

DA01/Director, Marshall Space Flight Center

FROM: W/Assistant Inspector General for Auditing

SUBJECT: Final Report on the Audit of X-34 Technology Demonstrator

Assignment Number A-HA-98-050

Report Number IG-00-029

The subject final report is provided for your information and use. Please refer to the Executive

Summary for the overall audit results. Our evaluation of your response is incorporated into the

body of the report. The corrective actions taken or planned for recommendations 1 through 7, 9,

10, 11, 13, 14, and 15 were responsive, but the recommendations will remain open for reporting

purposes until corrective action is completed. Please notify us when corrective action has been

completed on these recommendations, including the extent of testing performed to ensure

corrective actions are effective. The corrective actions completed on recommendations 8, 12,

and 16 were responsive, and those recommendations are considered closed for reporting

purposes.

If you have questions concerning the report, please contact Ms. Karen VanSant, Audit Program

Director, Aero-Space Technology Audits, at (256) 544-1149, or Mr. Jim Linville,

Auditor-in-Charge, at (256) 544-0971. We appreciate the courtesies extended to the audit staff.

The final report distribution is in Appendix J.

[Original signed by]

Russell A. Rau

Enclosure
2

cc:

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

JM/Director, Management Assessment Division
3

bcc:

AIGA, IG, Reading Chrons

AE/Audit Liaison Representative

HK/Audit Liaison Representative

RS/Audit Liaison Representative

Marshall/RS40/Audit Liaison Representative
Contents __________________ 

Executive Summary, i

Introduction, 1

Findings and Recommendations, 3

Finding A. Strategic Planning for Space Transportation, 3

Finding B. X-34 Project Flight Test Requirements, 12

Finding C. Documentation for Space Transportation Programs/Projects, 18

Appendix A - Objectives, Scope, and Methodology, 24

Appendix B - Background, 27

Appendix C - Federal and Agency Requirements, 29

Appendix D - Congressional, OMB, and GAO Concerns, 31

Appendix E - Metrics for Acquiring Needed Technologies, 33

Appendix F - Agency Strategic and Performance Plans, 35

Appendix G - Space Transportation Architecture Study, 37

Appendix H - Program Documentation Requirements, 38

Appendix I – Management’s Response, 40

Appendix J - Report Distribution, 51
NASA Office of Inspector General

IG-00-029 March 30, 2000

 A-HA-98-050 

X-34 Technology Demonstrator

Executive Summary

Background. The next-generation Reusable Launch Vehicle (RLV) concept is an attempt

to reduce the cost of access to space. The original RLV was the Space Shuttle. NASA is

now looking towards a second-generation RLV to reduce launch costs. The X-34 Project

is one of the three original “stepping stones” (DC-XA, X-34, and X-33) in Marshall

Space Flight Center’s (Marshall’s) RLV Program and is the first in a series of planned

Pathfinder3

 class technology demonstrators managed by Marshall’s new Future-X/

Pathfinder Program Office. Marshall manages the X-34 as part of its responsibility as the

Lead Center for Space Transportation. The NASA Headquarters Office of Aero-Space

Technology manages the overall Agency Space Transportation mission. The estimated

cost of the present X-34 Project (including the $18.6 million cost of the Fastrac engine4

and approximately $2 million in experiments) totals about $186 million as of November

1999, including the option exercised in January 1999 for 25 additional flights. Marshall

is contemplating additional experiments and vehicle tests.

Objectives. The overall objective was to assess the status of the X-34 Technology

Demonstrator Project, through fiscal year (FY) 1999, in meeting technology requirements

for the next-generation RLV. To evaluate NASA’s planned use of X-34 technologies, it

was necessary that our review also address strategic planning for Space Transportation

and the role the X-34 was expected to play in meeting Agency Space Transportation

technology requirements. Appendix A contains details on the objectives, scope, and

methodology used for this audit.

Results of Audit. Management of Space Transportation technologies needs

improvement at the Agency, Enterprise, Center, and program/project levels as illustrated

below:

Strategic Planning. NASA has not adequately performed strategic planning for

the Space Transportation mission. Specifically, improvements are needed at all

 

3 Pathfinder class technology demonstrators are technology-focused flight vehicle projects within the

Future-X/Pathfinder Program and generally cost less than $100 million each.

4

 Marshall is developing the Fastrac engine in-house, with contractor support, using currently available

technology. Initiated in 1996, separately from the X-34 Project, Fastrac is to reduce launch costs by

providing a low-cost booster for small payloads. Fastrac will be the main propulsion for the X-34 vehicle.
ii

levels in preparing effective strategic plans and in procedures for managing those

technologies (including the X-34) needed to develop the next-generation RLV and

to reduce launch costs. Strategic planning deficiencies may result in the Agency

not receiving technology benefits from the X-34 Project and may extend the time

required to develop needed technology, delaying realization of launch savings

anticipated from the next-generation RLV (see Finding A).

Flight Test Requirements. Marshall has not established mission-specific

requirements for each of the 27 currently planned X-34 flights, including the

option exercised prematurely by Marshall for 25 additional flights. Marshall has

not properly documented, with a cost/benefit analysis, the requirement for these

flights or the numerous changes to the proposed flight test program. Similarly,

Marshall also has not justified the need for an additional 7-10 flights proposed as

an expanded flight test program. In the absence of necessary documentation, there

is no assurance the appropriate number of test flights are planned or that the test

program content is the most cost-effective way to meet project objectives.

Canceling the additional flights will save $7 to10 million (see Finding B).

Program Documentation. Basic program documentation (to include the X-34

Project Plan and the Future-X/Pathfinder Program Commitment Agreement and

Program Plan), which is required prior to program implementation, had not been

finalized and approved, although the Project and Program were initiated in 1995

and 1997, respectively. Drafts of the program documentation lacked necessary

information to effectively implement Agency strategic plans. This delayed

establishing approved program/project objectives and concepts to be used to

manage these high-priority, costly programs and delayed formally defining the

roles and responsibilities of NASA officials. Delays in preparing program

documentation can adversely impact Agency management of technologies needed

to reduce the cost of access to space, a key NASA goal (see Finding C).

Recommendations. NASA management should:

• Correct deficiencies in the Agency Strategic Plan and issue an Enterprise Strategic

Plan and Lead Center Implementation Plan addressing space transportation that

defines technology requirements, establishes an implementing strategy including

technology metrics, and implements an appropriate cost/benefit analysis process.

• Revalidate X-34 flight test requirements to include the number and type of flights

and the number of engines. Delete any unneeded flights or engines, cancel plans

for the expanded flight test program, and properly document X-34 management

decisions.

• Place added emphasis (1) on compliance with Agency program documentation

requirements to include promptly completing X-34 and Future-X/ Pathfinder
iii

documentation, and (2) on establishment of internal control procedures to ensure

timely preparation of future program documentation.

Management's Response. Management concurred with all 16 recommendations,

agreeing to implement recommended actions, which should significantly improve the

overall effectiveness of Agency management of Space Transportation programs and

projects. The actions should help ensure that Agency and Enterprise Strategic Plans

comply with Agency directives and more effectively address required technologies, that

flight programs cost-effectively meet X-34 needs, and that basic program documentation

is promptly finalized and approved.

Evaluation of Management's Response. Management actions taken and planned are

responsive to all of the recommendations.
Introduction

The X-34 Project is part of the Future-X/Pathfinder Program within NASA’s Space

Transportation mission. The Headquarters Aero-Space Technology Enterprise and Marshall share

responsibility for preparing strategic plans for the Space Transportation mission. The strategic

planning process establishes the long-term direction for this mission and includes a vision of the

future, goals, objectives, and implementing strategies.5 During our review, NASA continued to

work on a Space Transportation Architecture Study, including the latest phase, the Integrated

Space Transportation Plan. Study results should be incorporated into the Aero-Space

Technology Enterprise Strategic Plan when completed. This study is being done in response to

1996 and 1997 Office of Management and Budget (OMB) concerns regarding how NASA

planned to reduce launch costs. Enterprise officials told us that during FY 2000 they will perform

extensive Space Transportation systems analyses to continue establishing the technology

requirements for the next-generation RLV. Additional background information is in Appendix

B.

In 1995, Marshall selected and awarded a cooperative agreement6 to the team of Orbital Sciences

Corporation (Orbital) and Rockwell International, Inc., to design, build, and fly the X-34

unmanned test vehicle. The proposed vehicle was to be an orbital spacecraft that could be put

into commercial production upon completion of the demonstration program. However, in early

1996 the industry partners withdrew from the partnership due to a number of factors including

changes in the projected business profitability. Shortly thereafter, on March 27, 1996, Marshall

issued NASA Research Announcement 8-147 and subsequently awarded Orbital a firm-fixedprice contract for a smaller, suborbital X-34 vehicle that is similar in some respects to Orbital’s

Pegasus expendable launch vehicle, but incorporates new technologies. The basic contract

provided for a test article (A1), one flight vehicle (A2), and only two flights at a total cost of

about $50 million, excluding the cost of the Fastrac engine.8

 However, in January 1998, NASA

exercised options for a Characterization and Validation effort9

 and a third flight test vehicle

(A3).10 Subsequently, in January 1999, Marshall exercised an option for an additional flight

program (25 more flights).

 

5

 NASA’s long-term goal to provide safe and affordable space travel to enable research and human expansion relies

on the development of a reusable launch vehicle and/or significant improvements to the Space Shuttle.

6

 Under the terms of the cooperative agreement partnership, NASA and the industry partners would each provide half

(50/50) of the $150 million estimated cost.

7

 The NASA Research Announcement solicited proposals for applied research that offered new and innovative

solutions to enhance key technologies applicable to future RLV’s. The proposals could address a demonstrator

vehicle, technology test bed, or RLV technology flight experiments.

8

 Marshall developed the Fastrac engine in-house for small payload launchers.

9

 The Characterization and Validation Effort is intended to provide risk mitigation to the X-34 Project. It provides

for additional hardware, including a second powered vehicle, permitting early validation of critical X-34 systems and

demonstration of vehicle performance and maturity.

10 The first X-34 vehicle, A1, was originally a test article and was not scheduled to fly. However, modified as the

A1A vehicle, it will be used for ground tests, including tow tests, and will perform the unpowered portion of the

X-34 flight tests using the wing and other components “borrowed” from the A3 vehicle.
2

The X-34 Project will demonstrate technologies necessary for a next-generation RLV. The

present X-34 vehicle is a suborbital, rocket-powered, Mach-8-capable flight demonstrator testbed

intended to enhance U.S. commercial space launch competitiveness through the development and

demonstration of key technologies applicable to future RLV’s. The X-34 will demonstrate

flexible integration capability and operability to include a high-flight rate,11 autonomous flight

operations, safe abort capability, landing in cross winds up to 20 knots, flights through rain and

fog, and a recurring flight cost of about $500,000.12

Several aspects of the X-34 have undergone change since the X-34 Project (formerly “Program”)

was initiated in 1995. These continual changes have had a significant effect on the X-34

Project’s cost, schedule, and potential results. For example, although the X-34 first flight (current

project) was to have occurred on November 21, 1998, the first flight is now planned for March

2000 or later. The proposed X-34 flight test program was still undergoing change (flight

trajectories, etc.) as of November 1999. It is anticipated there will be still more changes in the

Project.

 

11 The flight rate is equivalent to 25 flights per year and 2 flights within 24 hours.

12 The $500,000 does not include NASA in-house costs, range costs, or certain contractor costs. Actual cost is

estimated to be more than $1 million per flight.
3

Findings and Recommendations _______________ 

Finding A. Strategic Planning for Space Transportation

NASA has not adequately performed strategic planning for the Agency’s Space Transportation

mission, which includes the X-34 Project. Specifically, the Office of Aero-Space Technology

and Marshall have not: (1) defined technology requirements qualitatively and quantitatively for

the next-generation RLV; (2) established appropriate implementing strategies to include metrics,

performance indicators, and an evaluation process for measuring and reporting technology

progress; (3) routinely performed cost/benefit analyses of Space Transportation technology

initiatives and alternatives; or (4) prepared essential strategic planning documents. Neither the

Office of Aero-Space Technology nor Marshall adequately followed Agency requirements

regarding strategic planning. Strategic planning deficiencies may result in the Agency receiving

technology from the X-34 Project that will not be used and may extend the time required to

develop needed technology and/or delay realization of the launch savings anticipated from the

next-generation RLV.

Federal and Agency Requirements

Government Performance and Results Act. The Government Performance and Results Act

(GPRA), passed by Congress in 1993, requires Federal agencies including NASA to develop an

agency 5-Year Strategic Plan and update it at least every 3 years and to prepare an annual agency

Performance Plan and submit an agency Performance Report. NASA prepared the Agency’s first

GPRA Performance Plan in September 1998 (for FY 1999). NASA’s first annual GPRA

Performance Report (for FY 1999) is due to the President and Congress on March 31, 2000. A

key GPRA requirement is that performance plans include appropriate performance targets that

are measurable for the fiscal year reported. Details on GPRA requirements are in Appendix C.

NASA Strategic Management Handbook. NASA’s Strategic Management Handbook (NASA

Procedures and Guidelines (NPG) 1000.2), dated October 1996, documents the Agency’s

policies, processes, guidelines, and responsibilities for strategic management as required by

GPRA. The Handbook requires that all Agency strategic plans, the NASA Strategic Plan,

Enterprise Strategic Plans, and lower level strategic planning documents include specific

elements to ensure consistency and meet GPRA requirements. Agency strategic plans articulate

what NASA does, who its customers are, NASA’s goals, and how it will achieve them. Agency

strategic plans should have mission-critical but achievable goals, specific and measurable

objectives, implementing strategies describing how the goals and objectives will be achieved,

performance targets and appropriate metrics for measuring progress. The Handbook states that

Enterprise strategic plans must be reviewed annually and updated as necessary. Lower level

strategic planning documents such as Center Implementation Plans, supported by

program/project plans and Center of Excellence or Lead Center Plans, should prescribe

milestones, resource requirements, schedules, and performance criteria at both the program and

task level. Details on the NASA Strategic Management Handbook are in Appendix C.
4

NASA Strategic Planning

At the Agency, Enterprise, and Center levels, strategic planning documents for the Agency’s

Space Transportation mission did not adequately comply with Agency requirements, and some

key Enterprise and Lead Center plans had not been prepared. There has been continuing

congressional, OMB, and General Accounting Office concerns regarding the adequacy of

NASA’s strategic plans for Space Transportation. (See Appendix D for additional details on

these concerns.) The NASA Strategic Management Handbook states that “NASA must be able

to monitor its success in achieving each objective.” Space Transportation technology

management extensively involves the development of hardware (engines, composite structures,

composite/cryogenic fuel tanks, thermal protection systems, avionics, etc.) for new

boosters/launchers. NASA has been performing hardware research and development for 4

decades. Therefore, the Aero-Space Technology Enterprise and Marshall should have

established more definitive strategic plans that included specific data on the technologies

required to reduce launch costs and appropriate metrics to measure technology progress. The

NASA Technology Plan, dated December 1998, did include some broad technology requirements

and technology “challenges” for the Space Transportation mission. However, they were not

included in the strategic plans.

The handbook establishes requirements for strategic planning documents and requires

roadmaps,13 specific objectives, and implementing strategies to include performance targets as

well as metrics and an evaluation process to assess progress towards meeting objectives.

However, strategic planning documents for Space Transportation do not include specific

information on the technology requirements, including the technology “gap”,14 necessary to

significantly reduce launch costs or appropriate metrics to measure technology progress.

Appropriate metrics and an evaluation process for measuring and reporting annual progress are

not established and reflected in strategic plans for Space Transportation. It is essential that

appropriate technology metrics be established for each RLV core technology15 requirement.

Although existing planning documents frequently use the term “technology,” none used the

standard NASA metric, “Technology Readiness Level,”16 to identify current technology status, to

define the needed technology requirement, or to report technology progress. (Additional

information on metrics is in

Appendix E). NASA’s Space Science and Earth Science Enterprises used the term Technology

Readiness Level (TRL) in their Technology Strategy documents for 1998 or 1999. Although the

Office of Aero-Space Technology did not use TRL's in strategic planning documents for Space

Transportation, it did use TRL's as a metric in managing aviation technologies in the Level II

Plan for the Advanced Air Transportation Technologies Project, dated May 1999.

 

13 The Handbook requires that NASA and Enterprise strategic plans include a one page “Strategic Roadmap.”

Roadmaps are to show how NASA plans to attain goals, objectives, and milestones.

14 Difference between current technology capability and the more advanced technology requirement.

15 RLV core technologies include avionics, composite structures, composite/cryogenic tanks, thermal protection

systems, propulsion, and aircraft-like operations.

16 Per the NASA Technology Plan, dated December 1998, the metric Technology Readiness Level specifies the

maturity of a technology from 1 (not mature) through 9 (mature).
5

Deficiencies in the Agency, Enterprise, and Center strategic planning documents included the

following:

• Objectives in the 1998 and 1999 strategic plans generally meet the criteria of the handbook,

but were not measurable due to the lack of appropriate implementing strategies.

• Implementing strategies did not:

• state how the goals and objectives will be achieved,

• provide an investment strategy that addressed resource requirements, and

• provide metrics that identify performance indicators or the evaluation process to be used

to measure progress.

As a result of the deficiencies, NASA is unable to adequately monitor its success overall and the

contribution of the X-34 Project was not clear.

Agency Level. NASA Strategic Plans for FYs 1996, 1998,17 and 1999,18 and the NASA FY

1999 Performance Plan did not adequately define the role of the X-34 in meeting Space

Transportation technology requirements for the next-generation RLV. The Plans, prepared by the

Office of Aero-Space Technology, do not state how, when, or if the technologies addressed by

the X-34 Project will be used to satisfy Space Transportation technology needs. The Agency’s

FY’s 1998 and 1999 strategic plans for Space Transportation did not provide appropriate

implementing strategies stating how goals and objectives would be achieved and did not include

appropriate implementing strategies with performance indicators and an evaluation process to

measure progress and to address resource requirements. The plans state goals, objectives, and

related activities, but these related activities (that is, the X-33 and X-34) also are not readily

measurable. The FY 1999 NASA Performance Plan contains goals, objectives, and performance

targets. However, the performance targets were not in quantified, measurable metrics, and the

specific contribution of the X-34 and other Space Transportation programs/ projects to goals and

objectives was not clear. Deficiencies in the Agency strategic plans adversely affected the

strategic planning process at all levels. Additional details on specific weaknesses in the Space

Transportation (access to space) portion of the Agency’s FY’s 1998 and 1999 Strategic Plans and

FY 1999 Performance Plan are provided in Appendix F.19

Enterprise Level. The Office of Aero-Space Technology did not issue an Enterprise Strategic

Plan addressing the Space Transportation mission pursuant to the NASA Strategic Management

Handbook. The Enterprise was assigned responsibility for the Space Transportation mission in

February 1997, but had not issued an Enterprise strategic plan on this high-priority mission as of

November 1999, more than 2 years later. The Enterprise last issued a Strategic Plan in April

1995. The Office of Space Access and Technology, which was previously responsible for

 

17 There was no 1997 plan.

18 The FY 1999 Plan is the FY 1998 Plan with 1999 Interim Adjustments.

19 The NASA Strategic Management Handbook, dated October 1996, implements GPRA; we used the handbook as

audit criteria to evaluate the NASA FY 1998 and FY 1999 Strategic Plans.
6

NASA’s Space Transportation mission from 1994 to 1997, also did not issue an Enterprise

strategic plan on Space Transportation. As a result, there has been no Enterprise strategic plan

on the critical Space Transportation mission in at least 5 years.

Enterprise officials acknowledged they had not issued a specific Enterprise Strategic Plan since

1995, but stated three documents issued in 1997 and 1998, cumulatively, would satisfy

requirements for a Plan.20 Our review of the three strategic planning documents issued by the

Aero-Space Technology Enterprise showed that together, they would not adequately serve as an

Enterprise Strategic Plan. The NASA handbook states that Enterprise Strategic Plans must

elaborate on their respective missions and goals, as presented in NASA’s Strategic Plan, by

providing more detailed objectives, implementing strategies, and a description of the key

elements, programs, and/or processes. Documents the Enterprise issued did not adequately meet

these criteria.

For example, Enterprise documents should have provided specifics and details for achieving the

Agency objective to “Complete research and development (during the period 1998 thru 2002) to

enable U.S. industry to launch to Low Earth Orbit at $1,000 /lb.” as stated in the NASA 1998

Strategic Plan. Enterprise plans also should have established an implementing strategy for

achieving this objective to include appropriate metrics (performance indicators) and should have

defined an evaluation process for monitoring and reporting research and development progress

on needed technologies over the 5-year period. Enterprise documents did neither. Details not

provided by the Enterprise include specifics on the required technologies, including current status

and the technology “gap” of each technology being pursued by research and development; and

the metric (such as TRL's) for monitoring technology progress as well as cost/benefit data -- the

benefit to be provided by each of the Enterprise’s initiatives. Enterprise documents did not make

clear the contribution (benefit) of the X-34 toward this objective.

Center Level. Consistent with the Agency’s decentralization of program/project management,

the Associate Administrator for Aeronautics (now Aero-Space Technology) and the Associate

Administrator for Space Flight jointly designated Marshall the Lead Center for Space

Transportation in February 1997. The Associate Administrators directed that Marshall “will

develop a long-range technology plan that fully utilizes the capabilities of other NASA Centers,

other Government agencies, academia, and industry. The plan will be developed consistent with

Headquarters program and investment strategies and guidance.” However, as of November

1999, Marshall, as the Lead Center, had not prepared the required implementation plan.

Although Enterprise officials told us we would find the details and specifics missing from the

Agency and Enterprise strategic planning documents at the Center level, existing Center-level

documents prepared by Marshall including the Center Implementation Plans and Annual Reports

for FY’s 1996 through FY 1999 also lacked such details. Necessary detail implementing the

NASA and Enterprise Strategic Plans, such as approximate qualitative and quantitative

technology requirements, technology “gaps” that the program/project would address, or

appropriate implementing strategies including performance indicators, metrics, and an evaluation

 

20 The three Enterprise strategic planning documents reviewed were “Three Pillars and 10 Goals,” “Roadmaps to the

Future,” and “Progress Report for 1997-1998.”
7

process for measuring and reporting technology progress were missing from the FutureX/Pathfinder Program Plan (draft) and the X-34 Project Plan (draft). (See Finding C for details.)

Cost/Benefit Analyses

The Aero-Space Technology Enterprise strategic planning process for Space Transportation does

not include appropriate use of cost/benefit analyses to support management decisions, help

prioritize technology requirements, and ensure resources are utilized in the most cost-effective

manner. The NASA handbook assigns the Enterprise Associate Administrator responsibility to

develop an Enterprise Strategy, to include a Long-Term Investment Strategy. Similarly, NPG

7120.5A, “NASA Program and Project Management Processes and Requirements,” dated April

3, 1998, tasks Enterprise Associate Administrators to formulate programs and allocate resources.

The NPG requires that a system analysis and life-cycle costing analysis be performed during

program formulation to produce feasible concepts and explore a wide range of implementation

options. The value of a Space Transportation technology roadmap, depicting how NASA will

attain Space Transportation goals, depends heavily on the extent to which it effectively links the

cost of a given initiative or alternative to the technology benefit expected to be received.

However, the Enterprise’s strategic planning process for Space Transportation does not include

an effective process for routinely identifying, measuring, and documenting the technology benefit

to be provided by each initiative. The Enterprise should use cost/benefit analyses to prioritize

technology requirements and ensure cost-effective use of resources in attaining technology

benefits, to produce feasible concepts and explore options, and to develop an appropriate

investment strategy for Space Transportation.

Enterprise officials acknowledged they have not routinely used cost/benefit analyses to document

support for Space Transportation technology management decisions. Instead, they used a

“facilitated expert judgment”21 process to support management decisions. However, Enterprise

officials did not provide documentation on that process or on specific process results for Space

Transportation, stating the process was largely undocumented. While actions such as the

“facilitated expert judgment” process may contribute to management of Space Transportation

technologies, the basis for management decisions should be properly documented. Further, the

evaluation process inherent in a cost/benefit analysis can contribute significantly to the validity of

decisions for prioritizing technology requirements. However, the existing, undocumented

process does not adequately implement the requirements of NPG 7120.5A. As a result, Strategic

planning documents generally do not identify the specific technology contribution, in quantitative

or qualitative terms, to be provided by elements of the RLV Program such as the X-34

Technology Demonstrator. Enterprise plans for Space Transportation would be improved by

using an appropriate, documented cost/benefit analysis to prioritize technology needs and to help

ensure that scarce Agency resources are utilized in the most effective manner. The lack of an

effective, documented cost/benefit analysis process further undermines validity of the Agency

strategic planning process for Space Transportation.

 

21 The process was defined by Office of Aero-Space Technology officials as a roundtable dialogue at meetings

attended by NASA, industry, and other interested parties.
8

The Enterprise, with Marshall assistance, drafted a “Space Transportation Investment Strategy”

document, dated September 1997, in response to OMB direction that NASA submit a launch

plan. This document acknowledged the lack of an appropriate investment strategy for Space

Transportation, stating “Up to now, the agency has not had a systematic approach to defining,

prioritizing and implementing technology development that addresses both (Shuttle and new

RLV) needs as well as other future technology requirements.” However, this strategy document

generally addressed Space Transportation alternatives only at a high level (that is, Shuttle

upgrades as opposed to new RLV technology or other options) and unfortunately did not result in

the Enterprise establishing an appropriate cost/benefit analysis process at the technology level. In

September 1999, Marshall officials stated NASA was considering a proposed process for

determining (1) the “goodness” of a technology product produced within each Enterprise and (2)

the extent of progress toward strategic goals. The proposed process, which includes the use of

Technology Readiness Levels, may eventually be utilized by the Enterprise and Marshall to

provide improved management of space transportation resources.

Contributing Factors to Strategic Planning Problems

Waiting on Industry. NASA’s effort to purchase commercially available space goods and

services from the private sector, in compliance with the National Space Transportation Policy,

has contributed to the lack of identified technology requirements. Specifically, because NASA

anticipates being a customer and buying space launch services from the private sector, NASA

officials deferred to the private sector to define specifications and requirements, including

technology requirements, for the next-generation RLV. As a result, NASA’s implementation of

the National Space Transportation Policy focused on private industry requirements and did not

properly recognize Agency requirements for access to space.22

We believe NASA’s implementation of the Space Transportation Policy23 guidance in 1994

placed too much dependence on industry to identify requirements and insufficient recognition of

the Agency’s needs to carry out assigned missions in science and human exploration of space.

The Policy tasked NASA to “... be the lead agency for technology development and

demonstration for next generation reusable space transportation systems....” As the Lead

Government Agency responsible for civil Space Transportation, NASA should, with industry

input, take the lead in defining technology requirements for the next-generation RLV,

recognizing both industry and NASA requirements.

Consensus. NASA officials told us a major stumbling block to Agency preparation of an

acceptable strategic plan for Space Transportation in recent years has been a lack of consensus

within the Agency and between NASA and private industry on the future of Space Transportation

and the specific solution to high launch costs. The officials stated some elements of the Agency

were not receptive to anything less than a Shuttle-derived, next-generation RLV. Marshall

officials told us that the Space Transportation Architecture Study became bogged down in the

 

22 Our prior audit of the X-33 Advanced Technology Demonstrator (OIG Audit Report IG-99-001, "X-33 Funding

Issues," dated November 3, 1998) determined NASA’s failure to properly recognize Agency requirements as bona

fide needs was a contributing factor to significant financial management deficiencies on the X-33 program.

23 President Clinton issued the National Space Transportation Policy on August 5, 1994.
9

second quarter of calendar year 1999, during our review, due to a lack of consensus within

NASA regarding the future plans for Space Transportation. One difficulty was reaching

agreement on the extent to which Agency launch requirements, as opposed to industry

requirements should be recognized. As a result, OMB officials, who had been asking NASA for

an appropriate Space Transportation strategic plan (launch plan) for more than 2 years, advised

NASA that unless the Agency submitted an acceptable plan for Space Transportation, the $1.2

billion funding “wedge”24 for FY’s 2001 through 2004 would not be included in the Agency’s

Space Transportation budget. The possible loss of more than $1 billion in Space Transportation

funding, if NASA did not provide OMB an acceptable strategic plan, precipitated additional

effort by NASA and Marshall to initiate improvements in Space Transportation strategic plans.

The results of the ongoing Space Transportation Architecture Study (and the current study phase,

the Integrated Space Transportation Plan) may provide more definitive technology requirements

and “roadmaps” for NASA’s Space Transportation requirements. (Additional information on the

ongoing Space Transportation Architecture Study is in Appendix G).

Recommendations, Management's Response, and Evaluation of Response

The Associate Administrator for Aero-Space Technology, in coordination with the

Marshall Center Director, should:

1. Improve Agency strategic planning documents to ensure they comply with

requirements of the NASA Strategic Management Handbook and GPRA. At a

minimum, specify goals that support mission statements; results oriented,

measurable objectives; and implementing strategies that articulate how the goals

and objectives will be achieved, including metrics to be used as performance

indicators and a process for evaluating results.

Management’s Response. Concur. Agency strategic planning for space transportation has been

significantly enhanced with the development of the Integrated Space Transportation Plan (ISTP).

The specific goals, roadmaps, and implementing strategies that were defined through the ISTP

development process will be reflected, as appropriate, in future versions of agency

documentation. Specific performance metrics that contribute to the objective and provide a

meaningful measure of progress toward meeting the safety, reliability, and affordability

objectives will be identified and included in the annual NASA Performance Plan. These changes

will be reflected in the next Agency plan, scheduled for release in the fall of 2000. The complete

text of the comments is in Appendix I.

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. The recommendation is resolved, but will remain

undispositioned and open until agreed-to corrective actions are completed.

2. Prepare an Enterprise strategic plan, in compliance with the NASA Strategic

Management Handbook and GPRA, and establish procedures to review the plan

 

24 Separate budget line item for Future Space Launch Development.
10

annually, and update it as required. The plan should identify technology

requirements, as well as an implementing strategy, to include metrics and evaluation

processes to measure technology progress.

Management’s Response. Concur. The Enterprise is developing a detailed strategic plan that

will include the space transportation goal, objectives, performance metrics, and implementing

strategies. The Enterprise plan will be released in conjunction with the Agency plan in the fall of

2000. The plan will be reviewed on an annual basis and updated as required (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. The recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.

3. Implement an appropriate cost/benefit analysis process to support and document

the rationale for management decisions on Space Transportation, and develop an

investment strategy to implement Space Transportation strategic plans and to help

ensure effective use of resources.

Management’s Response. Concur. The ISTP development process implemented a rigorous and

thorough method of defining and understanding requirements and of evaluating and

recommending investment options. Through this process, NASA consolidated technology

requirements from industry, as well as the Agency, and prioritized them based on the importance

of improving safety, followed by cost and applicability to multiple industry concepts. The ISTP

process implemented an analytical hierarchy methodology of evaluating technology investment

options that was based on architecture-level economic metrics. Each technology was subjected

to a cost/benefit analysis. Each technology was also ranked based on potential payoff to cost,

safety, and technical risk. The results of the integration and analysis process served as the

foundation of a series of roadmaps that illustrate the development needed to advance the TRL's

of key technologies (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. Subsequent to receipt of management's comments, we

requested and management agreed to provide us an ISTP update status briefing and copies of

additional pertinent documentation. The recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.

4. Define technology requirements for the next-generation RLV using metrics that will

facilitate measuring and reporting incremental progress. Consideration should be

given to using Technology Readiness Levels as one Space Transportation metric.

Management’s Response. Concur. As stated in the response to Recommendation 3, NASA has,

through the ISTP development process, performed a comprehensive and detailed analysis of RLV

concept requirements and has established a bottoms-up development approach as demonstrated

by its integrated roadmap structure. Each roadmap illustrates a path to develop the necessary

technology from its existing TRL to at least a flight demonstration TRL (see Appendix I).
11

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. As stated earlier, management agreed to provide us an ISTP

update status briefing and copies of additional pertinent documentation. The recommendation is

resolved but will remain undispositioned and open until agreed-to corrective actions are

completed.

5. The Marshall Center Director should issue a Lead Center Implementation Plan on

Space Transportation. The plan should identify requirements, objectives, and

implementing strategies that are specific and measurable and should identify

metrics that include performance targets and the evaluation process for measuring

annual performance.

Management’s Response. Concur. The Marshall Space Transportation Directorate has created

an integrated strategic plan to implement its charter and accomplish its programmatic

assignments. The “Space Transportation Directorate 2010: Corporate Strategy” Volume I, March

2000, is a 10-year, integrated plan designed to focus the Directorate on the critical activities as

Marshall and its partners proceed to accomplish NASA’s goals of enabling future safe, reliable,

and affordable access to space. The plan is intended to provide guidance and direction for the

decisions and actions of the Space Transportation Team. To ensure that the Space Transportation

Directorate successfully accomplishes this strategy, specific action plans will be developed for

each of the core strategies. Each action plan will include a statement of measurable and

qualitative outcomes, significant partners, key decisions and actions, responsible personnel, and a

timetable for completion. The action plans will be focused on the next 3 to 5 years and will be

updated at least annually. Upon completion in June 2000, these plans will be included as

Volume II of the Corporate Strategy (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. The recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.
12

Finding B. X-34 Project Flight Test Requirements

Marshall had neither established mission-specific requirements for each X-34 flight test nor

properly documented, with a cost /benefit analysis, a requirement for the 27 flights currently on

contract. Additionally, Marshall has not properly documented the rationale for substantial

changes, including the proposed addition of 7 to 10 more flights, to the X-34 flight test program.

This condition exists because Marshall and the Office of Aero-Space Technology established the

number of X-34 flights without performing a cost/benefit analysis and have not revalidated flight

test requirements. Consequently, 25 flights do not adequately support Project objectives and

NASA has no assurance that the appropriate number of test flights are planned or that test

program content (number of X-34 vehicles, sites, powered vs. unpowered flights, etc.) is the most

cost-effective to meet Project objectives. Each flight test costs NASA about $1 million.

Agency Guidance

Neither NASA nor Marshall has issued specific guidance on the conduct of X-vehicle flight test

programs such as the X-34. However, the NASA Systems Engineering Handbook, SP-6105,

dated June 1995, provides “... a generic description of systems engineering as it should be

applied throughout NASA” 25 [emphasis added] and includes guidance on the methods and

techniques, including testing, that can be used to verify that project requirements have been meet.

This engineering handbook defines verification as “... the process of confirming that deliverable

ground and flight hardware and software are in compliance with functional, performance, and

design requirements. The verification process, which includes planning, requirements definition,

and compliance activities, begins early and continues throughout the project life cycle.”

NPG 7120.5A, “NASA Program and Project Management Processes and Requirements,” dated

April 3, 1998, establishes definitive requirements for managing Agency programs and projects

such as the X-34. The NPG contains requirements that program/project managers properly

document management decisions. For example, the NPG states “A system for documentation

and tracking of risk decisions shall be implemented.” The NPG also establishes certain

requirements for managing risk.

Flight Tests Requirements

The X-34 contract requires a Basic Flight Test Program of two flights. However, on January 22,

1999, Marshall exercised contract Option 1, Optional Flight Test Program for 25 additional

flights at a cost of $10,095,798. This increased the total number of flights under contract to 27.

In accordance with NASA Research Announcement 8-14 and the X-34 contract, the parameters

for X-34 flight tests include altitudes up to 250,000 feet and speeds up to Mach 8. In addition to

testing RLV technologies embedded in the X-34 vehicle and a variety of experiments, the project

 

25 The Handbook states, “The objective of systems engineering is to see to it that the system is designed, built, and

operated so that it accomplishes its purpose in the most cost-effective way possible, considering performance, cost,

schedule, and risk.” [emphasis added]
13

is to demonstrate “aircraft like” operability,26 that is, quick turnaround averaging 2 weeks

between flights, with a “surge” capability of 2 flights in 24 hours as defined in the draft X-34

Project Plan. As of October 1999, Marshall had separated the 27 flight X-34 flight test program

into four segments: unpowered flight segment (5 flights), first powered envelope expansion27

segment (7 flights), operability segment (7 flights including one contingency flight), and a final

maximum envelope expansion segment, with experiments, (8 flights).

Mission-Specific Requirements. The X-34 contract statement of work requires that the

contractor28 develop requirements compliance verification methodologies, which permit the

linking of NASA mission requirements to each test flight and to the verification process.

However, the contractor has not yet completed verification methodologies or flight software on

the Optional Flight Test Program because NASA has not yet established the mission- specific

requirements for each flight test.

Because Marshall had not determined the mission-specific requirements of the additional 25

flights, they were all identical 2.5 mach flights without additional speed, altitude, or other

performance mission requirements. The 25 “baseline” flights did not meet Project objectives,

which include testing the X-34 at speeds up to mach 8 and altitudes up to 250,000 feet. On

November 14, 1996, Marshall negotiated modification No. 2 to the X-34 contract which added a

pool of 100,000 labor hours to the contract in anticipation of the additional contractor effort that

would be required to develop necessary flight control software and to perform additional tasks to

support mission requirements, once Marshall identified them. As of November 1999, Marshall

still had not established the mission-specific requirements for each flight.

Contingency Flights. During our audit, Marshall officials acknowledged that the 27 test flights

include 1 or 2 “contingency”29 flights. A draft X-34 flight test schedule identified at least one

contingency flight to support the 24-hour turn around (two flights in 24 hours) portion of the

existing “operability” flight test segment, in the event the first attempt is not successful. The

need for this contingency flight does not appear warranted because the six-flight “operability”

segment of the flight test program offers other opportunities to attempt a 24-hour turnaround.

Until Marshall reviews and revalidates all X-34 flight test requirements, the Center should not

include such contingency flights in the X-34 flight test program.

X-34 Testbed for Space Transportation Experiments. The X-34 Project objectives include

use of the X-34 vehicle as a testbed to flight test Space Transportation technologies. In addition

to soliciting proposals for the X-34 test vehicle, NASA Research Announcement 8-14 also

solicited proposals for experiments to fly on the X-34 vehicle. Marshall has approved at least 10

experiments that it will fund30 and has identified a requirement for 15 flights to carry out these

 

26 "Aircraft like" operability is defined in the X-34 Project Plan as a quick turnaround averaging 2 weeks between

flights, with a “surge “ capability of 2 flights in 24 hours.

27 Envelope expansion refers to gradual increases in vehicle performance (speed and altitude) in each subsequent

flight.

28 The contractor is Orbital Sciences Corporation.

29 A contingency flight is an additional, back-up flight in the event the first attempt is not successful.

30 Marshall had an initial $2 million budget for the experiments, but more funding will be needed to support the

composite tank experiment.
14

experiments. Multiple experiments can be satisfied concurrently by a single flight. However, the

current flight test plan of 27 flights will satisfy only about 3 or 4 of the 15 required experiment

flights because the speed and altitude on many of the 27 flights is not sufficient to meet

experiment requirements. Marshall hopes the initial X-34 flight tests will generate interest

within the U. S. launch community to use the X-34 to fly additional experiments.

Fastrac Engine Requirements. Requirements for the Marshall-developed Fastrac31 engine used

on the X-34 vehicles are dependent on the number of X-34 flights required to meet X-34 Project

objectives. Marshall has contracted for 4 engines to satisfy the planned32 22 powered flights,33 at

a total of more than $10 million.34 Because Marshall has not yet revalidated the number of

powered flights needed, and further engine testing is planned, Marshall should reassess the

number of needed engines.

Changes in Proposed X-34 Project Flight Test Program. Since the contract was awarded in

FY 1996, the proposed X-34 Project flight test program has undergone continuous change as a

result of both internal and external factors. Some of the changes Marshall made are illustrated

below:

• Twenty-five flights were added in the optional flight test program.

• The number of unpowered flights increased from 1 to 5 from August 1996 through

August 1999. Although the X-34 Flight Test Planning Group determined in 1998 that a

total of three unpowered flights were required, Marshall now plans to have five

unpowered flights. The increase in unpowered flights is primarily related to a U.S. Air

Force decision that the X-34 powered flights could not be made at White Sands Missile

Range. As the number of unpowered flights increased, the number of powered flights

decreased from 27 to 22.

• Twenty X-34 tow test runs35 using the A1A vehicle were added. The tow tests are

considered risk mitigation and are intended to demonstrate vehicle landing gear, steering,

etc. are acceptable prior to actual flight.

• Marshall has decreased the number of flights needed to demonstrate the “operability” of

the X-34, from 25 flights in 12 months, to only 6 flights in 3 months.

 

31 Marshall is developing the Fastrac engine in-house, with contractor support, using currently available technology.

Initiated in 1996, separately from the X-34 Project, Fastrac is to reduce launch costs by providing a low cost booster

for small payloads. Fastrac will be the main propulsion for the X-34 vehicle.

32 Per current plans, five of the 27 flights are to be unpowered.

33 The number of required engines depends on the number of flights and the expected life of the engine. Marshall

estimated that each engine may be used for about seven flights.

34Marshall officials told us the 4 flight engines cost about $1 million each and that the 18 engine refurbishments

totaled about $6 million. The engines generally must be refurbished after each use. 35 The first X-34 vehicle assembled, A1, will be used for the tow tests. A1 will be towed across the White Sands

Missile Range to verify functioning of components including the landing gear and steering.
15

• There originally was to be a test article (A1) and one flight vehicle (A2). This

requirement changed, incrementally, to three flight vehicles (A1A, A2, and A3). A1A

will be used only for tow tests and the five unpowered flights.

Marshall’s X-34 Project Office did not properly document the rationale for such changes.

Specifically, Marshall did not document the available alternatives or options to meet X-34

requirements, including their benefits and disadvantages or the rationale for the options Marshall

selected. Substantial changes such as these should also be supported by cost/benefit analyses. In

the absence of adequate support for the proposed X-34 flight test program, NASA has no

assurance that all 27 flights are needed. Each flight will cost NASA about $500,00036 under the

fixed-price contract.

Expanded Flight Test Program. Marshall has proposed an expanded flight test program, which

will add up to 10 flights to the planned 27 X-34 flight tests. Marshall identified FY 2001-2003

funding requirements for 7 to 10 additional flights at about $1 million each. Marshall officials

told us these flights are needed to support the flight requirements of the 10 experiments identified

thus far using the X-34 vehicle as a flight testbed. Marshall has scheduled eight flights for the

A3 vehicle37 on which the experiments will fly. However, Marshall officials stated that it is

likely only three to five of the eight A3 flights will attain the speed and/or altitude needed to

support the experiments. For this reason, Marshall has identified the need for additional flights

under the expanded flight test program. Marshall should cancel the proposed expanded flight

test program, until a reassessment of all X-34 flight test requirements has been completed.

Canceling the proposed 7 to 10 additional flights would save up to $10 million. The extent of

any savings is contingent on the results of Marshall’s reassessment of X-34 flight test

requirements.

Use of Other Flight Test Vehicles. Marshall does not plan to use the A2 vehicle, which is

expected to fly 14 flights, to support the experiments. The limited speed (up to mach 5) planned

for the A2 vehicle may be sufficient to benefit some of the experiments. Therefore, Marshall

should give consideration to also using the A2 vehicle to support the experiments. This

alternative should be addressed as part of the reassessment of overall flight test requirements.

Recommendations, Management's Response, and Evaluation of Response

The Marshall Center Director should:

6. Establish mission-specific requirements for X-34 flight tests, determine the

minimum number of flights required to satisfy X-34 Project objectives, and delete

from the X-34 contract those flights that are not justified.

Management’s Response. Concur. In January 2000, Marshall provided mission-specific flight

test requirements to Orbital Sciences Corporation. As each flight is planned, minor adjustments

 

36 This amount reflects Orbital contractor costs only and does not include in-house Agency costs, range costs, etc.

37 The third X-34 vehicle is designated the A3 and will be the only vehicle on which the experiments will be

installed.
16

may be made for the individual mission flight software. The X-34 Project Plan identifies the

requirement for a project controlled Flight Test Plan that documents the requirement for 27 test

flights. The X-34 Project Manager will ensure that, as part of the ongoing replanning activity,

flight requirements will be appropriately revalidated (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. The recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.

7. Implement internal controls to appropriately document management decisions,

including changes to the proposed flight test program.

Management’s Response. Concur. A project configuration control process has been put in

place to manage changes to documentation such as the flight test program and to better document

management decisions (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. However, X-34 Project Office officials did not provide us

data on their configuration control process and, after receipt of the comments, stated the process

is not yet totally in place. Therefore, the recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.

8. Cancel the proposed expansion of the flight test program until justification for the

existing 27 flight test program has been reassessed and the total number of flights

needed to meet Project objectives have been determined and revalidated. The

reassessment should consider using the A2 vehicle to support experiments.

Management’s Response. Concur. Currently, there are no budget or contractual actions planned

for an expanded flight program beyond the existing 27-flight test program (see Appendix I).

Evaluation of Management’s Response. The actions taken by management are sufficient to

disposition and close the recommendation for reporting purposes.

9. Reassess the number of Fastrac engines required to support the X-34 Project based

on results of the reassessment of test flight requirements for the X-34 and on engine

reliability tests, and eliminate any unneeded engines from the contract.

Management’s Response. Concur. The X-34 Project Office will reassess the number of Fastrac

engines required to support the Project and, based on this assessment, will eliminate any

unneeded engines from the contract (see Appendix I).

Evaluation of Management’s Response. The actions planned by management are responsive to

the recommendation. The recommendation is resolved but will remain undispositioned and open

until agreed-to corrective actions are completed.
17
18

Finding C. Documentation for Space Transportation Programs/Projects

NASA has not finalized and approved the X-34 Project Plan or the program commitment

agreement and program plan for the Future-X/Pathfinder Program, which includes the X-34

Project. This condition exists because Marshall and the Office of Aero-Space Technology have

not complied with Agency directives, have not placed sufficient emphasis on timely preparation

of program documentation, and have not established appropriate internal controls to ensure such

documentation is prepared prior to granting approval for program/project implementation. As a

result, delays occurred (1) in obtaining senior Agency approval for the program management

concepts and the internal control functions to be used in managing these high-priority, costly

programs, and (2) in defining the roles and responsibilities of NASA officials for managing

them. Delays in preparing program documentation can also adversely affect the Agency’s

management of technology needed to reduce the cost of access to space.

Agency Guidance on Program Documentation

Agency directives on program/project management require that both a program commitment

agreement and a program/project plan be prepared early, in the formulation phase, prior to actual

implementation of the program or project. NASA Policy Directive (NPD) 7120.4A,

"Program/Project Management," dated November 14, 1996, requires that the plans be in place

prior to the start of project implementation. NPG 7120.5A, "NASA Program and Project

Management Processes and Requirements," dated April 3, 1998, defines the requirements that

managers must meet in formulating, approving, implementing, and evaluating programs and

projects. (Additional information on program documentation requirements is in Appendix H).

Compliance with Agency Directives. Marshall and the Office of Aero-Space Technology have

not finalized and approved X-34 program documentation. In February 1998, more than 3 years

after the original X-34 Program was initiated, Marshall prepared a Draft X-34 Program Plan.

However, neither the Marshall Center Director nor NASA Headquarters approved the plan. In

February 1999, when NASA Headquarters redesignated the X-34 a “Project” within the

Future-X/Pathfinder Program, the draft plan was to be rewritten as a Project Plan; however as of

November 1999, Marshall and the Office of Aero-Space Technology have not completed and

approved it.

As of November 1999, Marshall and the Office of Aero-Space Technology also did not have an

approved Program Commitment Agreement or Program Plan for the Future-X/Pathfinder

Program. Although both documents have been drafted, they have not yet been approved by

NASA Headquarters. Marshall initiated funding/budgets for the Future-X/Pathfinder Program in

1997, requesting that initial funding be provided in FY 1999. NASA Headquarters approved the

funding requests, constituting program implementation, and funds were included in the Agency

budget for FY 1999. We attribute the noncompliance with Agency directives primarily to a lack

of emphasis by Marshall and the Office of Aero-Space Technology on program documentation

requirements.
19

Compliance with Procurement Notices. On February 22, 1999, NASA’s Associate Deputy

Administrator issued a letter, subject “Integration of Acquisition Planning with NPG 7120.5A,

NASA Program and Project Management Processes and Requirements,” to the Associate

Administrator for Procurement. The letter emphasized the importance of complying with

applicable Agency directives and directed the Associate Administrator for Procurement not to

process solicitations initiated by Agency programs/projects if program documentation was

incomplete. To ensure compliance with the Associate Deputy Administrator’s guidance, the

Associate Administrator for Procurement issued Procurement Information Circular 99-6, “ NPG

7120.5 Approvals and Requirements for the Release of Solicitations,” dated March 26, 1999; and

Procurement Notice 97-28, “NASA Internal Programmatic Approval Documentation,” dated

March 26, 1999. The Notice and Circular both prohibit Center Procurement Officers from

issuing solicitations on Agency programs and projects for which program documentation is

incomplete. The Circular and Notice require a new certification to accompany all purchase

requests, stating that all program documentation has been completed and approved.

Marshall issued a Request for Offer for the Fastrac engines on April 20,1999. Marshall

subsequently awarded contract NAS8-99103 valued at $11.3 million on July 30, 1999, for the

Fastrac engines to support the X-34 flight test program. When the Request for Offer was issued

and the contract was awarded, program documentation for both the X-34 Project and the FutureX/Pathfinder Program had not been completed and approved by NASA Headquarters. The

Marshall Contracting Officer provided us a copy of a letter from the X-34 Project Office stating,

“This confirms prior verbal certification that all required documentation under NPG 7120.5 for

X-34 is current and approved.” The X-34 Project Office officials stated they certified that all

X-34 documentation was complete because they did not think the requirement in the Circular and

Notice applied to the X-34 since the project had been previously approved for implementation

(upon funding approval) by NASA Headquarters.

We do not agree that the X-34 Project is exempt from requirements in Procurement Notice

97-28. The Procurement Notice clearly states the “Purpose” is “To ensure that no affected

solicitation is released prior to the approval of key programmatic documentation required by

NPG 7120.5, NASA Program and Project Management Processes and Requirement.”

Program Documentation Content. Agency Space Transportation programs and projects

respond to the goals, objectives, and requirements specified in NASA and Enterprise strategic

plans. The NASA Strategic Management Handbook requires increasing detail at lower levels of

program documentation. Program documentation should provide full details for implementing

strategic plans. Aero-Space Technology Enterprise officials stated that the specific, quantified

technology requirements and metrics missing from Agency and Enterprise strategic plans (see

Finding A) would be provided in Center-level, program and project documents. However,

Marshall’s draft plans for Future-X/Pathfinder and X-34 do not provide adequate levels of detail

on the Space Transportation technologies that are to be addressed. Specifically, FutureX/Pathfinder and X-34 are technology development/demonstration initiatives, however:

• The draft program/project plans did not comply with NPG 7120.5A and address the

technology readiness levels of either those technologies embedded in the X-34 vehicle or
20

the experiments to be flown as part of the X-34 flight test program. NPG 7120.5A

requires a Technology Requirements Synthesis at both program and project levels which

identifies and establishes the status of all technologies required. A Technology

Requirements Synthesis is performed to “...examine project concepts and assess

technology requirements for feasibility, availability, technology readiness, opportunities

for leveraging research, and new technologies.”

• The draft X-34 plan did not adequately address flight test objectives, specify the

procedures to be used to analyze test results, or effectively identify processes to

validate/verify embedded RLV technologies (see Finding B for details on flight tests).

• The draft X-34 plan also did not indicate specifically how X-34 technology results would

be used by NASA (see Finding A for details).

• The draft program/project plans did not adequately identify the metrics or processes for

evaluating technology results.

• The draft program/project plans did not establish a requirement to input results of the

X-34 Project or other Future-X/Pathfinder projects into the Agency technology database.

The lack of appropriate strategic plans at the Agency and Enterprise levels (see Finding A) may

be a contributing factor to the untimely and deficient program and project documentation for

Space Transportation initiatives identified by this and prior NASA Office of Inspector General

audits of Space Transportation. Complete and accurate strategic planning documents at Agency

and Enterprise levels would facilitate the preparation and maintenance of current program

documentation.

Recommendations, Management's Response and Evaluation of Response

The Associate Administrator, Office of Aero-Space Technology, and the Marshall Center

Director should:

10. Finalize all required program documentation for the X-34 and Future–X/Pathfinder

as required by NPD 7120.4A, “Program/Project Management,” and NPG 7120.5A,

“NASA Program and Project Management Processes and Requirements.”

Management’s Response. Concur. The Marshall Future-X/ Pathfinder Program Office is

working with the Office of Aero-Space Technology to complete the Future-X/Pathfinder

Program Commitment Agreement and Program Plan and the X-34 Project Plan at the earliest

possible date (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. The recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.
21

11. Revise the existing draft program commitment agreement and draft

program/project plans for the Future-X/Pathfinder and X-34 Project to better

define approximate technology requirements and expected results (benefits), and

establish performance metrics and an evaluation process to evaluate results.

Management’s Response. Concur. The Future-X Pathfinder Program Commitment Agreement

and Program Plan and X-34 Project Plan are being updated, and management will ensure that

appropriate technology requirements, expected results, and performance metrics are included.

The Pathfinder Program Office and the Office of Aero-Space Technology will work together to

complete these documents at the earliest possible date (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. The recommendation is resolved but will remain

undispositioned and open until agreed-to corrective actions are completed.

The Associate Administrator, Office of Aero-Space Technology, should:

12. Establish internal control procedures within the Enterprise’s Programs Division to

ensure that responsible Centers submit required program/project documentation

during the program formulation process.

Management’s Response. Concur. The Office of Aero-Space Technology Headquarters Office

Work Instruction for "Program Project Formulation and Approval (HOWI: 7100-ROO7B,

September 1999)" established procedures for the preparation of documentation during the

program formulation phase. Management will continue to ensure that responsible Centers

submit required program/project documentation during the program formulation process (see

Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

sufficient to disposition and close the recommendation for reporting purposes.

13. Discontinue the practice of approving programs and projects for which Program

Commitment Agreements and program/project plans are not yet prepared or

approved, and place added emphasis on compliance with program documentation

requirements specified in Agency directives NPD 7120.4A and NPG 7120.5A.

Management’s Response. Concur. Management will ensure that all Enterprise programs and

projects will be in compliance with and have the proper documentation specified in NPD

7120.4A and NPG 7120.5A (see Appendix I)

Evaluation of Management’s Response. Management's original response was only partially

responsive because management did not identify specific corrective actions. However, on

March 27, 2000, the Office of Aero-Space Technology provided additional comments that

identified planned actions as follows:

Since development of the documentation is a shared responsibility, we [Headquarters and

Marshall] are putting special emphasis and working with the Chief Engineer’s Office to ensure that
22

the X-34 program is in compliance with the NPD. Additionally, the Office will lead a

Headquarters-Center team to coordinate the review and update, as appropriate, of ISO

[International Organization for Standardization] 9000 Office Work Instructions to ensure these

procedures cover all requirements of and contain appropriate emphasis on compliance with the

NPD 7120.4A and NPG 7120.5A.

Based on the additional comments, we consider the recommendation resolved but

undispositioned and open until agreed-to corrective actions are completed.

14. Require responsible Centers to clearly identify in program documentation the

approximate technology requirements, expected results (benefits), and performance

metrics for evaluating actual results and to establish internal control procedures to

ensure that the documentation effectively implements Enterprise strategic plans.

Management’s Response. Concur. Management has recognized this deficiency. The

documentation for the two major programs (Aviation Safety and Ultra Efficient Engine

Technology) initiated in FY 2000 identifies the expected requirements, benefits, and metrics (see

Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management that

were included in their original response were only partially responsive because management did

not identify specific corrective actions. As stated above, we received additional comments that

identified planned actions as follows:

The wide variety of programs managed by this Office (flight programs, focused research and basic

research) precludes a set standard for the metrics as they vary from program to program. As noted

in the original response, the Office is improving the documentation and the Office is in the process

of establishing internal controls by modifying appropriate OWI's [Office Work Instructions] to

ensure all future documentation, as submitted by the centers and approved by the Office clearly

defines technology requirements and effectively implements strategic plans.

Based on the additional comments, we consider the recommendation resolved but

undispositioned and open until agreed-to corrective actions are completed.

The Marshall Center Director should:

15. Establish internal controls to ensure solicitations are not issued on

programs/projects for which program documentation is not complete as required by

Procurement Notice 97-28 and Procurement Information Circular 99-6.

Management’s Response. Concur. Internal controls have now been established to ensure

compliance with the policy. The Automated Procurement Request System at Marshall has been

revised to include mandatory fields regarding applicability and compliance with NPD-7120

documentation requirements. Procurement requests cannot be processed without completing the

mandatory fields (see Appendix I).

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. However, we requested additional clarification of corrective
23

actions to determine their effectiveness, which disclosed the need for further actions. On

March 28, 2000, Marshall officials agreed to take additional actions to further strengthen internal

controls and to ensure that the revisions to the Automated Procurement Request System are

effective.

The recommendation is resolved but will remain undispositioned and open until agreed-to

corrective actions are completed.

16. Place added emphasis on compliance with program documentation requirements

specified in Agency Directives NPD 7120.4A and NPG 7120.5A, and ensure that

programs and projects prepare documentation in a timely manner.

Management’s Response. Concur. Marshall has established a Systems Management Office that

has charter responsibility for ensuring that Center programs and projects comply with NPD

7120.4A and NPG 7120.5A. The Systems Management Office conducts independent evaluations

(as required by NPG 7120.5A) of Marshall programs and projects (Microgravity and Space

Transportation) that report to the Marshall Project Management Council and Marshall LeadCenter Program Management Councils. If the Systems Management Office or the Directorate

program/project review process had been in place previously, the X-34 Project’s noncompliances

would have been identified and corrected in a timely manner.

Evaluation of Management’s Response. The actions taken and planned by management are

responsive to the recommendation. Management actions are sufficient to disposition and close

the recommendation for reporting purposes.
24

Appendix A. Objectives, Scope, and Methodology

Objectives

The overall objective was to assess the status of the X-34 Demonstrator Project in meeting

technology requirements for the next generation RLV. Specifically, we determined whether:

• Orbital Sciences Corporation was fulfilling its obligations under the contract,

• key technologies were being identified and tested in accordance with

 milestones, and

• NASA’s process for transferring X-34 technology was effective.

To evaluate NASA’s planned use of X-34 technologies, we reviewed the management of Space

Transportation technology requirements by the Aero-Space Technology Enterprise and Marshall

in view of the significant role Marshall has as Lead Center for Space Transportation. We also

reviewed the interrelationship of the X-34 technology to overall Space Transportation technology

requirements.

Scope and Methodology

We performed audit work at Marshall; NASA Headquarters; and the Orbital Sciences

Corporation, Dulles, Virginia, facility.

• To determine whether Orbital Sciences Corporation was fulfilling its obligations under the

contract, we reviewed contractor compliance with various contract requirements to include

meeting milestone due dates, cost and schedule targets, and deliverable requirements.

• To determine whether key technologies were being identified and tested in accordance with

milestones, we reviewed the contractor’s record of completion of technology milestones

under the X-34 contract.

• To determine whether NASA’s process for transferring X-34 technology was effective, we

reviewed NASA’s operation of the Agency’s technology database to include the extent of

input from the X-34 Project as well as the number of queries by database users. We also

reviewed the adequacy of Agency guidance on technology management.

• To determine how NASA planned to use the technologies developed on the X-34 Project, we

reviewed Future-X/Pathfinder and X-34 Program/Project documentation and NASA,

Marshall, and Aero-Space Technology Enterprise strategic planning documents, and we

evaluated adequacy of the overall process established to acquire the technologies needed to

reduce launch costs.
25

Appendix A

We examined the policies, procedures, and practices NASA and the Aero-Space Technology

Enterprise used in preparing strategic planning documents for Space Transportation. We reviewed the

limited documentation on the rationale and support for the proposed X-34 flight test program. We

also reviewed the current program status for the X-34. We interviewed personnel involved in

preparing program documentation, flight test plans, and strategic planning documents. Our interviews

included Aero-Space Technology officials, Marshall personnel, and personnel at the Orbital Sciences

Corporation headquarters facility at Dulles, Virginia. We did not assess the scientific requirements

underlying the X-34 flight requirements.

Management Controls Reviewed. We reviewed NASA’s policies on program/project

management and strategic planning. Specifically, we reviewed NPD 7120.4A, “Program/Project

Management”; NPG 7120.5A, “NASA Program and Project Management Processes and

Requirements”; and NPG 1000.2, “NASA Strategic Management Handbook.” Management

controls regarding strategic planning for Space Transportation, flight test plans, and program

documentation should be improved, as discussed in the findings.

Computer-Processed Data. We used computer-processed reports from NASA Headquarters,

Marshall, and Orbital Sciences Corporation to assess the status of the X-34 Project and to

evaluate the proposed X-34 flight test program. We reviewed and tested selected data but did not

verify the overall validity of the reports. The lack of verification did not affect our audit results.

Prior Audit Coverage. There has been no prior audit coverage on the X-34. However, we

issued the following reports on the X-33 Advanced Technology Demonstrator, a “sister” program

of the X-34 under the Marshall-managed RLV Program:

On March 29, 1999, we issued audit report IG-99-019, “X-33 Cooperative Agreement,” which

states that NASA’s use of a cooperative agreement contributed to deficiencies in program

management processes including planning and execution, resource management, and property

control. The report made recommendations to improve these processes.

On November 3, 1998, we issued audit report IG-99-001, “X-33 Funding Issues,” which

discusses NASA’s lack of properly recognizing Agency requirements as bona fide needs which

contributed to significant financial management deficiencies on the X-33 program.

This approach not only did not properly recognize Agency requirements, but also may

have made it more difficult for NASA to reach a consensus on RLV technology requirements.

In August 1999, the General Accounting Office (GAO) issued Audit Report Number

NSIAD-99-176: “Space Transportation – Status of the X-33 Reusable Launch Vehicle Program,”

which identifies problems in NASA’s strategic planning for Space Transportation. Specifically,

the report states that NASA’s Fiscal Year 2000 Performance Plan does not include performance

targets that establish a clear “growth path leading from the X-33 flight test vehicle to an

operational SSTO [Single-Stage-To-Orbit] vehicle.” The GAO report points out that the
26

Appendix A

NASA Advisory Council has also voiced similar concerns. GAO recommended “the NASA

Administrator include in the agency’s Fiscal Year 2001 Performance Plan performance targets

for the X-33 Program that establish a clear path leading from the X-33 flight-test vehicle to an

operational RLV and show progress toward meeting the agency’s objective of significantly

reducing launch costs.” NASA management concurred with the recommendation.

Audit Field Work. During February through December 1999, we conducted field work at

Marshall; NASA Headquarters; and Orbital Sciences Corporation Headquarters, Dulles, Virginia.

We performed the audit in accordance with generally accepted government auditing standards.
27

Appendix B. Background

Since the 1970's, NASA has searched for a cheaper launch vehicle, which led to the decision to

build the Space Shuttle. Although the original intent of the Space Shuttle Program was to

significantly reduce NASA’s cost of access to space, the Shuttle has not come close to the flight

rates (as high as 57 flights annually) anticipated early in the program. The Shuttle fleet is

currently able to sustain only about seven or eight flights per year, and the maximum flights in

any year was nine (1985). As a result, the Shuttle has proven expensive to launch (about $300 -

$500 million per launch depending on the number of flights launched annually), and NASA has

spent much of the last decade looking for a way to reduce launch costs. NASA’s original goal of

reducing the launch cost from $10,000 per pound to $1,000 per pound by 2006 may not be

realized until 2010–2011 or later.

NASA’s Access to Space study in 1993 determined that an RLV offered advantages over use of

an expendable launch vehicle. Specifically, while the development and production costs of an

expendable launch vehicle primarily determine launch costs, the cost of launching an RLV is

determined mainly by the number of times the RLV can be used. A higher flight rate for an RLV

allows better amortization of the development costs, resulting in a lower overall launch cost per

flight.

The fact that the Shuttle has not been successful in significantly reducing launch costs has lead to

increased concern by Congress and OMB (see Appendix D) that NASA find a solution. NASA

expected to make a decision before December 31, 2000, on an RLV to replace the Shuttle.

Under the original RLV/Venture Star38 timetable, NASA and industry partners had hoped the X33 Program would permit them to make a decision on Venture Star by the end of 1999. The

decision on a next-generation RLV involves several decisions addressing what the next

generation RLV will look like, when it can be operational, how it will be phased in to replace the

existing Shuttle, and who will own and operate it. According to a NASA in-house study

completed in February 1999, “Not enough knowledge is available today to commit to a Shuttle

replacement.” NASA simply does not have the technology(s) at this time to support the decision.

It is unlikely that NASA can make the decision prior to 2002 or 2003 at the earliest, and it may

be the middle of the next decade before an informed decision can be made.

As the Lead Center for Space Transportation, Marshall has extensive responsibility for

implementing the Agency’s Space Transportation mission managed by NASA’s Office of AeroSpace Technology. Marshall’s responsibility includes participating in strategic planning for the

Space Transportation mission area.

 

38 Venture Star is a next-generation RLV concept proposed by Lockheed Martin Skunk Works.
28

Appendix B

The Center’s Space Transportation Directorate has responsibility for various Space

Transportation elements including the following:

• The Advanced Space Transportation Program Office is responsible for developing, maturing,

and performing ground testing of the technologies needed to meet NASA’s unique mission

requirements and for reducing space transportation costs to enable commercial development

of space.

• The Future-X /Pathfinder Program Office has responsibility for the X-34 and X-37 Projects

and is responsible to demonstrate advanced space transportation technologies through the use

of flight experiments and experimental vehicles.

• The X-33 Program is a one-half scale size advanced technology demonstrator and is an

integral part of the RLV Program, which NASA hoped would lead to a commercially owned

and operated RLV --- the Venture Star. While Venture Star is still an option for the nextgeneration RLV, NASA’s ongoing Integrated Space Transportation Plan does not identify

Venture Star as the likely successor to the Space Shuttle.

The X-34 Project is subject to oversight by several groups, internal and external to NASA.

Marshall’s Program Management Council, a senior-management group chaired by the Deputy

Center Director, has primary oversight of the X-34 Project.39 Additionally, the Space

Transportation Council, the NASA Advisory Council, and the Aero-Space Technology Advisory

Council all provide Space Transportation expertise. Independent reviews such as the NonAdvocate Reviews40 and Independent Annual Reviews41 provide the Program Management

Council with objective evaluations of Project conformance to plans and objectives.

 

39 NPG 7120.5A permits certain projects to have Center-level oversight.

40 The Non-Advocate Review team performs its reviews during program formulation phases as input to the program

approval process; the reviews include the verification of life-cycle cost estimates.

41 The Independent Annual Review is an analysis of the status of commitments (performance, cost, and schedule) as

compared to the program/project baseline and established thresholds.
29

Appendix C. Federal and Agency Requirements

The Government Performance and Results Act (GPRA), passed by Congress in 1993, requires

Federal agencies including NASA to accomplish the following:

• Develop an Agency 5-year Strategic Plan, setting forth NASA’s mission, long-term goals,

objectives, implementing strategies, and associated resource requirements. Initial

submission to the OMB and Congress was September 30, 1997, with updates at least

every 3 years.

• Prepare annual Agency Performance Plans that establish performance goals, measurable

objectives, and associated resource requirements needed to achieve long-term goals. The

initial plan, for FY 1999, was to be submitted to OMB by September 1997. As part of the

Performance Plans, agencies may request that OMB grant managerial waivers on

personnel levels, salaries, and budget constraints for programs indicating improved

performance within program funding.

• Submit annual Agency Performance Reports that will measure goal achievement and/or

identify reasons for failure in goal achievement. The initial report is to be submitted to

the President and Congress by March 31, 2000, for FY 1999.

NPG 1000.2, “NASA Strategic Management Handbook,” documents the Agency’s management

policies, processes, guidelines, and responsibilities for strategic management as required by

GPRA. Specifically, the handbook delineates the processes, schedules, and management

responsibilities for the following:

• Integrating Strategic Planning with the budget process

• Developing detailed Implementation Plans and strategies

• Managing the execution of strategies through programs and processes

• Evaluating performance and assessing and reporting results

The handbook requires that the NASA Strategic Plan and the Enterprise Plans include specific

elements to ensure consistency and meet GPRA requirements. The specific elements should

include broad vision and mission statements, identification of customers and applicable

environments, as well as appropriate goals, objectives, and implementing strategies.

• Goals elaborate on the mission statement and constitute a specific set of policy,

programmatic, or management outcomes for the programs and operations covered

in the strategic plan.
30

Appendix C

• Objectives are specific milestones and target levels of near-term outputs that are

to be achieved during strategic implementation. Objectives should be:

• Specific - reflect the accomplishments to be achieved

• Measurable – either quantifiable or verifiable

• Aggressive, but attainable

• Results-oriented – focused on the desired outputs and outcomes

• Time-bound – achieved within a specific time frame

• Implementing Strategies should include a description of how the goals and

objectives will be achieved. Each strategy should have details addressing:

• Key elements, programs, processes, values, and culture

• Metrics to include performance indicators, and an evaluation process to

determine progress and indicate how evaluations are to be used to establish or

revise goals

• Resource requirements with an investment strategy

The handbook states that annual Performance Plans required by GPRA are to establish

performance goals, measurable objectives, and associated resource requirements needed to

achieve long-term goals. The handbook also requires that Enterprise Strategic Plans, which flow

down from, and must align with, the NASA Strategic Plan be reviewed and updated, as required,

annually. The plans are to include performance goals and metrics for the Enterprise as well as

top-level resource requirements. NASA’s annual GPRA Performance Plan presents the

integration of the performance planning inputs provided by the Agency’s Strategic Enterprises.

Consistent with GPRA requirements, this plan should include the following:

• Quantifiable performance goals

• Level of performance to be achieved during the budget year

• Human, capital, or other resources required to meet performance goals

• Specific nonprogrammatic actions planned within the time frame and fiscal scope of the

proposed budget

• Metrics to be used as performance indicators

The NASA Strategic Management Handbook and the NASA strategic plans assign Marshall the

mission area of Space Transportation Systems.
31

Appendix D. Congressional, OMB, and GAO Concerns

Congress, OMB, and the GAO have expressed concern about NASA’s strategic planning for

Space Transportation. Other NASA external oversight elements have also voiced concerns. In

response to congressional concerns in the early 1990’s over the continued high cost of launching

payloads into space, NASA performed the Access to Space Study in 1993. This study concluded

that a Single Stage to Orbit RLV offered the most potential for significantly reducing the cost of

access to space, but at substantial risk. NASA’s X-33 Program continues to pursue the Single

Stage to Orbit RLV concept, although numerous difficulties have been encountered.

OMB Concerns

OMB expressed concern over the adequacy of NASA’s strategic planning for replacing the

Shuttle and lowering launch costs. In November 1996, OMB stated in its Passback42 guidance:

NASA’s budget assumes about $4B over 5 years for a variety of technologies, upgrades and

development activities related to space launch (e.g., X-33, X-34, ASTP [Advanced Space

Transportation Program], Shuttle upgrades, CRV [Crew Return Vehicle], Hyper-X). Yet it is

difficult to understand the strategy that will enable a decision on a future space transportation

architecture for NASA. OMB requests that the appropriate offices within NASA and NASA’s

Independent Program Assessment Office coordinate to create a strategic plan that outlines: (1) the

Agency’s space launch requirements; (2) current investments in technology, development, and

operations of space launch vehicles; and, (3) the objectives, strategy, budget and key decisions that

will enable a future space transportation architecture for NASA. OMB also requests that the RLV

criteria documents be incorporated in such a plan, the decision criteria for Phase 3 be updated to

reflect the X-33 award, and the criteria be independently reviewed as earlier agreed to by NASA.

NASA should submit the launch plan to OMB no later than September 1997.

OMB was concerned about the adequacy of NASA Strategic Plans for acquiring the technology

needed to develop a next-generation RLV. Agency efforts for strategic planning in response to

this 1996 OMB Passback were not sufficient, and OMB again expressed concern about NASA’s

plans for reducing launch costs in the 1997 Passback guidance:

The President’s 1994 National Space Transportation Policy calls for an end-of-the-decade decision

on the development of an operational launch system to reduce NASA’s launch costs. ... OMB ...

recommends that before decisions are made on the allocation of funds, NASA present options

developed by the private sector to OMB and OSTP [Office of Science and Technology Policy] on

how they could lower NASA’s overall launch costs through either existing, planned, or new

vehicles. Industry inputs could be accomplished through an RFP [Request for Proposal] with

appropriate Terms of reference for the studies.

 

42 OMB’s feedback/comments to NASA on previously submitted budget.
32

Appendix D

As a result of OMB concerns, NASA initiated the Space Transportation Architecture Study in

1998, and initiated phase III of the study, the Integrated Space Transportation Plan, in July 1999

to ensure NASA’s access to space requirements and industry requirements were appropriately

addressed.

GAO Concerns

Based on its recent audit43 of the X-33, a sister program of the X-34 within NASA’s RLV

Program, GAO concluded that NASA Strategic Plans for Space Transportation need

improvement. GAO reported that NASA’s FY 2000 Performance Plan for Space Transportation

does not contain sufficient metrics for measuring performance. GAO reported that NASA’s

Fiscal Year 2000 Performance Plan does not include performance targets that establish a clear

“growth path leading from the X-33 flight test vehicle to an operational SSTO [Single Stage to

Orbit] vehicle.” GAO recommended that “the NASA Administrator include in the agency’s

Fiscal Year 2001 Performance Plan performance targets for the X-33 Program that establish a

clear path leading from the X-33 flight-test vehicle to an operational RLV and show progress

toward meeting the Agency’s objective of significantly reducing launch costs.” NASA

management concurred with the recommendation and will include performance targets in the FY

2001 Performance Plan.

Oversight Organization Concerns

NASA oversight committees that are responsible to provide Space Transportation expertise have

also expressed concerns regarding NASA’s strategic planning for Space Transportation. In an

August 26, 1997, report to the Aero-Space Technology Advisory Committee, the Space

Transportation Subcommittee indicated its concern that NASA’s space technology programs

(X-34, Advanced Space Transportation Program, Hyper-X, Future-X/Pathfinder, etc.) are

somewhat unfocused across a wide range of future technologies, while the immense, near-term

challenge of achieving an operational RLV by 2012 was underfunded. Similarly, the NASA

Advisory Council indicated that the strategic path for the X-33 was not clear and that the goals

and expectations appear to be overstated and unrealistic. One of the members of the NASA

Advisory Council added that “... the strategic plan for Aero-Space Technology is in shambles.”

 

43Report Number NSIAD 99-176, Status of the X-33 Reusable Launch Vehicle Program, dated August 1999.
33

Appendix E. Metrics for Acquiring Needed Technologies

To be effective, strategic planning should involve significant use of metrics. The NASA

Strategic Management Handbook requires that strategic planning documents include objectives

that are “quantifiable” or “measurable,” and states “NASA must be able to monitor its success in

achieving each objective.” The Handbook further requires that strategic planning include

implementing strategies, to include performance indicators, metrics, and a program evaluation

process to determine progress. Space Transportation technology management extensively

involves the development of hardware (engines, composite structures, composite/cryogenic fuel

tanks, thermal protection systems, avionics, etc.) for new boosters/launchers. NASA has been

performing hardware research and development for 4 decades; therefore, the Agency should be

able to establish measurable goals and metrics for Space Transportation. However, as stated in

Finding A, strategic planning documents do not clearly identify specific technology objectives,

requirements, and implementing strategies with appropriate metrics in quantitative and/or

qualitative terms to permit measurement and reporting of incremental progress. Additionally, the

plans do not identify how NASA will attain the critical goal of reducing launch costs. Agency,

Enterprise, and Marshall strategic planning documents do not indicate how the X-34 will

contribute to the needed technology for the next-generation RLV.

The technology requirement for each of the six core technologies44 needed for the nextgeneration RLV should be clearly specified in quantitative and qualitative terms, to the extent

possible, that can be used to measure and report annual progress toward each requirement. For

example, if a better Thermal Protection System is needed for the next generation RLV, criteria

for defining the technology requirement might include:

• Temperature endurance capability

• Penetration (hardness) resistance to space debris

• Durability, maintainability (replacement) and operability

Specific requirement criteria should be established for all six RLV core technologies.

Achieving reduced cost of access to space is dependent on NASA’s successful maturing of

several core technologies that could be used on the next-generation RLV. In this endeavor,

NASA must identify the required technology in meaningful, measurable terms that include

metrics for reporting incremental progress over time. A comparison of current status with the

requirement identifies the existing “technology gap.” However, NASA has not yet identified an

acceptable method/metric for stating the technology requirement or for measuring progress.

 

44 Technologies to be demonstrated are reusable cryogenic tankage, composite structures, durable thermal protection

systems, advanced avionics, reliable propulsion systems, and aircraft-like operations.
34

Appendix E

Space Transportation strategic planning documents at all levels use the term “technology”

extensively, but do not use the term Technology Readiness Level to identify Space

Transportation technology requirements and technology status or to report progress. According

to the NASA Technology Plan, dated December 1998, the metric TRL specifies the maturity of a

technology, incrementally, from 1 (not mature) through 9 (mature). Therefore, TRL's can be

used to identify the incremental change in each Space Transportation technology as it matures

(from TRL 1 to 2 to 3, etc.). The TRL metric is “quantitative” (1 – 9), but the definition of each

TRL level may also add a “qualitative” aspect to this metric. The absence of an appropriate

Space Transportation technology metric can adversely affect the Space Transportation strategic

planning function, including the ability to effectively specify technology requirements,

technology status, and technology progress. Without a means to effectively identify, measure,

and report technology progress, strategic planning documents do not adequately serve their main

purpose as planning tools. We believe the use of TRL's can be a useful metric in carrying out

NASA’s technology management task for Space Transportation. Other Enterprises and the

Aeronautics element of the Office of Aero-Space Technology already use the TRL metric.

However, other metrics may also be appropriate. Regardless of which metric(s) is chosen,

NASA must determine the appropriate metrics to support Agency Space Transportation strategic

planning.
35

Appendix F. Agency Strategic and Performance Plans

Strategic Plans

The NASA Strategic Management Handbook, NPG1000.2, requires a strategic plan to have

goals, objectives, and implementing strategies. Goals elaborate on the mission statement and

need not be in a quantitative or measurable form, but they must be expressed in a manner that

allows a future assessment of whether each is being achieved.

Objectives are specific milestones and target levels of near-term outputs that are to be achieved

during strategic implementation. Objectives should be specific, measurable (either quantifiable

or verifiable to allow NASA to monitor success in achieving each objective), results oriented,

and time phased. Implementing strategies should include a description of how the goals and

objectives will be achieved. The Space Transportation objectives in the FY 1998 Strategic Plan

related to X-33 and X-34 are to:

• Complete research and development to enable U.S. industry to launch to Low Earth Orbit at

$1,000 /lb. (1998 – 2002)

• Revolutionize space launch capabilities – reduce the payload cost to low earth orbit by an

order of magnitude ($1,000 per pound to orbit) (2003 – 2009)

• Revolutionize space launch capabilities – reduce the payload cost to low earth orbit by an

order of magnitude (hundreds of dollars per pound to orbit). (2010 – 2023)

The objectives for the FY 1999 Strategic plan for Space Transportation are similar to those for

FY 1998, but do not include dollar values for reduced launch costs. The time-phased objectives

in the FY 1999 Strategic Plan are to:

• Complete R&D [research and development] to enable US industry to significantly reduce cost

of launches to Low Earth Orbit (LEO) [1998 – 2002];

• Reduce the payload cost to Low Earth Orbit by an order of magnitude. [2003 – 2009]; and

• Reduce the payload cost to LEO by an additional order of magnitude. [2010 – 2023].

Performance Plans

Our review of the objectives and performance measures in the FY 1999 NASA Performance Plan

included a comparison of strategic planning data for the Aero-Space Technology Enterprise’s

Aeronautics and Space Transportation missions. Our comparison showed Aeronautics objectives

and performance measures were more specific and more readily related to the Aeronautics goals

than the objectives and performance measures for Space Transportation. The Space

Transportation goal in the Enterprise’s FY 1999 Performance Plan is “Enable the full commercial

potential of space research and exploration,” and the objective is “Revolutionize space launch

capabilities.” The two performance measures are:
36

Appendix F

• Continue the X-33 vehicle assembly in preparation for flight testing

• Complete vehicle assembly and begin flight-testing of the X-34

Assuming these performance measures were met (neither the X-33 nor X-34 flew in FY 1999),

such objectives and performance measures do not readily provide meaningful metrics to measure

progress toward the goal. Appropriate metrics are needed to measure progress in acquiring

needed technologies and in moving toward the Agency goal of reduced Space Transportation

launch costs.

An example of one Aeronautics goal is “Enable U.S. leadership in global civil aviation through

safer, cleaner, quieter, and more affordable air travel.” An objective is “Contribute to aviation

safety by reducing the aircraft accident rate.” The performance measures for these objectives are:

• Characterize the Super-cooled Large Droplets (SLD) icing environment, determine its

effects on aircraft performance, and acquire and publish data to improve SLD forecasting

confidence.

• For the aviation safety areas of Controlled Flight into Terrain, runway incursion, and loss

of control, identify the contributing causes to be addressed, potential solutions using

current capabilities, and gaps that require technology solutions.

These Aeronautics performance measures clearly contribute to the objective and ultimately to

the goal. Performance measures for Space Transportation should likewise clearly contribute to

the objective and goal stated in the Performance Plan.
37

Appendix G. Space Transportation Architecture Study

In response to OMB direction in the FY 1999 budget passback guidance, NASA is performing a

Space Transportation Architecture Study. On July 20, 1998, NASA Headquarters issued a

Request for Offer to obtain industry input on the architecture study. The purpose of the study is

for industry to develop architecture options to meet NASA’s future space transportation

requirements with significant reductions in costs. Industry will compete its options for award,

and OMB will include the selected architecture in the President’s FY 2001 budget. The Space

Transportation Architecture Study called for up to nine architecture scenarios.45 The study is

being implemented in multiple phases including an initial phase to develop a detailed study

approach (completed in 1998) and one or more additional follow-on phases to fully develop

proposed space launch architectures with sufficient levels of detail to support decisions regarding

NASA budget.

In February 1999, in the second phase of the study, NASA received six proposed architectures

from private industry and the results of the NASA in-house study team. The Chief Engineer’s

Office briefed the Aeronautics and Space Engineering Board in June 1999. The study results

proposed alternative scenarios for keeping the Shuttle and/or for replacing it. Based on the study,

NASA initially proposed upgrading the Shuttle through 2008 and phasing it out by 2011 and

postponing the decision on specific criteria for the 2nd generation RLV until 2000.

However, in July 1999, NASA began Phase three of the architecture study because the NASA

Administrator was not satisfied with the Agency-identified Space Transportation requirements,

which were the baseline for the study. The Administrator considered the first set of requirements

too narrow in scope, because they addressed only currently planned Agency missions and did not

include other potential missions (space flights). NASA officials told us the Administrator

wanted another study (Phase 3) performed that considers the requirements not only for planned

missions, but also for missions not yet planned, such as satellite recovery, space exploration, and

building platforms in space. Human-rated missions46 and vehicles will be highlighted in the

Phase 3 study. NASA provided an initial briefing to OMB on October 5, 1999, with preliminary

recommendations on a Space Transportation architecture plan. NASA plans to give OMB a

final, more detailed briefing later in FY 2000.

NASA’s original goal of reducing launch costs to $1,000 per pound by 2006 probably will not be

met. Preliminary results of the current Space Transportation Architecture Study move the goal

beyond 2007 because a decision on the next-generation RLV cannot be made before 2002 or

2003 at the earliest and may be delayed until 2005.

 

45 Each scenario addresses various new vehicle options and/or Shuttle upgrades.

46 NASA space flight missions that have humans on board.
38

Appendix H. Program Documentation Requirements

NASA Policy Directive (NPD) 7120.4A, “Program/Project Management,” dated November 14,

1996, requires that program documentation, to include program commitment agreements and

program/project plans, be in place prior to the start of project implementation. Paragraphs 1.b.(6)

and (b) state:

• The Agency commitment to execute the program will be documented in the Program Commitment

Agreement which forms the basis for program implementation and evaluation plans.

• Signature on the Program Commitment Agreement by the Administrator and the Enterprise

Associate Administrator formalizes Agency approval of the program.

• In order to ensure that all supporting organizations and entities understand and commit to

providing the required support defined in the Program Commitment Agreement, all such

agreements/commitments must be represented in Program and Project Plans. These plans must

be in place prior to the start of implementation. [Emphasis added.]

NPG 7120.5A, "NASA Program and Project Management Processes and Requirements," dated

April 3, 1998, defines the requirements for formulating, approving, implementing, and evaluating

programs and projects and states: “The disciplined approach of program and project

management is now being applied to technology development programs to enable future

Agency missions.” [emphasis added]

NPD 7120.4A, "NASA Program/Project Management," dated November 14, 1996, establishes

the management system by which NASA shall formulate, approve, implement, and evaluate all

programs and projects established for development and operation of aeronautical and space

ground and flight systems and technologies. The process for approval of projects within a

program varies, depending on the size, nature, and criticality of the project. The specific process

for each program should be specified in the Program Commitment Agreement.

NPG 7120.5A, "NASA Program and Project Management Processes and Requirements," dated

April 3, 1998, establishes the management system for processes, requirements, and

responsibilities for implementing NPD 7120.4A. This management system governs the

formulation, approval, implementation, and evaluation of all Agency programs and projects

established to Provide Aero-Space Products and Capabilities. It is intended to support

accomplishment of the NASA programs and projects, consistent with established Agency

strategic planning, on schedule, and within budget, while satisfying the requirements of multiple

stakeholders and customers. The NPG provides the basic processes and requirements for the life

cycle of all programs and projects. The NPG shall be used specifically for programs/projects that

provide aerospace products or capabilities, that is, provide space and aeronautics, flight, and

ground systems, technologies, and operations satisfying the requirements of multiple

stakeholders and customers.
39

Appendix H

The NPG requires that program commitment agreements and program/project plans be prepared

in the formulation stage, prior to approval of the program/project. In fact, approval of the

agreement and plans constitutes Agency approval of the program/project. The NPG also

stipulates that program/project documentation should be consistent with and provide details for

implementing Agency and Enterprise strategic plans.
40

Appendix I. Management’s Response
41

Appendix I
42

Appendix I
43

Appendix I
44

Appendix I
45

Appendix I
46

Appendix I
47

Appendix I
48

Appendix I
49

Appendix I
50

Appendix I
51

Appendix J. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters

A/Administrator

AE/Chief Engineer

AI/Associate Deputy Administrator

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

H/Associate Administrator for Procurement

J/Associate Administrator for Management Systems

JM/Director, Management Assessment Division

R/Associate Administrator for Aero-Space Technology

RG/Director, Goals Division

RP/Director, Programs Division

NASA Center Directors

Director, George C. Marshall Space Flight Center

Chief Counsel, Kennedy Space Center

Non-NASA Federal Organizations and Individuals

Assistant to the President for Science and Technology Policy

Deputy Associate Director, Energy and Science Division, Office of Management and Budget

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office of

 Management and Budget

Associate Director, National Security and International Affairs Division, Defense Acquisition

 Issues, General Accounting Office

Professional Assistant, Senate Subcommittee on Science, Technology, and Space

Chairman and Ranking Minority Member - Congressional Committees and Subcommittees

Senate Committee on Appropriations

Senate Subcommittee on VA, HUD, and Independent Agencies

Senate Committee on Commerce, Science, and Transportation

Senate Subcommittee on Science, Technology, and Space

Senate Committee on Governmental Affairs

House Committee on Appropriations

House Subcommittee on VA, HUD, and Independent Agencies

House Committee on Government Reform
52

Appendix J

Chairman and Ranking Minority Member - Congressional Committees and Subcommittees

(Cont.)

House Subcommittee on Government Management, Information, and Technology

House Subcommittee on National Security, Veterans Affairs, and International Relations

House Committee on Science

House Subcommittee on Space and Aeronautics

Congressional Member

Honorable Pete Sessions, U.S. House of Representatives
NASA Reader Survey

Assistant Inspector General for Auditing

Reader Survey

The NASA Office of Inspector General has a continuing interest in improving the

usefulness of our reports. We wish to make our reports responsive to our customers’

interests, consistent with our statutory responsibility. Could you help us by completing our

reader survey? For your convenience, the questionnaire can be completed electronically

through our homepage at http://www.hq.nasa.gov/office/oig/hq/audits.html or can be

mailed to the Assistant Inspector General for Auditing; NASA Headquarters, Code W,

Washington, DC 20546-0001.

Report Title: X-34 Technology Demonstrator 

Report Number: Report Date: 

Circle the appropriate rating for the following statements.

Strongly

Agree Agree Neutral Disagree

Strongly

Disagree N/A

1. The report was clear, readable, and logically

organized.

5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A

3. We effectively communicated the audit

objectives, scope, and methodology.

5 4 3 2 1 N/A

4. The report contained sufficient information to

support the finding(s) in a balanced and

objective manner.

5 4 3 2 1 N/A

Overall, how would you rate the report?

Excellent Fair

Very Good Poor

Good

If you have any additional comments or wish to elaborate on any of the above responses,

please write them here. Use additional paper if

 necessary. 

 

 

 
 

 

 

How did you use the report? 

 

 

 

 

 

 

How could we improve our report? 

 

 

 

 

 

 

How would you identify yourself? (Select one)

Congressional Staff Media

NASA Employee Public Interest

Private Citizen Other: 

Government: Federal: State: Local: 

May we contact you about your comments?

Yes: ______ No: ______

Name: ____________________________

Telephone: ________________________

Thank you for your cooperation in completing this survey.
Major Contributors to this Report

Karen E. VanSant, Director, Aero-Space Technology Audits

Jim Linville, Auditor-in-Charge

Bill Falter, Auditor

Betty Weber, Operations Research Manager

Patricia Atkinson, Program Assistant

Nancy Cipolla, Report Process Manager