OCTOBER 31, 2008 

AUDIT REPORT

REPORT NO. IG-09-004 (ASSIGNMENT NO. A-07-011-01) 

OFFICE OF AUDITS

MORE STRINGENT ENTRANCE CRITERIA NEEDED FOR 

PROJECT LIFE-CYCLE REVIEWS

OFFICE OF INSPECTOR GENERAL

National Aeronautics and 

Space Administration 

 
REPORT NO. IG-09-004 

Final report released by: 

 signed 

Evelyn R. Klemstine 

Assistant Inspector General for Auditing 

Acronyms 

CxP Constellation Program 

DAC Design Analysis Cycle 

GAO Government Accountability Office 

NPR NASA Procedural Requirements 

PDR Preliminary Design Review 

POD Point-Of-Departure 

PNAR Preliminary Non-Advocate Review 

RID Review Item Discrepancy 

SDR System Definition Review 

SRB Standing Review Board 

SRR System Requirements Review 
OCTOBER 31, 2008 

REPORT NO. IG-09-004 

OVERVIEW

MORE STRINGENT ENTRANCE CRITERIA NEEDED FOR PROJECT 

LIFE-CYCLE REVIEWS

The Issue 

In January 2004, the President announced a Vision for U.S. Space Exploration that 

directed NASA to retire the Space Shuttle by 2010 and develop new launch vehicles for 

missions beyond low Earth orbit. To achieve the Vision, one of NASA’s first challenges 

is the acquisition and development of the Orion Crew Exploration Vehicle (Orion). In 

September 2006, NASA awarded a $3.9 billion contract to Lockheed Martin Corporation 

(Lockheed) for the design, development, testing, and evaluation of Orion with a period of 

performance through December 2011. In April 2007, the Agency and Lockheed 

negotiated a change to the contract effectively increasing the contract value to 

$4.3 billion and extending the performance period to October 2013. 

Since the 1980s, NASA has had difficulty bringing a number of projects to completion 

including the building of a next-generation spacecraft. The Agency spent an estimated 

$4.8 billion and years of effort on never completed projects such as the National AeroSpace Plane, X-33, X-34, Space Launch Initiative, and the International Space Station 

Propulsion Module. To improve project outcomes, the Government Accountability 

Office (GAO) issued several reports recommending that NASA develop sound 

acquisition concepts to capture specific product knowledge at key junctures in product 

development prior to proceeding with the project. As a result, NASA revised its program 

and project management guidance to require program and project “life cycles” with 

life-cycle reviews at key decision points. 

We initiated this audit because of the overall importance of the Orion Project and because 

it is one of the first space flight projects to implement the revised guidance. We 

evaluated NASA’s management of the Orion acquisition during the initial life-cycle 

phase established by NASA Procedural Requirements (NPR) 7120.5D, “NASA Space 

Flight Program and Project Management Requirements,” March 6, 2007, and the lifecycle reviews that occur during the initial life-cycle phase, “Concept and Technology 

Development” (Phase A).

Results 

The Orion Project Office (Project Office) conducted a Phase A life-cycle review with a 

vehicle configuration (606 vehicle) that was not at the proper maturity level to proceed to 

Phase B. Specifically, a required engineering design analysis conducted prior to the 
OVERVIEW

ii REPORT NO. IG-09-004 

life-cycle review disclosed that the vehicle configuration required a reduction in weight, 

power, and instrumentation. However, instead of delaying the Phase A life-cycle review 

until the correct vehicle configuration (607 vehicle) could be reviewed, the Project Office 

proceeded with a nonconforming vehicle. As a result, a significant portion of the vehicle 

configuration that eventually did proceed to Phase B did not receive the benefit of a 

Phase A life-cycle review, nor was it completely evaluated for compliance with 

requirements. 

NASA policy requires space flight projects to conduct life-cycle reviews during each 

phase of the project’s life cycle. These reviews are considered essential elements of 

conducting, managing, evaluating, and approving space flight projects. The Project 

Office conducted each of the required Phase A life-cycle reviews for projects—the 

System Requirements Review (SRR) and the System Definition Review (SDR)—in 

accordance with NPR 7120.5D. The Orion Standing Review Board (SRB) conducted an 

independent life-cycle review—the Preliminary Non-Advocate Review (PNAR)—which 

provides the Agency an expert assessment of the progress against the project baseline. 

As part of Orion’s systems engineering process, the Project Office performs analysis 

cycle reviews during and after each life-cycle review. Included in Phase A is a design 

analysis cycle (DAC) that integrates the analysis required to support the SDR and 

demonstrates that the proposed system design and operational concept meet the mission 

goals and objectives. The pre-SDR DAC analysis disclosed that the 606 vehicle 

configuration required a redesign to reduce weight, power, and instrumentation to fulfill 

the mission. In addition, the analysis showed that because the vehicle did not meet the 

mass and weight requirements, the 606 vehicle would not be at the maturity level to 

proceed to Phase B, “Preliminary Design and Technology Completion.” 

NPR 7123.1A, “NASA Systems Engineering Processes and Requirements,” March 26, 

2007, establishes the entrance criteria that each project needs to fulfill and the success 

criteria that the project must successfully demonstrate for each life-cycle review. 

NPR 7123.1A entrance criteria do not require known changes from engineering design 

analysis be incorporated into the relevant technical baseline products prior to holding the 

life-cycle review. Therefore, the Project Office could hold the SDR, on schedule, on the 

nonconforming vehicle instead of delaying the SDR until meeting the Constellation 

Program (CxP) mass and weight requirements. The Project Office entered the SDR 

process with the nonconforming vehicle configuration because it initiated a parallel 

“point-of-departure” (POD) activity, which it believed would ultimately satisfy 

NPR 7120.5D requirements with the least expenditure of Agency resources. 

By conducting the SDR with the 606 vehicle configuration and declaring the SDR a 

success, the Project Office (and the SRB) communicated the impression that the 

606 vehicle was at the maturity level needed to proceed to Phase B of the project’s life 

cycle, when it was not. In addition, the Agency was obligated to pay Lockheed its award 

fee of $41.4 million, which was based, in part, on the SDR being held by August 31, 

2007. The Project Office was able to conduct the review with the nonconforming vehicle 
OVERVIEW

REPORT NO. IG-09-004 iii 

because NASA’s life-cycle reviews’ entrance criteria do not require that the project 

incorporate the results from engineering analysis cycles conducted prior to the life-cycle 

review. 

Without a fully successful SDR, the acquisition methodology is materially undermined 

and the Agency cannot be assured that life-cycle milestones vital to success are achieved. 

NASA needs to ensure that the vehicle configuration and requirements are at the proper 

maturity level prior to starting and completing future life-cycle phases. By not doing so, 

there is increased risk that new development is conducted prematurely, with concomitant 

risk of costly rework and schedule slips in managing the multibillion dollar Orion 

Project. 

Management Action 

Our September 9, 2008, draft of this report recommended that NASA revise 

NPR 7123.1A entrance criteria for internal life-cycle reviews to require that the technical 

products reviewed incorporate into the technical baseline known requirement changes 

resulting from engineering analysis cycle assessments. In addition, to ensure that the 

607 vehicle configuration was scrutinized at a minimum of the Phase A SDR level, we 

recommended that the Agency evaluate the vehicle configuration to determine the impact 

of any missed work and, if material, perform an SDR on affected systems. Finally, to 

ensure that the contractor does not prematurely receive award fees, we recommended the 

Agency ensure the contractor receive award fees based on the maturity of the relevant 

technical baseline rather than on holding a scheduled milestone. 

Management’s Comments and OIG Response. In response to a draft of this report 

(see Appendix G, “Management Comments”), the Chief Engineer did not concur with 

our recommendation to revise NPR 7123.1A entrance criteria, stating that the current 

policy is effective as written. The Chief Engineer also stated that there is no technical 

baseline of the design at this point in the life cycle. Trade studies and technology 

maturation are still occurring such that by the Preliminary Design Review (PDR), a 

preliminary design is in place that meets all system requirements. 

We consider management’s comment to this recommendation to be nonresponsive. We 

disagree that the current policy is effective. The policy as written provides for what 

amounts to a system status review rather than a life-cycle review to examine whether 

NASA had achieved a milestone in project development. While system status is useful, 

it is not what we believe is intended by the life-cycle review process, which focuses on 

technical maturity as opposed to calendar milestones. If the technical maturity 

established pursuant to criteria does not exist, a life-cycle review should not be declared 

a “success.” Material deficiencies should be addressed and resubmitted for review. 

The policy as written allowed the Orion Project Office to hold the SDR on a 

nonconforming vehicle configuration that did not meet known requirements nor was the 
OVERVIEW

iv REPORT NO. IG-09-004 

vehicle able to meet the review’s success criteria. In addition, although there may not be 

a technical baseline for the actual design available for review at the SDR, NASA’s 

guidance is clear that the Project is developing the technical baseline during this lifecycle phase. We commend the Project Office for delaying its PDR on two different 

occasions (and its consideration of a third delay), to ensure that the relevant technical 

baseline is reviewed. We believe these delays are consistent with making the PDR a true 

life-cycle review. However, without a revision to NPR 7123.1A requiring review of the 

relevant technical baseline at each life-cycle review, instances might again arise when 

reviews are performed on a premature configuration in compliance with the entrance 

criteria, yet be unable to meet known technical requirements or the success criteria. We 

request that the Chief Engineer reconsider his response to this recommendation and 

provide additional comments by November 26, 2008. 

The Associate Administrator for Exploration Systems Mission Directorate did not concur 

with our recommendation to evaluate the Phase B vehicle configuration to assure that it 

was scrutinized at a minimum of the SDR level, stating that no additional review is 

warranted. However, the Associate Administrator did concur with our recommendation 

that the contractor receives award fees based on the maturity of the relevant technical 

baseline, stating that NASA had already acted accordingly for the first Orion award fee 

period covered during the audit. 

Although the Associate Administrator for Exploration Systems Mission Directorate did 

not concur with our recommendation to evaluate the Phase B vehicle configuration, the 

Agency’s actions are responsive to the intent of our recommendation. Technical subject 

matter experts and Agency stakeholders eventually evaluated the 607 vehicle 

configuration, albeit outside of the formal SDR process. We believe the Agency’s delay 

of the Phase B PDR because the known technical baseline changes are not at the proper 

maturity level is consistent with our recommendation. Likewise, although we disagree 

with the Associate Administrator’s assertion that NASA acted in accordance with the 

recommendation during award fee period one, Agency actions taken during award fee 

period two are in accordance with the recommendation. NASA adjusted the period two 

award fee milestone to coincide with the delayed PDR and by so doing, NASA 

demonstrated that the Orion Project acted in accordance with the intent of our 

recommendation. We consider these recommendations resolved and closed. 
OCTOBER 31, 2008 

REPORT NO. IG-09-004 

CONTENTS

INTRODUCTION

Background __________________________________________ 1 

Objectives ___________________________________________ 3 

RESULTS

Orion SDR Conducted with a Nonconforming Vehicle __________ 4 

APPENDIX A 

Scope and Methodology _______________________________ 19 

Review of Internal Controls _____________________________ 21 

Prior Coverage_______________________________________ 21 

APPENDIX B 

Life Cycle of a NASA Space Flight Project __________________ 24 

APPENDIX C 

System Requirements Review Entrance/Success Criteria ______ 25 

APPENDIX D 

System Definition Review Entrance/Success Criteria _________ 26 

APPENDIX E 

Orion’s Analysis Cycles Performed Between Life-Cycle Reviews _ 27 

APPENDIX F 

Preliminary Design Review Definition Activity Memorandum ___ 28 

APPENDIX G 

Management Comments _______________________________ 33 

APPENDIX H 

Report Distribution ___________________________________ 37 

OCTOBER 31, 2008 

REPORT NO. IG-09-004 1 

INTRODUCTION

Background 

Orion is one of six projects under the Constellation Program (CxP).1 CxP is managed 

from the Johnson Space Center (Johnson) and is responsible for designing and 

developing the next generation of NASA space vehicles, which will send human 

explorers back to the moon and onward to Mars and other destinations in the solar 

system. CxP scheduled the new space vehicles for three unmanned test flights in 

April 2009, September 2012, and March 2013, with the first manned flight about March 

2015. NASA plans a manned flight to the moon before 2020.

Orion Crew Exploration Vehicle (CEV). Orion is the crew capsule for the next 

generation of space vehicles and will be used to transport crew and cargo from Earth to 

space and return. As shown in Figure 1, the Orion Project includes four elements: (1) a 

launch abort system (emergency escape during launch), (2) a crew module (crew and 

cargo transport), (3) a service module (propulsion, electrical power, and fluids storage), 

and (4) a spacecraft adapter (structural transition to CEV launch vehicle). 

Orion will be capable of transporting up to six crewmembers to and from the 

International Space Station, four crewmembers for lunar missions, and pressurized cargo 

to and from the International Space Station without a crew. Orion will also support crew 

transfers for future Mars missions. 

 

1

 The other five CxP projects are the Crew and Cargo Launch Vehicles (Ares I and Ares V), Ground 

Operations, Mission Operations, Extra-Vehicular Activity Systems, and Lunar Lander projects. 
INTRODUCTION

2 REPORT NO. IG-09-004 

Figure 1. Orion Project Elements 

Source: Crew Exploration Vehicle Project Plan, December 18, 2006 

Orion Project Management. The development of Orion is a joint effort involving every 

NASA Center. While Johnson is leading the development of Orion’s crew module, 

Langley Research Center is leading the development of the Orion launch abort system, 

and Glenn Research Center is leading development of the Orion service module and 

spacecraft adapter. Kennedy Space Center will lead the pre-flight processing and launch 

operations for Orion. Lockheed is NASA’s prime contractor for the design, 

development, testing, and construction of Orion. 

Project Management Guidance. NASA’s primary project management guidance for 

space flight projects is contained in NPR 7120.5D, “NASA Space Flight Program and 

Project Management Requirements,” March 6, 2007, and NPR 7123.1A, “NASA 

Systems Engineering Processes and Requirements,” March 26, 2007. NASA revised 

NPR 7120.5D and NPR 7123.1A in response to multiple GAO reports that recommended 

NASA change its approach to program and project management, specifically as it related 

to the acquisition process. GAO stated that NASA’s acquisition framework did not 

provide the information needed to make major investment decisions, which contributed 

to NASA’s difficulties in meeting cost, schedule, and performance objectives for its 
INTRODUCTION

REPORT NO. IG-09-004 3 

programs and projects.2 The revised NPR 7120.5D introduced the concept of program 

and project “life cycles” and defined the various key decision points and life-cycle 

reviews to manage each space flight program or project (see Appendix B for a space 

flight project’s life cycle). The revised NPR 7123.1A established entrance and success 

criteria for each of the life-cycle reviews. The Orion Project was one of NASA’s first 

space flight projects to implement that new guidance. 

Objectives 

The overall objective of this audit was to evaluate the management of the Orion 

acquisition. Because the acquisition was in the “Concept and Technology Development” 

phase of the project life cycle (Phase A), we reviewed and evaluated management’s 

efforts to ready the project for the “Preliminary Design and Technology Completion” 

phase of the project life cycle (Phase B). We specifically focused on the life-cycle 

reviews conducted during Phase A—System Requirements Review (SRR), System 

Definition Review (SDR), and Preliminary Non-Advocate Review (PNAR). We also 

reviewed internal controls as they related to the overall objective. See Appendix A for 

details of the audit’s scope and methodology, our review of internal controls, and a list of 

prior coverage. 

 

2

 Government Accountability Office, “NASA-Implementing a Knowledge-Based Acquisition Framework 

Could Lead to Better Investment Decisions and Project Outcomes” (GAO-06-218, December 21, 2005). 
RESULTS

4 REPORT NO. IG-09-004 

ORION SDR CONDUCTED WITH A 

NONCONFORMING VEHICLE

Although the Orion Project conducted each of the Phase A life-cycle reviews 

required by NPR 7120.5D (SRR, SDR, and PNAR), the SDR was conducted with a 

vehicle configuration (606 vehicle) that did not meet mass and weight requirements 

levied by the CxP. An engineering design analysis conducted to refine the 

spacecraft for the SDR determined that the 606 vehicle configuration required a 

reduction in weight, power, and instrumentation before it could meet CxP 

requirements. Instead of delaying the SDR until a vehicle redesign was complete 

(607 vehicle), the Orion Project elected to hold the SDR with the nonconforming 

606 vehicle and on August 30, 2007, reported in the SDR Board minutes that the 

SDR had been “successful.” The Project Office was able to conduct the SDR with a 

nonconforming vehicle because NASA guidance does not require that results of 

engineering design analyses be considered when determining whether a project is 

ready for its next life-cycle review. By conducting the SDR and declaring it a 

success, the Project Office communicated the impression that the vehicle was at the 

maturity level needed to proceed to Phase B of the project’s life cycle, when it was 

not. In addition, the Agency was obligated to pay the prime contractor its award fee 

of $41.4 million, which was based, in part, on the contractor holding the SDR by 

August 31, 2007. 

Life-Cycle Review Process 

NPR 7120.5D requires NASA space flight projects to conduct life-cycle reviews during 

each phase of the project’s life cycle. The reviews are considered essential elements of 

conducting, managing, evaluating, and approving space flight projects. Generally, a 

project first conducts an “internal review,” wherein the project management solidifies 

their plans, technical approaches, and programmatic commitments. During the internal 

review, functional area experts from across the Agency evaluate the project’s technical 

documentation for noncompliance or conflicts with requirements. If a discrepancy is 

identified, the functional area expert will initiate and submit a review item discrepancy 

(RID). 

RIDs can be initiated against incorrect, incomplete, or missing requirements or other 

areas in which the technical documentation is incomplete or in error. The RIDs go 

through a series of reviews whereby selected technical personnel further screen the RIDs 

and determine whether to recommend the RID for approval, approval with modification, 

dismissal, or merger with another RID. Approved RIDs are forwarded to a review panel 

and a pre-board; they ensure that the project’s technical documents are revised in 

accordance with the RIDs and that the RIDs are properly dispositioned and closed. 
RESULTS

REPORT NO. IG-09-004 5 

Lastly, the internal project review board assesses the RID dispositions, schedules of 

forwarded RID work, and recommendations and closure plans for any outstanding RIDs. 

While all RIDs must be dispositioned prior to proceeding to the next life-cycle review, 

the RIDs do not have to be closed if the project has a plan to address the discrepancy. If 

a RID is dismissed during the review process, the initiator is notified and may issue a 

reclama directly to the internal project review board for consideration. 

Once the internal review is complete, the project undergoes an independent review by the 

project’s Standing Review Board (SRB). NPR 7120.5D requires that an SRB be 

established for each space flight project. The SRB reports on the adequacy and 

credibility of the project’s technical and management approach, schedule, resources, cost, 

and risk; the project’s compliance with Agency management and systems engineering 

guidance; and the project’s readiness to proceed to the next life-cycle phase. SRB 

members are chosen based on their management, technical, and safety and mission 

assurance expertise; their objectivity; and their ability to make a broad assessment of 

space flight projects. To ensure the objectivity of the SRB, NPR 7120.5D requires that 

the members be independent of the project under review; that is, none of the members 

should have a stake in the outcome of any of the life-cycle reviews or in the project 

itself.3 Because the SRB is solely an advisory board, NASA management is not required 

to act on the SRB’s findings and recommendations; however, NASA management must 

consider the SRB report when deciding whether the project should proceed to the next 

life-cycle phase. 

Phase A Life-Cycle Reviews 

During Phase A of a space flight project’s life cycle, the project undergoes three lifecycle reviews—SRR, SDR, and PNAR. The results from those reviews are presented to 

the decision authority who determines the readiness of the project to progress to Phase B4

of the life cycle. 

SRR. The SRR is the project’s first Phase A life-cycle review in which the functional 

and performance requirements and the preliminary project plan are examined to ensure 

that the requirements and selected concept will satisfy the mission. Prior to initiating the 

SRR, a project needs to fulfill the entrance criteria listed in NPR 7123.1A, Appendix G, 

Table G-4, “SRR Entrance and Success Criteria” (see Appendix C). Those entrance 

 

3

 In a previously issued NASA OIG report (IG-08-018, “Final Memorandum on the Standing Review 

Board for the Orion Crew Exploration Vehicle Project,” April 28, 2008), we found that not all Orion 

SRB members met the independence requirements as defined in NPR 7120.5D. Based on our finding, the 

Agency is revising its SRB guidance. 

4

 Phase B is the second phase of a project’s life cycle and culminates with the preliminary design review, 

which is the first life-cycle review in the Phase B life-cycle and provides information for Key Decision 

Point C. 
RESULTS

6 REPORT NO. IG-09-004 

criteria require that the project successfully complete a Mission Concept Review,5

prepare a preliminary SRR agenda, and define success criteria by which the project will 

be measured. The project must also have a number of technical products available for the 

functional area experts and the SRB to review including the system requirements 

documents, the baselined systems engineering management plan, the risk management 

plan, and the system safety and mission assurance plan. At the conclusion of the SRR, 

the project must demonstrate to the SRR internal review board that it has met the 

SRR success criteria listed in NPR 7123.1A, to include development of a sound process 

for allocating and controlling requirements and a method of validating those 

requirements. The project must also have identified and technically assessed its major 

risks and developed mitigation strategies to address those risks. 

SDR. The SDR is the project’s second Phase A life-cycle review in which the proposed 

requirements, the mission/system architecture, and the flow down of requirements to all 

functional elements are reviewed. As with the SRR, the project needs to fulfill the SDR 

entrance criteria before proceeding with the review (see Appendix D). Those entrance 

criteria require that the project have successfully completed its SRR, prepared a 

preliminary SDR agenda, and have certain technical products, such as the preliminary 

functional baseline (with supporting trade-off analyses and data), available for the 

functional area experts and SRB review. In addition, any updates to the technical 

products initially reviewed during the SRR are resubmitted for the SDR. At the 

conclusion of the SDR, the project must demonstrate to the SDR internal review board 

that it has met the SDR success criteria. Included in NPR 7123.1A’s list of success 

criteria is an allocation of all technical requirements, development of a credible technical 

approach that is responsive to the identified requirements, and the existence of a process 

to manage the identified and technically assessed development, mission, and safety risks. 

The project must also demonstrate that the requirements, design approaches, and 

conceptual design can fulfill the mission needs consistent with available resources (cost, 

schedule, mass, and power). 

PNAR. The PNAR is the Phase A independent life-cycle review and is conducted to 

specifically provide NASA management with an independent assessment of the project’s 

readiness to proceed to Phase B of its life cycle. The PNAR is conducted solely by the 

SRB and is designed to be held in conjunction with the project’s SDR. The PNAR is 

considered complete when the SRB out-briefs the decision authority. 

SRR Entrance and Success Criteria Met 

The Orion 606 vehicle configuration met the SRR entrance and success criteria 

established in NPR 7123.1A. As required by the NPR’s entrance criteria, the project 

documented its SRR agenda, defined success criteria to measure the Orion’s progress, 

 

5

 The Mission Concept Review is conducted during Pre-Phase A to affirm the mission needs, examine the 

proposed objectives, and examine the concept for meeting those objectives. 
RESULTS

REPORT NO. IG-09-004 7 

and provided the required technical products to the functional area experts and the SRB 

for review. Although the project did not conduct a separate Mission Concept Review as 

required by the SRR entrance criteria, the project was able to justify that the Mission 

Concept Review objectives were accomplished as part of the Exploration Systems 

Architecture Study,6

 which preceded the Orion Project start. 

The Orion SRR was initiated in January 2007 and from January 23, 2007, through 

February 26, 2007, the functional area experts evaluated the SRR technical 

documentation for assurance that requirements were necessary, achievable, verifiable, 

clear, and consistent. Deficiencies resulted in RIDs, which were processed through the 

formal RID disposition process. Ultimately, the Orion Project’s internal SRR Board 

reviewed a summary of the RID dispositions along with its review of project requirements, 

technical status, design compliance, closure criteria, schedules of forwarded work, and 

recommendations and closure plans for open items. Based on that review, the SRR Board 

concluded on March 1, 2007, that the Orion Project had met all the requirements needed 

for a successful SRR to include meeting the SRR success criteria. In addition, the SRB 

also completed its independent review, concluded that the SRR was complete, and 

recommended the Project proceed to SDR with the 606 vehicle configuration. 

Based on our analysis of the SRR entrance and success criteria and evaluation of the 

Orion Project’s support for meeting those criteria, we agreed with management’s 

determination that the Project successfully completed its SRR and that the 606 vehicle 

configuration was at the maturity level needed to proceed to the SDR. Our evaluation 

included reviewing documentation to support that the Exploration Systems Architecture 

Study accomplished the intent of the Orion Mission Concept Review, verifying that key 

driving requirements were identified, as prepared, and verifying that traceability 

functions were performed on those requirements. The Project Office provided us with 

documentation illustrating how CxP requirements trace to the subsystem level and we 

observed an example of the requirements flow down. We verified that the system and 

subsystem conceptual design approach and operational concepts existed and were 

consistent with the Orion’s system requirement set. In addition, we verified that the 

Project had a Risk Management Plan, identified major risks, and defined mitigation 

strategies. We evaluated the Orion Project’s RID process and the Project was able to 

show evidentiary matter that all RIDs and action items were dispositioned. We attended 

the SRR Board meeting on March 1, 2007, and reviewed the Board minutes for decisions 

made and forward action plans. We also reviewed the SRB’s issued report to verify that 

the SRB concluded that Orion should proceed to the SDR. 

 

6

 The Exploration Systems Architecture Study was chartered by the NASA Administrator to complete an 

assessment of the top-level Crew Exploration Vehicle requirements. 
RESULTS

8 REPORT NO. IG-09-004 

SDR Success Criteria Not Met 

Although the Orion 606 vehicle configuration met the SDR entrance criteria, it did not 

meet the SDR success criteria because the 606 vehicle configuration used during the 

SDR did not meet known mass and weight requirements levied by the CxP. 

At the completion of the SRR, NASA management and the SRB agreed that the 

606 vehicle configuration was adequate to proceed to SDR. In preparation for the SDR, 

the Project Office initiated a “pre-SDR DAC” on the 606 vehicle configuration. The 

DAC is required by Orion’s System Engineering Management Plan7 and supports the 

SDR by demonstrating that the proposed system design and operational concept meets 

the mission goals and objectives. During the DAC, multiple project office organizations 

analyzed the system requirements, risks, system concept and architecture, trades 

assessments, and cost and schedule. (See Appendix E for the summary of all Orion’s 

planned analysis cycles during its life cycle.) The Project Office completed the DAC in 

June 2007 and concluded that the SRR 606 vehicle configuration required a 

reconfiguration to reduce weight, power, and instrumentation to meet CxP requirements 

and proceed to Phase B. Known CxP requirements included a vehicle with a target 

payload mass of 50,250 pounds (5,522 pounds less than the 606 vehicle’s effective 

payload mass of 55,772 pounds) and gross lift-off target weight of 66,504 pounds 

(4,387 pounds less than the 606 vehicle’s gross liftoff weight of 70,891 pounds). 

To address the mass and weight deficiencies disclosed during the DAC, the Project 

Office initiated a reconfiguration activity denoted as the POD activity. On June 8, 2007,8

the Orion Project Manager and the Lockheed Orion Program Manager signed 

memorandum ZV-07-005 outlining that a Phase B vehicle reconfiguration activity would 

be performed simultaneously with the SDR. The memorandum stated that the vehicle 

configuration resulting from the POD activity would be the 607 vehicle that would 

proceed to Phase B. (See Appendix F for a copy of the complete memorandum.) To 

accomplish the POD activity, the Project Office examined the existing 606 vehicle and 

“stripped back” (took out) all components and functions requirements that were not vital 

to the lunar mission. The stripped back vehicle was denoted as the “Zero Based Vehicle” 

and the components and functions removed were maintained in a database that was the 

principal repository for identifying any removed items. Each item maintained in the 

database was prioritized to meet safety, robustness, and mission objectives and the 

weight, power, and thermal margins. After the prioritization, components and functions 

were then added back to the Zero Based Vehicle as allowed by the mass and weight 

margins. The Project Office had two rounds of “buy backs” where components and 

functions were added back to the vehicle. The first round of buy backs focused on safety, 

while the second round of buy backs focused on the remaining components and functions 

 

7

 CxP 72088, “Crew Exploration Vehicle Systems Engineering Management Plan,” November 8, 2006. 

8

 The Project Office erroneously dated the memo June 8, 2006. We confirmed with the Project Office that 

the date should be June 8, 2007. 
RESULTS

REPORT NO. IG-09-004 9 

in the database. The Orion Vehicle Integration Office and Lockheed Systems 

Engineering and Integration Team led the effort with the Orion Vehicle Engineering 

Integrated Working Group performing the day-to-day activities. The Orion Vehicle 

Engineering Integrated Working Group performed the POD activity to establish 

requirements for the 607 vehicle in parallel with the Project Office’s SDR of the 

606 vehicle. 

Consistent with the decision to hold the POD and SDR simultaneously, the Project Office 

initiated the SDR on July 11, 2007, with the 606 vehicle configuration, which did not 

meet known CxP mass and weight requirements and, therefore, could not meet the SDR 

success criteria. Specifically, the 606 vehicle did not meet the SDR success criteria that 

states “[t]he requirements, design approaches, and conceptual design will fulfill the 

mission needs consistent with the available resources (cost, schedule, mass, and power).” 

In addition, because the 606 vehicle’s payload mass and lift-off weight could not meet 

the mission needs, we do not believe that Orion met the success criteria that “[t]he 

technical approach is credible and responsive to the identified requirements.” 

Furthermore, until the POD activity was complete, the success criteria “[t]he tradeoffs are 

completed, and those planned for Phase B adequately address the option space” could not 

be met. 

The Project Office conducted the SDR in accordance with NPR 7120.5D and met the 

NPR 7123.1A entrance criteria as written. The Project Office met NPR 7123.1A’s 

entrance criteria by demonstrating the successful completion of the SRR, providing a 

preliminary SDR agenda, providing the SDR success criteria, and providing the technical 

documents for review to the functional area experts and the SRB. According to the 

Assistant Orion Project Manager for Integration, the Project Office believed that its 

strategy of performing the SDR parallel with the POD activity would ultimately address 

NPR 7120.5D requirements with the least expenditure of Agency resources. 

At the conclusion of the SDR, the SDR Board documented in the Board minutes that the 

606 vehicle design did not meet the mass requirements and trade studies could not be 

completed because the POD activity was ongoing. In addition, the Board minutes 

identified that although the SDR process allowed for a thorough review of the 

606 vehicle, the POD activity (607 vehicle) would have relatively little review time and 

there was a concern about the tight schedule. On August 30, 2007, the SDR Board 

declared the SDR successful by asserting that the SDR was a snapshot review to 

understand the issues and the Project had plans, such as the POD activity, to address 

those issues. 

To evaluate whether the Project Office conducted the SDR in accordance with 

NPR 7123.1A, we verified how the entrance criteria were met. As discussed earlier, the 

Project Office successfully completed the SRR and dispositioned all the SRR RIDs and 

action items resulting from the SRR. The SDR Process Plan9

 documented the purpose of 

 

9

 CxP 72107, “Constellation Program Crew Exploration Vehicle System Definition Review Process Plan,” 

June 22, 2007. 
RESULTS

10 REPORT NO. IG-09-004 

the SDR as well as the entrance and success criteria the Project would meet. We 

attended the SDR Kick-off meeting and observed several pre-board meetings. In 

addition, the required technical products were made available for the functional area 

experts and the SRB to review. We concluded that the Project Office’s documented 

entrance criteria were met. However, the entrance criteria as written failed to incorporate 

known technical baseline changes prior to the life-cycle review. The POD activity was 

based on known mass and weight issues that could result in integrated design impacts 

across all subsystems addressing Orion’s mass and power and result in reassessments of 

reliability, operability, safety, and cost. 

Based on our review of the parallel POD activity, attendance at the board meetings, and 

the SDR Board minutes, we believe that the 606 vehicle did not meet the success criteria 

because the design did not fulfill the known mission needs consistent with the available 

resources. The vehicle did not meet the mass and weight criteria, thus the technical 

approach was not responsive to identified CxP requirements and the tradeoffs (POD 

activity) were not complete. As a result, the SRB was not able to conduct its PNAR in 

conjunction with the SDR. 

PNAR Not Conducted in Conjunction with SDR 

The SRB declined to conduct the PNAR in conjunction with the SDR because the 

606 vehicle reviewed at the SDR did not meet requirements. The SRB requested that the 

Project Office provide them a briefing after the POD activity established the correct 

vehicle configuration (607 vehicle), which was scheduled for November 2007. During a 

September 27, 2007, interview with our office, the SRB Chairman stated that the vehicle 

configuration that went through the SDR (606 vehicle) was irrelevant. The SRB was 

subsequently briefed on the 607 vehicle configuration in December 2007 and was able to 

review the Phase B vehicle. Based on the December 2007 briefing, the SRB issued its 

PNAR report10 on January 23, 2008, which stated that the Project faced significant 

technical issues11 that would require major efforts to resolve before a valid Preliminary 

Design Review (PDR) could be conducted. The SRB report also noted that the schedule 

was not achievable within the defined budget and that the assessed cost was significantly 

higher than the currently defined budget. However, despite the issues and concerns 

documented in the PNAR report, the SRB recommended to the decision authority that the 

Project should proceed, with the 607 vehicle configuration, to its PDR in Phase B. The 

SRB out-briefed the decision authority on April 29, 2008, with the results as written in 

the PNAR report. 

 

10 Orion Project Standing Review Board “Assessment Report System Definition Review (SDR) & 

Preliminary Program Approval Review (PPAR),” January 23, 2008 

11 The technical issues include mass-margin threats, design impact threats from Ares I thrust oscillations, 

and an adequate definition for loss of crew and loss of mission. 
RESULTS

REPORT NO. IG-09-004 11 

Entrance Criteria Do Not Require Incorporation of Design Analysis 

Results 

NASA’s entrance criteria do not require that projects incorporate results of engineering 

design analyses (the DAC in this case) before initiating the life-cycle review, which 

allowed the Project Office to conduct the SDR with the nonconforming 606 vehicle. The 

entrance criteria do require that projects respond to all prior life-cycle review’s request 

for actions and RIDs (the SRR in this case). However, the entrance criteria do not 

require that known changes to the technical baseline resulting from the required 

engineering analysis cycles performed between life-cycle reviews be incorporated into 

project technical documents. If the entrance criteria had considered the design analysis 

results, the SDR would not have concluded on August 30, 2007, but would have been 

delayed until the POD activity assessment was complete. NPR 7123.1A states that an 

important point of a life-cycle review is that “. . . reviews are event based and occur when 

the entrance criteria for the applicable review as specified in Appendix G are satisfied. 

They occur based on maturity of the relevant technical baseline as opposed to calendar 

milestones . . ..” Although NPR 7123.1A provides entrance criteria for a life-cycle 

review to begin, the criteria do not address the maturity of the relevant technical baseline. 

The entrance criteria state that the previous life-cycle review (SRR in this case) must be 

complete; however, the criteria do not require that known changes in the technical 

baseline be incorporated prior to starting the review. The Project Office was able to hold 

the SDR on the nonconforming 606 vehicle, on schedule, because NPR 7123.1A entrance 

criteria do not incorporate the results of engineering design analyses even though the 

project is required to conduct the design analysis prior to the life-cycle review. The 

Project Office proceeded with the SDR instead of adjusting the SDR milestone even 

though it became apparent in June 2007 that the appropriate technical baseline 

(607 vehicle) would not be ready for the scheduled SDR. 

“Successful” SDR Implied Phase B Maturity Level 

The Orion Project Office did not fully meet the criteria to declare the SDR successful 

because the 606 vehicle configuration did not fulfill the mission needs. The 606 vehicle 

required a reconfiguration to reduce weight, power, and instrumentation in order to meet 

requirements before proceeding to Phase B. By conducting the SDR and declaring it a 

success, the Project Office communicated the impression that the SDR vehicle was at the 

maturity level needed to proceed to Phase B of its life cycle, when it was not. As stated 

in NPR 7120.5D, “NASA places significant emphasis on project formulation because 

adequate preparation of project concepts and plans is vital to success.” During 

formulation (Phase A and Phase B), project staff develops and defines the requirements, 

cost, and schedule and designs a plan for the project to proceed to the implementation 

phases.12 NPR 7120.5D also states that project reviews “are essential elements of 

 

12 Implementation phases are Phase C, Final Design and Fabrication; Phase D, System Assembly, 

Integration and Test, and Launch; Phase E, Operations and Sustainment; and Phase F, Closeout. 
RESULTS

12 REPORT NO. IG-09-004 

conducting, managing, evaluating, and approving space flight programs/projects.” 

Additionally, as part of the required NPR 7123.1A engineering process, Orion’s 

documented engineering process states that any changes to the requirements that 

occurred since the SRR will be verified at the SDR.13 The purpose of the SDR is 

essentially negated if a known incorrect or irrelevant design is reviewed. Without a 

proper SDR, the Agency has limited assurance that project concepts and plans vital to 

success are achievable. 

Since the 607 vehicle configuration that proceeded to Phase B was not reviewed by all 

functional area experts during the SDR, the redesigned vehicle did not go through the 

rigorous RID process to determine whether the technical data was in conflict with 

requirements. As such, the functional area experts did not validate a significant portion 

of the redesigned vehicle. A vehicle design that does not benefit from the rigors of the 

full SDR review process could lead to extensive rework, schedule slippage, and increased 

cost because an unacceptable design was allowed to proceed. For example, on the 

International Space Station’s Propulsion Module Project, NASA did not validate 

requirements from the SRR before beginning the PDR. As a result, the Agency spent 

$97 million and 19 months of effort before it determined that the design was 

unacceptable.14 The Orion SDR process evaluated the 606 vehicle and implied it was at 

the maturity level to proceed to Phase B; however, the vehicle configuration that will 

proceed to Phase B is the POD 607 vehicle. Although the 607 vehicle configuration 

received constant review and analysis from the Project Office and CxP, and the SRB was 

ultimately able to perform the independent PNAR (albeit about 4 months later), the 

607 vehicle configuration did not obtain the full SDR environment review process as 

required by NPR 7120.5D. The Project Office should have delayed the SDR until the 

607 vehicle configuration was ready because they knew the 606 vehicle configuration 

was not the vehicle going to Phase B. 

Award Fee Paid Prior to SDR Milestone Completion 

NASA awarded the prime contractor its award fee of $41.4 million15 because, in part, the 

contractor held the SDR by August 31, 2007, and the Project Office considered the SDR 

successful. NASA’s contract with Lockheed is performance milestone driven, which 

means that interim award fee periods and provisional payments are based on the 

contractor’s successful completion of milestones within the agreed-to schedule. 

Lockheed’s award fee plan states that 

 

13 Orion’s engineering process is documented in CxP 72088, “Crew Exploration Vehicle, Systems 

Engineering Management Plan,” November 8, 2006, and Revision A, July 10, 2007. 

14 NASA OIG, “Acquisition of the Space Station Propulsion Module” (IG-01-027, May 21, 2001). 

15 The total potential award fee was $XX.X million. However, NASA only pays up to 80 percent of the 

maximum potential award until the end of the contract when any adjustments may be calculated and 

applied. 
RESULTS

REPORT NO. IG-09-004 13 

Each award fee period will be based on objective project milestones identified in this 

plan and contract schedule. Each Interim award fee period and provisional payment 

is based on the Contractor’s successful completion of the milestone(s) agreed to and 

as identified in the Contractor’s Integrated Master Schedule. The award fee 

distribution table contained in Enclosure II, Performance Milestones, provides a list 

of performance milestones and available provisional fee for each milestone. 

Enclosure II of the award fee plan specified that the SDR would be held by August 31, 

2007. If the Project Office had delayed the SDR until the 607 vehicle configuration 

activity was completed, NASA would have been required to modify Lockheed’s award 

fee plan to compensate for the SDR slippage past August 31, 2007. However, the 

Project’s SDR Board met on August 30, 2007, and declared the SDR “successful” despite 

performing the review on a vehicle configuration that did not meet CxP requirements. In 

its self-evaluation for the award fee milestone, Lockheed stated that the SDR was 

successful. The contracting officer’s technical representative’s16 September 27, 2007, 

presentation to the award fee evaluation board confirmed that an activity for this award 

fee period was to conduct the SDR Board on August 30, 2007. Based on these and other 

presentations, on September 27, 2007, the award fee evaluation board graded Lockheed’s 

performance as excellent,17 which resulted in Lockheed receiving the maximum award 

fee available. 

Conclusion 

History has demonstrated that NASA has had difficulty bringing a number of projects to 

completion including building a next-generation spacecraft. Since the 1980s, NASA has 

unsuccessfully attempted several development efforts estimated to have cost 

approximately $4.8 billion (National Aero-Space Plane, X-33, X-34, and Space Launch 

Initiative). In July 2006, GAO reported concern about NASA’s acquisition strategy for 

Orion.18 GAO stated that NASA’s strategy of awarding a long-term contract for design, 

development, production, and sustainment before developing a sound business case19

placed the Orion Project at risk of significant cost overruns, schedule delays, and 

performance shortfalls. 

In October 2007, GAO reported that NASA had been taking steps to build a business 

case for the Ares I Crew Launch Vehicle Project,20 to demonstrate that the project was 

 

16 The contracting officer’s technical representative supports the contracting officer through surveillance of 

the contractor’s performance and provides overall technical management of the contract. 

17 Award fee evaluation scoring follows standard adjectival ratings. The score of “excellent” is defined as 

“of exceptional merit; exemplary performance in a timely, efficient, and economical manner; very minor 

(if any) deficiencies with no adverse effect on overall performance.” 

18 GAO, “NASA: Long-Term Commitment to and Investment in Space Exploration Program Requires 

More Knowledge” (GAO-06-817R, July 17, 2006). 

19 GAO defines a sound business case as one in which requirements match available and reasonably 

expected resources before committing to a new product development effort. 

20 The Ares I Crew Launch Vehicle is being designed to launch the Orion into space. 
RESULTS

14 REPORT NO. IG-09-004 

achievable within the constraints of time, money, and other resources but noted gaps in 

knowledge about requirements, costs, schedule, technology, design, and production 

feasibility. GAO stated that while NASA still had 10 months under its own schedule to 

close gaps in the development of the Ares I system, the gaps GAO identified were 

significant and challenging given the complexity and interdependencies in the CxP. For 

example, continued instability in the design of Orion hampers the Ares I Project’s efforts 

to establish firm requirements.21 

In April 2008, GAO testified that while NASA was working toward PDR for Ares I and 

Orion, there are considerable unknowns as to whether NASA’s plans could be executed 

within schedule and cost goals. This was because NASA was still in the process of 

defining many performance requirements that could affect the mass, loads, and weight 

requirements. GAO stated that NASA was aiming to complete this process in 2008, but 

the Agency would be challenged to do so with the level of knowledge that still needed to 

be attained. GAO also indicated that NASA recognized the risks involved with its 

approach and had taken steps to mitigate some of these risks.22 

In response to the multiple GAO reports, NASA revised its acquisition policy in 2005 

and again in 2007. The policy revisions were a positive step in improving NASA’s 

ability to complete its programs and projects within cost, schedule, and performance 

parameters. However, implementation of those revisions created its own challenges 

because it fundamentally changed NASA’s approach to acquisition. For example, 

personnel within the Exploration Systems Mission Directorate had to balance the need to 

timely develop new space vehicles with the discipline necessary to follow and comply 

with the revised guidance. 

The purpose of the life-cycle approach to project acquisition is to develop the maturity of 

a project to a satisfactory degree before proceeding to the next life-cycle phase of 

development. This process is important to assure the project is ready for additional 

investment and to signal readiness for new and more advanced development. Reviews 

are incorporated in each life cycle to assure readiness to proceed to the next phase. 

This acquisition methodology is materially undermined when the review process is 

engaged when material challenges and questions with the current stage, or its design, are 

substantially unresolved. First, without resolution of questions that are fundamental to 

the life cycle’s phase of maturity, conducting a review is inherently untimely and 

wasteful of effort. Second, it denies the project of the life-cycle review necessary and 

appropriate when the project achieves the maturity level at which it should be reviewed. 

 

21 GAO, “Agency Has Taken Steps Toward Making Sound Investment Decisions for Ares I but Still Faces 

Challenging Knowledge Gaps” (GAO-08-51, October 31, 2007). 

22 GAO, Ares I and Orion Project Risks and Key Indicators to Measure Progress” (GAO-08-186T, April 3, 

2008). 
RESULTS

REPORT NO. IG-09-004 15 

NASA has a unique opportunity to improve its processes concurrently with the 

acquisition of new space vehicles. However, successful implementation of those 

processes will depend on management’s commitment to change and its ability to 

encourage compliance by all personnel in the acquisition process. 

The Orion Project Office believed its strategy of the parallel POD activity ultimately 

addressed the NPR 7120.5D life-cycle review requirements with the least expenditure of 

Agency resources. However, by not delaying the SDR in order to review the proper 

vehicle configuration, a significant portion of Orion requirements did not go through the 

SDR RID process. Although the SDR process and life-cycle reviews are designed to find 

and correct discrepancies, the process cannot be expected to work as designed if a 

significant portion of the requirements do not go through that process. In addition, the 

project risks the untimely identification of discrepancies in Phase B that should have 

been identified earlier in the Phase A reviews. NASA needs to ensure that the vehicle 

configuration and requirements are at the proper maturity level prior to starting and 

completing future life-cycle milestones. By not doing so, there is increased risk of costly 

rework and schedule slippage. 

Management Action 

The Orion Project Office’s effort in recognizing and correcting significant deficiencies in 

the vehicle configuration deserves special consideration. After the SDR, the Project 

Office correctly delayed the planned Phase B timeline because the SDR vehicle 

configuration was not ready and the Project had not made sufficient progress on meeting 

design changes adopted from the POD activity (see Figure 2). 

Figure 2. SDR and Changes to Planned Phase B Timeline 

Event

SDR

Award Fee Paid

POD Activity

Project Briefing to SRB

SRB Report Issued

PNAR Completion

PDR Board

7/11 8/30

9/27

6/8 11/15

12/6 

10/01 114 Day Slip 1/23

10/15 **** 197 Day Slip   4/29

5/16 ** ** 189 Day Slip **** 11/21

Apr Ma Jun Jul AugSep Oct NovDec Jan Feb Mar Apr May Jun Jul AugSep Oct NovDec Jan

2007 2008

With the extension, the SRB provided a report on January 23, 2008, instead of 

October 2007, the conclusion of the PNAR slipped from October 2007 to April 2008, and 

Orion’s PDR Board, originally scheduled for May 2008, has slipped twice from 

September 2008 to November 2008. The Project Office is considering an additional 
RESULTS

16 REPORT NO. IG-09-004 

delay for the PDR to 2009. In anticipation of the PDR slip to 2009, NASA extended the 

second performance milestone of the award fee plan, which requires that Phase B’s PDR 

Board be held by May 31, 2009. NASA needs to continue to adjust milestones and the 

contract as needed and avoid performing future life-cycle reviews prematurely. 

Recommendations, Management’s Response, and Evaluation of 

Management’s Response 

Recommendation 1. The Chief Engineer should revise NPR 7123.1A’s internal life-cycle 

reviews’ entrance criteria to require the project to update the life-cycle review’s technical 

products with known changes to the technical baseline resulting from engineering analysis 

cycle assessments. 

Management’s Response. The Chief Engineer nonconcurred, stating that while the 

intent of the recommendation is understood, the actual events show that the current 

policy is effective. He stated that the members of the Agency Program Management 

Council reviewed the results of the Orion SDR and the PNAR and determined that the 

SDR/PNAR milestone was successfully completed. In addition, Management approved 

the strategy of executing the SDR activities in parallel with the “Point of Departure” 

review to fully inform the Agency decision process and ensure a sound basis for 

beginning Phase B. The Chief Engineer also stated that there is no technical baseline of 

the design at this point in the life cycle. Trade studies, technology maturation, etc. are 

still occurring such that by the PDR, a preliminary design is in place that meets all 

system requirements. 

Evaluation of Management’s Response. Management’s comments are nonresponsive. 

We disagree that the actual events show that the current policy is effective. We believe 

that the policy allowed the Orion Project Office to prematurely hold the SDR on a 

vehicle configuration that did not meet known requirements and therefore, would not 

meet the review’s success criteria. 

Although we agree that there may not be a technical baseline for the actual design 

available at the SDR, NASA’s guidance explains that the Project is developing the 

technical baseline during the life cycle phase. As stated in our report and in 

NPR 7123.1A, life-cycle reviews “...occur based on maturity of the relevant technical 

baseline as opposed to calendar milestones....” If the SDR had disclosed the mass and 

weight discrepancies with the 606 vehicle configuration, then the 606 vehicle would 

probably have been the relevant technical baseline. However, the 606 vehicle 

configuration was not the relevant technical baseline because discrepancies were revealed 

by the DAC conducted prior to the SDR. The Orion Project Office knew that the 

606 vehicle was not the relevant technical baseline and therefore, the SDR should have 

been delayed. 

The Agency has acted in a prudent manner during Phase B and provides further support 

for the implementation of our recommendation. The pre-PDR DACs continue to show 
RESULTS

REPORT NO. IG-09-004 17 

the vehicle configuration’s technical baseline is not at the PDR maturity level. As a 

result, the Project has delayed its PDR on two different occasions, and is considering 

delaying the PDR a third time, to ensure that the Project reviews the relevant technical 

baseline. However, without the Agency taking the necessary step to revise NPR 7123.1A 

to require that technical products incorporate known changes resulting from analysis 

cycles, there is a risk that life-cycle reviews could again be performed on premature 

configurations. Therefore, we request that the Chief Engineer reconsider his position and 

provide additional comments with regard to requiring projects to update the life-cycle 

review’s technical products with known changes to the technical baseline resulting from 

engineering analysis cycle assessments. 

Recommendation 2.a. The Associate Administrator for Exploration Systems Mission 

Directorate should evaluate the 607 vehicle configuration for assurance that it was assessed 

at a minimum of the SDR level and, if not, determine the impact for any affected systems 

and, if material, perform an SDR on that part. 

Management’s Response. The Associate Administrator for Exploration Systems 

Mission Directorate nonconcurred, stating that no additional review is warranted. He 

stated that the configuration changes involved extensive participation and review by the 

entire community of technical subject matter experts and stakeholders from across the 

Agency. In addition, the community assessed the results against the SDR success criteria 

and identified actions to revise requirements, risks, etc. The Orion Project reviewed the 

607 vehicle configuration with the CxP, the Exploration Systems Mission Directorate, 

the Agency independent technical authorities, the Standing Review Board, and the 

Administrator. The Orion Project and the Standing Review Board presented the results at 

the Key Decision Point B meeting in April 2008. The Associate Administrator also 

stated, as noted in the OIG report, that NASA did not begin Phase B tasks until the 

607 configuration was defined. 

Evaluation of Management’s Response. Although the Associate Administrator 

nonconcurred, his comments are responsive to the intent of our recommendation. In his 

response, the Associate Administrator stated that the Agency is in strong agreement that 

adhering to the discipline of the life-cycle development approach is vital and that the 

purpose of any life-cycle review is to provide complete and objective information to 

Agency management. The Associate Administrator also stated that the 607 vehicle 

configuration that proceeded to Phase B was extensively reviewed by the Project, CxP, 

Exploration Systems Mission Directorate, independent technical authorities, and the 

Standing Review Board who all came to the conclusion to proceed to Phase B. 

Although the 607 vehicle configuration did not proceed through a formal SDR process, 

namely the configuration did not benefit from the rigors of the RID process, we agree 

that the configuration changes were eventually reviewed by technical subject matter 

experts and Agency stakeholders. Furthermore, the Agency has delayed the PDR 

because the known technical baseline changes are not at the proper maturity level and 
RESULTS

18 REPORT NO. IG-09-004 

those changes will obtain the full rigors of the PDR process. Therefore, we consider the 

recommendation resolved and closed. 

Recommendation 2.b. The Associate Administrator for Exploration Systems Mission 

Directorate should ensure that the contractor receives future award fees based on achieving 

the expected technological maturity level required at a particular milestone as opposed to 

convening and prematurely conducting a scheduled review at that milestone. 

Management’s Response. The Associate Administrator for Exploration Systems 

Mission Directorate concurred stating that NASA acted in accordance with the 

recommendation in the case of the first Orion award fee period. He stated that the award 

fee period associated with this milestone had no bearing on scheduling the SDR review 

or in the technical evaluation of the SDR products. The award fee approach for Orion 

was successful in that the evaluation took place at a logical point in development of 

products at the end of a DAC. Based on the quality of the delivered products, the 

performance of the contractor was assessed for all activities in that period against the 

established criteria. A weakness was included for the first award fee period relative to 

the contractor’s inability to adequately manage and track mass through DAC-1. This was 

a clear indicator the contractor was not meeting technical or management maturity 

expectations at that time. NASA has extended award fee period two in order to 

accommodate programmatic changes and to assure the period maintains the original 

intent, which is to evaluate expected performance and maturity to be demonstrated at the 

Pad Abort 1 test and the PDR milestones. 

Evaluation of Management’s Response. The Associate Administrator’s comments are 

generally responsive to our recommendation. Although we were able to verify that a 

weakness was noted for the contractor’s inability to adequately manage and track mass, 

we disagree that NASA acted in accordance with the recommendation during award fee 

period one in that the SDR did not review the appropriate vehicle configuration nor meet 

the SDR success criteria. However, the actions taken during award fee period two 

demonstrate that NASA has acted in accordance with the intent of our recommendation 

by adjusting the milestone period to coincide with the delayed PDR. Therefore, we 

consider the recommendation resolved and closed. 
APPENDIXES

REPORT NO. IG-09-004 19 

APPENDIX A 

Scope and Methodology 

We performed this audit from June 2007 through October 2008 in accordance with 

generally accepted government auditing standards. Those standards require that we plan 

and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable 

basis for our findings and conclusions based on our audit objectives. We believe that the 

evidence obtained provides a reasonable basis for our findings and conclusions based on 

our audit objectives. We performed audit fieldwork at NASA Headquarters and Johnson 

Space Center (Johnson). 

To assess the effectiveness of NASA’s Orion acquisition management, we reviewed and 

analyzed NASA’s efforts to transition Orion from the “Concept and Technology 

Development” phase (Phase A) to the “Preliminary Design and Technology Completion” 

phase (Phase B). We focused on Phase A’s life-cycle processes to include the SRR, 

SDR, and PNAR. We also reviewed internal controls as they related to the overall 

objective. We issued a report related to the audit objectives that concerned the 

establishment and operation of the SRB. 

During our pre-audit planning, the Project Office allowed us to observe the SRR process 

at Johnson. We observed the SRR, in March 2007, and the SDR in August 2007. We 

also attended the Project Office’s SDR/PNAR briefings to the SRB in December 2007 

and the SRB’s out-briefing to the Integrated Center Management Council in 

February 2008. 

We interviewed personnel from the Exploration Systems Mission Directorate, CxP, and 

the Project Office at NASA Headquarters and Johnson to discuss the overall process used 

in the acquisition of Orion. We also interviewed: 

• NASA Headquarters officials from the Office of the Chief Engineer responsible 

for the policy and guidance. We discussed their approach in updating NPRs 7120 

and 7123 and the entrance and success criteria used during the SRR and SDR 

milestone reviews. 

• Engineering, Safety and Mission Assurance, and Health and Medical, Technical 

Authorities that are matrixed to the Orion Project to discuss their processes of 

ensuring visibility and timely communications within the Project Office and to 

understand their role in the SRR and SDR milestone reviews. 

• Johnson contracting officials to discuss the development and selection of the 

prime contractor for Orion and the evaluation of the contractor award fees. 
APPENDIX A 

20 REPORT NO. IG-09-004 

• Orion Project Officials to discuss the flow down of the requirements from CxP to 

Lockheed for functional elements and subsystems. 

In addition to reviewing NPR 7120.5D, “NASA Space Flight Program and Project 

Management Requirements,” March 6, 2007, and NPR 7123.1A, “NASA Systems 

Engineering Processes and Requirements,” March 26, 2007, we also reviewed 

NASA Policy Directive 1000.0, “Strategic Management Governance Handbook,” 

August 30, 2005; NPR 7123.1, “NASA Systems Engineering Processes and 

Requirements,” March 13, 2006; and “Program and Project Formulation Document,” 

July 7, 2006. 

We also reviewed other documents concerning the management and development of 

Orion: 

• CxP 70003, “Constellation Program Plan,” September 28, 2006; 

• CxP 70006, “Constellation Program, Crew Exploration Vehicle SRR Process 

Plan,” Revision A, Change 001, May 17, 2007; 

• CxP 70006, Annex 2.2, “Constellation Program, Crew Exploration Vehicle SRR 

Process Plan,” Revision A, Change 001, May 17, 2007; 

• CxP 70016, “Constellation Program Requirements Management Plan,” Change 

1, May 17, 2007; 

• CxP 72008, “Crew Exploration Vehicle Project Plan,” Draft, Draft Maturity 

(80%), December 18, 2006; 

• CxP 72088, “Crew Exploration Vehicle, Systems Engineering Management 

Plan,” November 8, 2006 and Revision A, July 10, 2007; 

• CxP 72091, “Crew Exploration Vehicle Risk Management Plan,” September 18, 

2006; 

• CxP 72097, “Crew Exploration Vehicle Master Verification Plan,” March 1, 

2007; and 

• CxP 72107, “Crew Exploration Vehicle, System Definition Review, Process 

Plan,” June 22, 2007. 

Further, we reviewed internal and external reports and documents relating to Orion 

Project management: 

• Report of the Commission on the Implementation of U.S. Space Exploration 

Policy, “A Journey to Inspire, Innovate, and Discover” June 2004; 
APPENDIX A 

REPORT NO. IG-09-004 21 

• “NASA Exploration Systems Architecture Study,” November 2005; 

• “NASA Award Fee Process,” November 1, 2006; 

• ZV-07-005, “Preliminary Design Review Point of Departure Definition Activity,” 

June 8, 2007; 

• “SRR Review Item Discrepancy Database,” as of August 3, 2007; 

• “CEV Contract Award Fee Evaluation Plan,” August 27, 2007; and 

• “SRR Action Closeout Summary,” as of September 10, 2007. 

Computer-Processed Data. We did not perform a detailed assessment on the reliability 

and validity of the SRR Review Item Discrepancy (RID) database or the SRR Action 

Closeout Summary Report. We used the database to verify that all RIDS were 

dispositioned and to reconcile differences between the database and the RIDs the Project 

reported as reviewed. However, we did not verify that the database was all-inclusive. 

We used the Action Closeout Report to verify that all actions assigned to responsible 

parties had been dispositioned, but we did not verify that the report included every action. 

However, changes in the number of RIDs in the database or actions in the report would 

not change our conclusions or recommendations. 

Review of Internal Controls 

We reviewed the internal controls associated with project life-cycle management. The 

Project Office performed the required assessments for the SRR and the SDR to determine 

if Orion was ready to proceed to the next life-cycle phase. Additionally, the SRB 

assessed the life-cycle reviews. However, the Project Office proceeded through the SDR 

milestone knowing that the vehicle configuration was not at the appropriate maturity 

level. Until the Project can provide assurance that the appropriate vehicle design is the 

one reviewed, we cannot place full reliance on the life-cycle processes. We discussed the 

internal control deficiency within the body of this report. Implementing the 

recommendation in this report should improve the internal controls over life-cycle project 

management. 

Prior Coverage 

During the last 6 years, GAO and the NASA Office of Inspector General have issued 

five reports on NASA’s acquisition process. Unrestricted reports can be accessed over 

the Internet at http://www.gao.gov (GAO) and 

http://www.hq.nasa.gov/office/oig/hq/audits/reports/FY07/index.html (NASA). 
APPENDIX A 

22 REPORT NO. IG-09-004 

Government Accountability Office

“Ares I and Orion Project Risks and Key Indicators to Measure Progress” 

(GAO-08-186T, April 3, 2008) 

The report discussed challenges NASA is facing in developing the systems to 

achieve its goals for the President’s Vision for Space Exploration. The report 

focused on the Ares I Crew Launch Vehicle and the Orion because NASA is 

currently working toward PDR. The report states that there are considerable 

unknowns as to whether NASA’s plans for these vehicles can be executed within 

cost and schedule. NASA is still in the process of defining many performance 

requirements and such uncertainties could affect the mass, loads, and weight 

requirements for the vehicles. The upcoming PDR represents the most critical 

juncture where hard decisions can be made as to whether the programs should 

proceed forward. 

“Agency Has Taken Steps Toward Making Sound Investment Decisions for Ares I but 

Still Faces Challenging Knowledge Gaps” (GAO-08-051, October 31, 2007) 

The report, which discusses NASA’s efforts to implement the President’s plan to 

return humans to the moon and prepare for eventual human space flight to Mars, is 

the development of the Ares I Crew Launch Vehicle. NASA plans to conduct the 

first human space flight in 2015. The report also states that the agency is seeking to 

speed development efforts in order to reduce the gap in our nation’s ability to provide 

human access to space caused by the Space Shuttle’s retirement in 2010. There are 

gaps in knowledge about requirements, costs, schedule, technology, design, and 

production feasibility. These gaps are significant and challenging given the 

complexity and interdependencies in the program. 

“NASA: Long-Term Commitment to and Investment in Space Exploration Program 

Requires More Knowledge” (GAO-06-817R, July 17, 2006) 

The report, which discusses NASA’s status on implementing the President’s Vision 

for Space Exploration, states that although NASA is continuing to refine its 

exploration architecture cost estimates, the Agency cannot provide a firm estimate of 

what it will take to implement the architecture. In addition, the report states that 

NASA will be challenged to implement the exploration architecture with its projected 

budget. The report also discusses NASA’s acquisition strategy for the Crew 

Exploration Vehicle and states the strategy places the Project at risk for cost 

overruns, schedule delays, and performance shortfalls. 

“NASA-Implementing a Knowledge-Based Acquisition Framework Could Lead to Better 

Investment Decisions and Project Outcomes” (GAO-06-218, December 21, 2005) 

The report discusses NASA’s revised policy for developing flight systems and ground 

support projects incorporates some of the best practices used by successful 
APPENDIX A 

REPORT NO. IG-09-004 23 

developers. By not establishing a minimum threshold for technology maturity, 

NASA increases the risk that design changes will be required later in development, 

when such changes are typically more costly to make. In addition, although NASA’s 

policy does require project managers to establish a continuum of technical and 

management reviews, it does not specify what these reviews should be, nor does it 

require major decision reviews at other key points in a product’s development. 

Acquiring knowledge at key junctures will become increasingly important as NASA 

proceeds to implement elements of the Vision. Without a major decision review at 

key milestones to ensure that the appropriate level of knowledge has been achieved to 

proceed to the next phase, the risk of cost and schedule overruns, as well as 

performance shortfalls, increases. 

National Aeronautics and Space Administration Office of Inspector General

“Acquisition of the Space Station Propulsion Module” (IG-01-027, May 21, 2001) 

The report, which discusses the Acquisition of the Space Station Propulsion Module, 

states the Propulsion Module was not cost effective. In addition, NASA implemented the 

United States Propulsion Module design before properly accomplishing acquisition 

planning and preparing project documents. Specifically, NASA did not validate 

requirements from the SRR before beginning a PDR of the propulsion module; Agency 

pursued implementation of the propulsion system without an approved project plan or 

risk management plan; and NASA selected The Boeing Company as the sole-source 

contractor without properly documenting the justification for the noncompetitive 

selection. 
APPENDIX B 

24 REPORT NO. IG-09-004 

LIFE CYCLE OF A NASA SPACE 

FLIGHT PROJECT

The following figure shows the complete life cycle of a NASA space flight project. Our 

audit of the Orion Project focused primarily on Phases A and B of the life cycle. 

Source: NPR 7120.5D (figure 2-4) 

Acronyms

FAD Formulation Authorization Document 

KDP Key Decision Point (for the Orion, KDP B was held on April 29, 2008) 

NAR Non-Advocate Review 

PDR Preliminary Design Review 

PNAR Preliminary Non-Advocate Review (for Orion, completion of the 

 PNAR was April 29, 2008) 

SDR System Definition Review 

SRR System Requirements Review
APPENDIX C 

REPORT NO. IG-09-004 25 

SYSTEM REQUIREMENTS REVIEW 

ENTRANCE/SUCCESS 

CRITERIA

Source: NPR 7123.1A (Table G-4) 

The SRR examines the functional and performance requirements defined for the system 

and the preliminary program or project plan and ensures that the requirements and the 

selected concept will satisfy the mission. 

System Requirements Review 

Entrance Criteria Success Criteria 

1. Successful completion of the MCR and responses made to all 

MCR Requests for Actions (RFAs) and Review Item 

Discrepancies (RIDs). 

2. A preliminary SRR agenda, success criteria, and charge to the 

board have been agreed to by the technical team, project 

manager, and review chair prior to the SRR. 

3. The following technical products for hardware and software 

system elements are available to the cognizant participants prior 

to the review: 

a. system requirements document; 

b. system software functionality description; 

c. updated concept of operations; 

d. updated mission requirements, if applicable; 

e. baselined SEMP; 

f. risk management plan; 

g. preliminary system requirements allocation to the next lower 

 level system; 

h. updated cost estimate; 

i. Technology Development Maturity Assessment Plan; 

j. updated risk assessment and mitigations (including PRA as 

 applicable). 

k. logistics documentation (e.g., preliminary maintenance plan); 

l. preliminary human rating plan, if applicable; 

m. Software Development Plan (SDP); 

n. system safety and mission assurance plan; 

o. configuration management plan; 

p. initial document tree; 

q. verification and validation approach; 

r. preliminary system safety analysis; and 

s. other specialty disciplines, as required. 

1. The project utilizes a sound process 

for the allocation and control of 

requirements throughout all levels, 

and a plan has been defined to 

complete the definition activity 

within schedule constraints. 

2. Requirements definition is complete 

with respect to top-level mission 

and science requirements, and 

interfaces with external entities and 

between major internal elements 

have been defined. 

3. Requirements allocation and flow 

down of key driving requirements 

have been defined down to 

subsystems. 

4. Preliminary approaches have been 

determined for how requirements 

will be verified and validated down 

to the subsystem level. 

5. Major risks have been identified 

and technically assessed, and viable 

mitigation strategies have been 

defined. 
APPENDIX D 

26 REPORT NO. IG-09-004 

SYSTEM DEFINITION REVIEW 

ENTRANCE/SUCCESS 

CRITERIA

System Definition Review 

Entrance Criteria Success Criteria 

1. Successful completion of the SRR and responses made to 

all SRR RFAs and RIDs. 

2. A preliminary SDR agenda, success criteria, and charge to 

the board have been agreed to by the technical team, project 

manager, and review chair prior to the SDR. 

3. SDR technical products listed below for both hardware and 

software system elements have been made available to the 

cognizant participants prior to the review: 

a. system architecture; 

b. preferred system solution definition including major 

tradeoffs and options; 

c. updated baselined documentation, as required; 

d. preliminary functional baseline (with supporting tradeoff analyses and data); 

e. preliminary system software functional requirements; 

f. SEMP changes, if any; 

g. updated risk management plan; 

h. updated risk assessment and mitigations (including 

PRA, as applicable); 

i. updated technology development, maturity, and 

assessment plan; 

j. updated cost and schedule data; 

k. updated logistics documentation; 

l. based on system complexity, updated human rating 

plan; 

m. software test plan; 

n. software requirements document(s); 

o. interface requirements documents (including software); 

p. technical resource utilization estimates and margins; 

q. updated safety and mission assurance (S&MA) plan; 

and 

r. updated preliminary safety analysis. 

1. Systems requirements, including mission 

success criteria and any sponsor-imposed 

constraints, are defined and form the 

basis for the proposed conceptual design. 

2. All technical requirements are allocated 

and the flow down to subsystems is 

adequate. The requirements, design 

approaches, and conceptual design will 

fulfill the mission needs consistent with 

the available resources (cost, schedule, 

mass, and power). 

3. The requirements process is sound and 

can reasonably be expected to continue to 

identify and flow detailed requirements in 

a manner timely for development. 

4. The technical approach is credible and 

responsive to the identified requirements. 

5. Technical plans have been updated, as 

necessary. 

6. The tradeoffs are completed, and those 

planned for Phase B adequately address 

the option space. 

7. Significant development, mission, and 

safety risks are identified and technically 

assessed, and a process and resources 

exist to manage the risks. 

8. Adequate planning exists for the 

development of any enabling new 

technology. 

9. The operations concept is consistent with 

proposed design concept(s) and is in 

alignment with the mission requirements. 

Source: NPR 7123.1A (Table G-6)

The SDR examines the proposed system architecture and design and the flow down to all 

functional elements of the system. 
APPENDIX E 

REPORT NO. IG-09-004 27 

ORION’S ANALYSIS CYCLES 

PERFORMED BETWEEN 

LIFE-CYCLE REVIEWS

Source: CxP 72088, “Crew Exploration Systems Engineering Management Plan,” Figure 5.4-1 

Orion’s integrated analysis reviews comprise three major analysis cycles that 

progressively support definition, design, verification, and readiness, through the 

operational readiness review. 

Analysis Cycle Acronyms/Definitions

RAC Requirements Analysis Cycle - Supports the analysis and trades performed early 

in the life cycle to develop and validate technical requirements. 

DAC Design Analysis Cycle - Directly supports achievement of the goals of the next 

design review including the SDR, PDR, and CDR (Critical Design Review). 

VAC Verification Analysis Cycle - Supports the achievement of overall readiness in 

time for the ORR (Operational Readiness Review). 
APPENDIX F 

28 REPORT NO. IG-09-004 

PRELIMINARY DESIGN REVIEW 

DEFINITION ACTIVITY 

MEMORANDUM
APPENDIX F 

REPORT NO. IG-09-004 29 
APPENDIX F 

30 REPORT NO. IG-09-004 
APPENDIX F 

REPORT NO. IG-09-004 31 
APPENDIX F 

32 REPORT NO. IG-09-004 
APPENDIX G 

REPORT NO. IG-09-004 33 

MANAGEMENT COMMENTS
APPENDIX G 

34 REPORT NO. IG-09-004 
APPENDIX G 

REPORT NO. IG-09-004 35 
APPENDIX G 

36 REPORT NO. IG-09-004 
APPENDIX H 

REPORT NO. IG-09-004 37 

REPORT DISTRIBUTION

National Aeronautics and Space Administration 

Administrator 

Deputy Administrator 

Chief of Staff 

Chief Engineer 

Associate Administrator for Explorations Systems Mission Directorate 

Constellation Program Manager 

Orion Project Manager 

Non-NASA Organizations and Individuals 

Office of Management and Budget 

Deputy Associate Director, Energy and Science Division 

Branch Chief, Science and Space Programs Branch 

Government Accountability Office 

Director, Defense, State, and NASA Financial Management, Office of Financial 

Management and Assurance 

Director, NASA Issues, Office of Acquisition and Sourcing Management 

Congressional Committees and Subcommittees, Chairman and 

Ranking Member 

Senate Committee on Appropriations 

Subcommittee on Commerce, Justice, Science, and Related Agencies 

Senate Committee on Commerce, Science, and Transportation 

Subcommittee on Space, Aeronautics, and Related Sciences 

Senate Committee on Homeland Security and Governmental Affairs 

House Committee on Appropriations 

Subcommittee on Commerce, Justice, Science, and Related Agencies 

House Committee on Oversight and Government Reform 

Subcommittee on Government Management, Organization, and Procurement 

House Committee on Science and Technology 

Subcommittee on Investigations and Oversight 

Subcommittee on Space and Aeronautics 

REPORT NO. IG-09-004 39 

Major Contributors to the Report: 

Carol Gorman, Director, Space Operations and Exploration Directorate 

Ray Tolomeo, Director, Mission Programs and Projects 

Loretta Atkinson, Project Manager 

Jim Griggs, Team Lead 

Douglas Orton, Team Lead 

Barbara Moody, Auditor 

Eugene Bauer, Auditor 

Frank Martin, Aerospace Technologist 

Cindy Stein, Aerospace Technologist 

Janet Overton, Report Process Manager 
OCTOBER 31, 2008 

REPORT No. IG-09-004 

OFFICE OF AUDITS

OFFICE OF INSPECTOR GENERAL

ADDITIONAL COPIES

Visit http://oig.nasa.gov/audits/reports/FY09 to obtain additional copies of this report, or contact the 

Assistant Inspector General for Auditing at 202-358-1232. 

COMMENTS ON THIS REPORT

In order to help us improve the quality of our products, if you wish to comment on the quality or 

usefulness of this report, please send your comments to Ms. Jacqueline White, Director of Special 

Projects and Quality Assurance, at Jacqueline.White@nasa.gov or call 202-358-0203. 

SUGGESTIONS FOR FUTURE AUDITS

To suggest ideas for or to request future audits, contact the Assistant Inspector General for Auditing. 

Ideas and requests can also be mailed to: 

Assistant Inspector General for Auditing 

NASA Headquarters 

Washington, DC 20546-0001 

NASA HOTLINE

To report fraud, waste, abuse, or mismanagement, contact the NASA OIG Hotline at 800-424-9183 or 

800-535-8134 (TDD). You may also write to the NASA Inspector General, P.O. Box 23089, L’Enfant 

Plaza Station, Washington, DC 20026, or use http://oig.nasa.gov/hotline.html#form. The identity of 

each writer and caller can be kept confidential, upon request, to the extent permitted by law.