a

GAO

United States General Accounting Office

Report to the Committee on Commerce, 

Science, and Transportation, U.S. Senate, 

and the Committee on Science, House of 

Representatives

April 2003 BUSINESS 

MODERNIZATION

Improvements Needed 

in Management of 

NASA’s Integrated 

Financial Management 

Program

GAO-03-507
The core financial module, if implemented as planned, may provide some 

improvement to NASA’s accounting system environment. However, NASA is 

not following key best practices for acquiring and implementing IFMP. In 

acquiring IFMP components, NASA is facing risks in understanding 

dependencies among commercial components. NASA has not analyzed the 

interdependencies among selected and proposed IFMP components, and it 

does not have a methodology for doing so. For programs like IFMP, which 

involve building a system from commercial components, it is essential to 

understand the characteristics and credentials of each component to select 

ones that are compatible and can be integrated without having to build and 

maintain expensive interfaces. By acquiring IFMP components without first 

understanding system component relationships, NASA has increased its 

risks of implementing a system that will not optimize mission performance 

and will cost more and take longer to implement than necessary. 

In implementing the core financial module, NASA is facing risks in two 

additional areas: 

• User needs. NASA did not consider the information needs of key 

system users and deferred addressing the requirements of program 

managers, cost estimators, and the Congress. Although this module 

should eliminate NASA’s separate, incompatible accounting systems, 

little has been done to reengineer acquisition management processes. 

Program managers and cost estimators indicated that they will continue 

to rely on other means to capture the data needed to manage programs 

such as the International Space Station. 

• Requirements management. NASA is relying on a requirements 

management process that does not require documentation of detailed 

system requirements prior to system implementation and testing. Over 

80 percent of the requirements GAO reviewed lacked specificity, and 

several could not be traced among various documents. These defects 

also significantly impaired the testing phase of the system 

implementation effort. Further, NASA has not implemented metrics to 

help gauge the effectiveness of its requirements management process. 

NASA’s approach will likely result in increasing amounts of time spent 

on costly rework and reduced progress. 

Unless these issues are successfully addressed, NASA is at increased risk of 

having IFMP become its third unsuccessful attempt to transform its financial 

management and business operations. 

BUSINESS MODERNIZATION 

Improvements Needed in Management of 

NASA's Integrated Financial Management 

Program 

www.gao.gov/cgi-bin/getrpt?GAO-03-507.

To view the full report, including the scope 

and methodology, click on the link above. 

For more information, contact Gregory D. 

Kutz at (202) 512-9095 or kutzg@gao.gov. 

Highlights of GAO-03-507, a report to the 

Committee on Commerce, Science, and 

Transportation, U.S. Senate, and the 

Committee on Science, House of 

Representatives 

April 2003

The National Aeronautics and 

Space Administration’s (NASA) 

nonintegrated financial 

management systems have 

weakened its ability to oversee its 

contractors, and its contract 

management has been on GAO’s 

high-risk list since 1990. In April 

2000, NASA began its Integrated 

Financial Management Program 

(IFMP), its third attempt in recent 

years at modernizing financial 

management processes and 

systems. GAO was asked to review 

whether NASA was following key 

best practices in acquiring IFMP 

components and implementing one 

of the first components—the core 

financial module. 

GAO is recommending that NASA 

develop and implement (1) a shortterm plan to identify and mitigate 

the risks currently associated with 

relying on already deployed IFMP 

commercial components and (2) a 

longer term strategy for acquiring 

additional IFMP components that 

includes implementing a 

methodology for commercial 

system component dependency 

analysis. NASA agreed with GAO’s 

recommendation related to a shortterm plan but disagreed with many 

of the findings related to user 

needs and requirements 

management. NASA also agreed 

with the importance of having an 

approach for acquiring additional 

IFMP components, but stated that 

it already has an effective strategy 

in place. GAO reaffirms its 

recommendations. 
Page i GAO-03-507 NASA's IFMP

Contents

Letter 1

Results in Brief 3

Background 6

NASA’s Acquisition Management Strategy Does Not Include 

Analyzing Component Interdependencies 9

Core Financial Module Does Not Fully Address Key User 

Information Requirements 11

NASA’s Requirements Management Process for the Core Financial 

Module Is Ineffective 21

Conclusions 36

Recommendations for Executive Action 37

Agency Comments and Our Evaluation 38

Appendixes

Appendix I: Objectives, Scope, and Methodology 47

Appendix II: Comments from the National Aeronautics and Space 

Administration 51

GAO Comments 70

Appendix III: GAO Contacts and Staff Acknowledgments 73

GAO Contacts 73

Acknowledgments 73

Tables Table 1: Five Contractors and Their Responsibilities 8

Figures Figure 1: Space Shuttle Flight Operations Contract 17

Figure 2: Example of Level of Detail Reported versus That 

Required by Cost Estimators 19

Figure 3: System Requirements for the “Manage Accounts Payable” 

Process 27

Figure 4: Requirements for “Validate Payment” Subprocess 28

Figure 5: Relationship between Requirements Development and 

Testing 30
Contents

Page ii GAO-03-507 NASA's IFMP

Abbreviations

CSC Computer Services Corporation

ERP Enterprise Resource Planning

FFMIA Federal Financial Management Improvement Act

IBM International Business Machines

IEEE Institute of Electrical and Electronics Engineers 

IFMP Integrated Financial Management Program

IMCE International Space Station Management and Cost Evaluation 

ISS International Space Station

NASA National Aeronautics and Space Administration

OMB Office of Management and Budget 

SEI Software Engineering Institute

This is a work of the U.S. Government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. It may contain copyrighted graphics, images or other materials. 

Permission from the copyright holder may be necessary should you wish to reproduce 

copyrighted materials separately from GAO’s product.
Page 1 GAO-03-507 NASA's IFMP

United States General Accounting Office

Washington, D.C. 20548

A

April 30, 2003 Letr

The Honorable John McCain

Chairman

The Honorable Ernest Hollings

Ranking Minority Member

Committee on Commerce, Science, 

and Transportation

United States Senate

The Honorable Sherwood L. Boehlert

Chairman

The Honorable Ralph Hall

Ranking Minority Member

Committee on Science

House of Representatives

Much of the National Aeronautics and Space Administration’s (NASA) 

success depends on the work of its contractors—on which it spends 

$12.7 billion, or 90 percent of its annual budget. For many years, NASA has 

not effectively overseen its contracts, principally because it has lacked 

accurate and reliable information on contract spending and performance 

and it has placed insufficient emphasis on end results, product 

performance, and cost control. Since 1990 we have identified NASA’s 

contract management function as an area at high risk.1

 NASA’s ability to 

collect, maintain, and analyze cost and performance data has been 

weakened by nonintegrated, incompatible financial management systems 

and processes, and uneven and nonstandard cost-reporting capabilities. 

NASA made two efforts in the past to improve its financial management 

processes and develop a supporting system intended to produce the kind of 

accurate and reliable information needed to manage its contracts 

effectively, but both of these efforts were eventually abandoned after a 

total of 12 years and a reported $180 million in spending.

1

At that time, we began a special effort to review and report on the federal program areas 

that our work had identified as high risk because of vulnerabilities to waste, fraud, abuse, 

and mismanagement. We first issued our High-Risk Series in December 1992 and have 

continued to include NASA’s contract management as an area of high risk since. See U.S. 

General Accounting Office, High-Risk Series: NASA Contract Management, GAO/HR-93-11

(Washington, D.C.: December 1992) and High-Risk Series: NASA Contract Management, 

GAO-03-119 (Washington, D.C.: January 2003).
Page 2 GAO-03-507 NASA's IFMP

In April 2000, NASA began its third attempt at modernizing its financial 

management processes and systems. NASA has estimated the life cycle 

cost of this effort through 2008 to be $861 million.2

 This effort, known as 

the Integrated Financial Management Program (IFMP), is expected to 

produce an integrated, NASA-wide financial management system through 

the acquisition and incremental implementation of commercial software 

packages and related hardware and software components.3 Through the 

proven business processes and centralized data management capabilities 

embedded in these commercial components, NASA intends to reengineer 

its management operations to “do business the way business does 

business.” The core financial management module, which NASA considers 

to be the backbone of IFMP, is currently operating at NASA headquarters 

and 6 of NASA’s 10 centers4

 and is expected to be fully operational in June 

2003. According to NASA’s business case analysis for the system, the core 

financial module will provide NASA’s financial and program managers with 

timely, consistent, and reliable cost and performance information for 

management decisions. 

Given the importance of IFMP to NASA’s mission performance, you asked 

us to review the program. The purpose of this report is to alert you now to 

concerns we have based on our work to date and to provide NASA 

management with constructive recommendations for improvement that it 

can initiate as soon as possible. We are continuing our work and plan to 

fully respond to your request later this year. 

Our work to date has focused on whether NASA has management 

processes in place for effective system acquisition and implementation. 

This report addresses three issues concerning the acquisition of IFMP 

components and the implementation of one of the first components—the 

core financial module. Specifically, we determined whether NASA (1) was 

2

For this estimate, NASA has defined life cycle costs to include implementation efforts 

through fiscal year 2008 and major upgrades, plus operation and support costs for each 

system module for the first 2 years after the module goes live. 

3

The system is to consist of nine modules: core financial management, resume management, 

travel management, position description management, human resource management, 

payroll, budget formulation, contract administration, and asset management. 

4

NASA is comprised of its headquarters offices, nine Centers located throughout the 

country, and the Jet Propulsion Laboratory. The Jet Propulsion Laboratory is operated by 

the California Institute of Technology, but for purposes of this report, we treat the Jet 

Propulsion Laboratory as a center.
Page 3 GAO-03-507 NASA's IFMP

effectively evaluating the relationship among commercial systems 

component options before acquiring them, (2) had adequately considered 

the information needs of key users in implementing the core financial 

module, and (3) had established and implemented an effective 

requirements management process to support implementation of the core 

financial module. 

We performed our work from April 2002 through February 2003 in 

accordance with generally accepted government auditing standards. We 

had intended to include our assessment of a key element of NASA’s 

acquisition strategy—whether NASA was acquiring IFMP components in 

the context of an agencywide blueprint, commonly called an enterprise 

architecture—in this report. However, because NASA did not provide the 

data needed to complete our assessment until after the conclusion of our 

fieldwork, we plan to address NASA’s enterprise architecture in a future 

product. Details on our objectives, scope, and methodology are in 

appendix I. 

Results in Brief If implemented as planned, IFMP may provide some improvement to 

NASA’s current accounting system environment because it should eliminate 

the separate, incompatible systems that have previously been used at each 

of NASA’s 10 centers and should result in standardized accounting data. 

However, NASA is not following key best practices for acquiring and 

implementing IFMP. Specifically, NASA has not established an analytical 

capability to guide and constrain its acquisition of IFMP commercial 

components. Further, in implementing the core financial module 

component, NASA has deferred addressing the needs of key system users 

and has not properly developed detailed system requirements. 

Consequently, the agency is at risk of making a substantial investment in a 

system that will fall far short of its stated goal of providing meaningful and 

reliable information to support effective program management and 

congressional oversight. 

NASA has not analyzed the interdependencies among selected and 

proposed IFMP components, and it does not have a methodology for doing 

so. For programs like IFMP, which involve building a system from 

commercial components, it is essential to understand the characteristics 

and credentials of each component in order to select ones that are 

compatible and can be integrated without having to build and maintain 

expensive interfaces. The alternative to such a structured and disciplined 

approach to building a commercial component-based system is trial and 
Page 4 GAO-03-507 NASA's IFMP

error, which is fraught with risk. Although NASA has already acquired the 

core financial module and three other IFMP commercial components, the 

agency has not performed the analysis necessary to understand the logical 

and physical relationships among the component parts it has acquired. By 

acquiring these IFMP components without first understanding system 

component relationships, NASA has increased its risks of implementing a 

system that will not optimize mission performance, and will cost more and 

take longer to implement than necessary. 

For the core financial module, NASA did not consider the information 

needs of key system users and deferred addressing the requirements of 

program managers, cost estimators, and the Congress. Since 1990, we have 

identified NASA’s contract management function as an area at high risk, in 

part because of the lack of effective systems and processes for managing 

and overseeing its procurement dollars, producing credible cost estimates, 

and providing the Congress with appropriate visibility over its large, 

complex programs. However, despite these previous problems, program 

managers, cost estimators, and congressional staffs were not included in 

defining system requirements for NASA’s core financial module. Instead, 

NASA’s financial managers and accountants have primary responsibility for 

this process. In addition, little has been done to reengineer acquisition 

management processes, particularly with respect to the consistency and 

detail of budget and actual cost data provided by contractors. Although 

capable of accepting the data needed to satisfy the information needs of 

these key users, NASA’s new core financial module is not being 

implemented to accommodate this information. According to IFMP 

program officials, they chose to defer certain system capabilities and 

related user requirements in order to expedite implementation of the core 

financial module. As a result, program managers and cost estimators told 

us that they will not rely on the core financial module and instead will 

continue to rely on other systems or use other labor-intensive means to 

capture the data they need to manage programs such as the International 

Space Station (ISS). 

Further, NASA did not have an effective requirements management process 

to support the implementation of the core financial module. Specifically, 

NASA was relying on a systems requirements management process that did 

not require documentation of detailed system requirements prior to system 

implementation and testing. Although industry best practices and NASA’s 

own system planning documents indicate that detailed requirements are 

needed to serve as the basis for effective system testing, NASA’s approach 

instead relied on certain subject matter experts’ knowledge of the detailed 
Page 5 GAO-03-507 NASA's IFMP

requirements necessary to evaluate the functionality actually provided. As 

a result of this approach, our review of the core financial module 

requirements found that, for many of them, (1) the functionality to be 

delivered was not adequately described or stated in a manner that allowed 

for quantitative evaluation and (2) the traceability between the various 

requirement documents was not maintained. Accordingly, the potential for 

these requirements defects to result in costly rework is significant and 

increases the risk that the project will not meet its cost, schedule, and 

performance objectives. Because of the direct relationship between 

requirements and testing, the lack of complete and unambiguous 

requirements also significantly impairs the testing phase of the system 

implementation effort. For example, the core financial module could not 

process vendor invoices that contained over 200 line items—a common 

occurrence on NASA’s large contracts—because NASA did not design an 

appropriate test case. If NASA had documented its requirements, it would 

have recognized that a properly designed test case had not been developed 

to cover this necessary functionality. Furthermore, NASA has not 

effectively implemented the types of metrics that can help the organization 

understand the effectiveness of its requirements management process, 

such as identifying and quantifying any weaknesses and then developing 

the corrective actions needed.

We are making recommendations that address the need for NASA to 

(1) develop and implement a short-term plan to identify and mitigate the 

risks currently associated with relying on already deployed IFMP 

commercial components and to expeditiously stabilize these components’ 

operation capability and performance, (2) as part of the short-term plan, 

develop and properly document requirements, reengineer acquisition 

management processes, and fully engage stakeholders—including program 

managers, cost estimators, and the Congress—in the development of user 

requirements, and (3) develop a longer term strategy for acquiring 

additional IFMP components that includes implementing a methodology 

for commercial system component dependency analysis. 

In written comments, which are reprinted in appendix II, NASA concurred 

with the need for a short-term plan but disagreed with most of our findings 

related to user needs and requirements and testing. We remain convinced 

that, as we have stated, NASA needs to (1) reengineer its acquisition 

management processes to ensure that program and financial managers as 

well as the Congress have needed budget and actual cost and schedule data 

and (2) document detailed requirements to reduce, to acceptable levels, the 

risks in implementing the selected processes. NASA also agreed with the 
Page 6 GAO-03-507 NASA's IFMP

importance of having an approach for acquiring additional IFMP 

components, but stated that it already has an effective strategy in place. We 

did not find convincing evidence to support NASA’s contention that it is 

using methodologically based dependency analysis—a best practice for 

implementing commercial component-based systems—in acquiring IFMP. 

Background NASA has a long and well-documented history of problems overseeing its 

procurement dollars, producing credible cost estimates, and providing the 

Congress with appropriate visibility over its large, complex programs. We 

first identified NASA’s contract management as an area at high risk in 1990 

because NASA lacked effective systems and processes for overseeing 

contractor activities. Over the past decade, other GAO, Inspector General, 

and task force reports have shown that NASA’s cost estimates lack 

credibility, in part because NASA does not collect the historical cost data 

needed to accurately project future costs or assess the validity of past 

estimates. Finally, because NASA had not provided the Congress with 

adequate visibility over the ISS program, the Congress had little advance 

warning when NASA reported that the estimated cost to complete the ISS 

had grown by about $5 billion in 1 year.

Since we first identified NASA’s contract management as an area of high 

risk, we have reported that one of NASA’s most formidable barriers to 

sound contract management is the lack of a modern, integrated financial 

management system. NASA’s ability to collect, maintain, and analyze cost 

and performance data has been weakened by nonintegrated, incompatible 

accounting systems and processes, and uneven and nonstandard costreporting capabilities. The weaknesses in NASA’s financial management 

systems also caused its independent auditor, PricewaterhouseCoopers, to 

conclude for fiscal years 2001 and 2002 that NASA’s financial management 

systems do not substantially comply with the requirements of the Federal 

Financial Management Improvement Act (FFMIA). FFMIA builds on 

previous financial management reform legislation by emphasizing the need 

for agencies to have systems that can generate timely, accurate, and useful 

information with which to make informed decisions and to ensure 

accountability on an ongoing basis. 

While NASA’s efforts to design and implement a new financial management 

system certainly move NASA forward in this area, technology alone will not 

solve NASA’s problems. Our reviews, as well as NASA’s, show that finance 

is not viewed as an integral part of NASA’s program management decision 

process. Moreover, an independent task force created by NASA to review 
Page 7 GAO-03-507 NASA's IFMP

and assess ISS costs, budget, and management reached a similar 

conclusion. In its November 1, 2001, report, the International Space Station 

Management and Cost Evaluation (IMCE) Task Force found that the ISS 

program office does not collect the historical cost data needed to project 

future costs accurately and thus perform major program-level financial 

forecasting and strategic planning. The task force also reported that NASA’s 

ability to forecast and plan is weakened by diverse and often incompatible 

center-level accounting systems and uneven and non-standard cost 

reporting capabilities. The IMCE Task Force also concluded that the 

current weaknesses in financial reporting are a symptom, not a cause, of 

the problem and that enhanced reporting capabilities, by way of a new 

integrated financial management system, will not thoroughly solve the 

problem. The root of the problem, according to the task force, is that 

finance is not viewed as intrinsic to NASA’s program management decision 

process.

NASA’s IFMP includes nine module projects supporting a range of financial, 

administrative, and functional areas. According to NASA officials, of the 

nine module projects, two are in operation, three are currently in 

implementation, and four are future modules. The two projects in 

operation are resume management and position description management; 

the three projects in implementation are travel management, core financial, 

and budget formulation; and the four future module projects are human 

resources, payroll, asset management, and contract administration. 

The core financial module project, which utilizes the SAP R/3 system, is the 

backbone of IFMP and will become NASA’s standard, integrated accounting 

system used agencywide. The other IFMP module projects will be 

integrated/interfaced with the core financial module, where applicable. The 

scope of the core financial module, when fully implemented, includes: 

standard general ledger, budget execution, purchasing, accounts 

receivable, accounts payable, and cost management. NASA plans to 

implement the core financial module at all 10 NASA centers by June 2003. 

The pilot for the core financial module—conducted at Marshall Space 

Flight Center—was implemented in October 2002. NASA is rolling out or 

deploying the core financial module at the other nine NASA centers and 

headquarters in three waves. The first wave, which consisted of Glenn 

Research Center, rolled out in October 2002. The second wave, which 

consisted of NASA headquarters, Johnson Space Center, Kennedy Space 

Center, and the Jet Propulsion Laboratory rolled out in February 2003. 

Ames Research Center, considered a second wave center, rolled out in 

April 2003. Finally, NASA plans to roll out the third wave, which consists of 
Page 8 GAO-03-507 NASA's IFMP

Dryden Flight Research Center, Goddard Space Flight Center, Langley 

Research Center, and Stennis Space Center, in June 2003.

IFMP Acquisition 

Management Structure

NASA is contracting with multiple companies to assist in the acquisition 

management, integration, and implementation of its IFMP “system of 

system components.” As shown in table 1, five contractors are assisting in 

the integration and implementation of the core financial module. However, 

none of these five contractors is responsible and accountable for 

successfully implementing the entire IFMP system. Instead, NASA has 

structured its IFMP acquisition so that NASA is the system integrator, 

meaning that NASA is responsible for integrating multiple commercial 

components and ensuring that they collectively perform in a manner that 

meets the defined requirements. 

Table 1: Five Contractors and Their Responsibilities

Source: NASA. 

a

NASA plans to solicit additional contracts for implementation of other selected and acquired modules. 

b

NASA has acquired SAP’s enterprise resource planning package and has thus far planned to 

implement the core financial and budget formulation modules. NASA has also acquired three other 

commercial software products—Travel Manager, Resumix, and Position Description Management.

Entity Responsibility/function

Accenture Implement the core financial module in accordance with agency 

requirements, including interfacing it with NASA’s existing systems 

environment.a

CSC (Computer 

Services 

Corporation)

Support the operations, maintenance, and administration of the 

new module, including integration efforts.

IBM (International 

Business Machines)

Develop training and user procedures and perform security and 

internal control reviews to ensure that the core financial module 

complies with accounting and financial reporting standards.

SAP Provide technical implementation support and training on NASA’s 

implementation of the core financial module.b

Titan Systems 

Corporation-Civil 

Government 

Services Group

Perform independent verification and validation of requirements 

and testing processes and results, such as tracing requirements to 

test cases.
Page 9 GAO-03-507 NASA's IFMP

NASA’s Acquisition 

Management Strategy 

Does Not Include 

Analyzing Component 

Interdependencies 

A key to effectively acquiring commercial component-based systems that 

are intended to support agencywide business needs, like IFMP, is 

employing recognized acquisition management controls. One such control 

is to acquire system components only after deliberate and comprehensive 

analysis and understanding of the components’ interdependencies. 

Although NASA has already acquired the core financial module and three 

other IFMP commercial components, the agency has not performed the 

analysis necessary to understand the logical and physical relationships 

among the component parts it has acquired. By acquiring these IFMP 

components without first understanding system component relationships, 

NASA has increased its risks of implementing a system that will not 

optimize mission performance and will cost more and take longer to 

implement than necessary. 

When acquiring a commercial component-based system or system of 

systems, such as IFMP, industry best practices5

 recognize the critical 

importance of understanding the logical and physical relationships among 

the component parts. To provide for this understanding, these practices 

advocate that the system integrator, which in the case of IFMP is NASA, 

employ an explicit methodology, including a risk-based process for 

deciding among product alternatives, that collects and verifies information 

about each component’s characteristics and credentials, evaluates the 

dependencies and constraints among these components, and permits 

informed decisions about which products to acquire and how to implement 

them. This is necessary because commercial products are built around 

each vendor’s functional and architectural assumptions and paradigms, 

such as approaches to error handling and data access, and these 

assumptions and paradigms are likely to be different among products from 

different sources. Such differences complicate product integration. 

Further, some commercial products have built-in dependencies with other 

products that if not known can further complicate integration. For these 

reasons, a structured and disciplined approach to systematically evaluating 

product to product relationships is critical. The alternative to such a 

structured and disciplined approach to building a commercial componentbased system is trial and error, which is fraught with risk. 

5

See for example, Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge, Ph.D., An 

Activity Framework for COTS-Based Systems, Technical Report CMU/SEI-2000-TR-010 

(Pittsburgh, Pa.: Software Engineering Institute, Carnegie Mellon University, October 2000).
Page 10 GAO-03-507 NASA's IFMP

In acquiring its IFMP components to date, NASA has not performed the 

above-cited dependency analysis, and it does not have a methodology for 

doing so. Despite this, NASA has acquired and is in the process of 

implementing a commercial product (SAP’s R/3 core financial module) to 

meet its needs in one business area (financial management), and it has 

acquired three additional commercial products from three separate 

vendors that are intended to meet its needs in other business areas (travel 

management, resume management, and human capital position description 

needs).6

 Beyond the four products that it has already acquired, NASA plans 

to acquire an unspecified number of additional commercial components 

that are intended to meet its needs in other business areas. To integrate 

those separate commercial products into a “system of system 

components,” NASA has executed several contracts and plans to execute 

more to build interfaces (hardware and software) to permit the 

components to interoperate. For example, a contractor is currently 

building an interface between the core financial module of the SAP product 

and the travel manager product. 

When acquiring and implementing commercial hardware and software 

solutions, organizations can generally pursue one of two basic courses of 

action. That is, an organization can opt for a single package of already 

integrated software components, which is referred to as the “best of suite” 

approach, or it can opt for different software components from different 

vendors, which is referred to as the “best of breed” approach. NASA is 

currently following the “best of breed” approach. According to the 

Integration Office Deputy Program Manager, NASA has not performed 

dependency analyses among the various components acquired to date, and 

those being considered for later acquisition, because NASA’s initial 

acquisition strategy was to acquire a single commercial solution (i.e., “best 

of suite”) and thus it did not consider product interoperability to be a 

concern. While NASA has since adopted a “best of breed” approach, the 

Integration Office Deputy Program Manager stated that it still does not plan 

to perform these analyses in the future because NASA will rely upon 

commercial tools that support the development of interfaces between 

commercial products, which the Integration Office Deputy Program 

Manager claimed will make integration easy and relatively inexpensive and 

negate the need for proactive dependency analysis. However, best 

6

NASA has acquired the following commercial products: (1) SAP AG’s R/3, version 4.62, 

(2) Gelco’s Travel Manager, version 8.0, (3) Resumix, version 6, and (4) Avue Digital 

Services’ Position Description Management, which is a subscription service.
Page 11 GAO-03-507 NASA's IFMP

practices advocate that proactive dependency analysis and evaluation are 

necessary for informed decision making regardless of whether integration 

tools will be used, and particularly when a “best of breed” approach is 

employed.

What this means is that NASA is implementing its “best of breed” approach 

using trial and error. This reactive method does not allow for adequate 

understanding of commercial product dependencies until the only 

alternative to integrating them is building and maintaining complex 

interfaces, which unnecessarily increase system acquisition and 

maintenance costs, delay promised capabilities and benefits, and do not 

optimize agency performance. The results of a recent study7

 commissioned 

by NASA recognize the added risk associated with the “best of breed” 

approach, and thus the importance of proactive dependency analysis and 

evaluation to minimize this risk. Specifically, the study states that NASA’s 

“best of breed” approach will result in a higher total cost of ownership 

because the agency will need to (1) acquire and maintain multiple software 

licenses, (2) hire and maintain technical staff knowledgeable about each 

commercial product, (3) build and maintain interfaces to integrate the 

various products, and (4) provide training to system users on each 

commercial product.

Core Financial Module 

Does Not Fully 

Address Key User 

Information 

Requirements 

If implemented as planned, the core financial module may improve NASA’s 

current system environment by eliminating the separate, incompatible 

accounting systems that have been used at each of NASA’s 10 centers 

previously. However, the core financial module currently being 

implemented does not fully address the information requirements of key 

users, such as program managers, cost estimators, or the Congress. Our 

previous work at leading public and private sector organizations8

 has 

shown that user involvement and effectively reengineering business 

processes are major factors in successfully implementing financial 

management systems. In contrast, at NASA, key users such as program 

managers and cost estimators were not involved in defining or 

implementing NASA’s system requirements and have played a limited role 

7

Gartner, Inc., A Report for NASA: IFMP Lessons Learned and Key Considerations for 

Future Module Projects, January 20, 2003.

8

U.S. General Accounting Office, Executive Guide: Creating Value Through World-class 

Financial Management, GAO/AIMD-00-134 (Washington, D.C.: April 2000).
Page 12 GAO-03-507 NASA's IFMP

in all aspects of the implementation of the core financial module. Instead, 

NASA’s financial managers and accountants have primary responsibility for 

this process. Consequently, NASA has not effectively used this opportunity 

to reengineer the way it does business and implement a financial 

management system that addresses many of its most significant 

management challenges, including improving contract management, 

producing credible cost estimates, and providing the Congress with 

appropriate visibility over its large, complex programs. According to IFMP 

officials, they chose to forgo certain system capabilities to expedite 

implementation of the core financial module and have stated that these 

capabilities can be added at a later date. In the meantime, program 

managers and cost estimators will continue to rely on other nonintegrated 

systems outside of IFMP and use other labor-intensive means to capture 

the data they need to manage programs such as the ISS. 

If Implemented as Planned, 

Core Financial Module May 

Provide Some 

Improvements

The core financial module, if implemented as planned, may provide some 

improvement to NASA’s current accounting system environment. 

According to IFMP planning documents, the core financial module should 

(1) eliminate much of the inconsistent data and lack of standardization, 

(2) collect agency costs and allocate those costs to cost centers, including 

civil service personnel costs, and (3) maintain a standard general ledger to 

provide control over financial transactions, resource balances, and assets 

and liabilities. If NASA is successful, the core financial module could 

reduce the extensive amount of time and resources currently required to 

consolidate NASA’s 10 different reporting entities and close the books each 

accounting period. However, as discussed later, our findings relating to 

NASA’s requirements management and testing processes may affect NASA’s 

ability to achieve these improvements. 

Key Users Were Not 

Involved in the 

Implementation of the Core 

Financial Module

The IFMP core financial module, although technologically capable of 

meeting the needs of program managers, cost estimators, and the 

Congress, is not being configured to do so because these key users have 

not been actively involved in the implementation of the module. Our 

previous work at leading public and private sector organizations has shown 

that user involvement in reengineering business processes and establishing 

and implementing system requirements are major factors in successfully
Page 13 GAO-03-507 NASA's IFMP

implementing financial management systems.9 In fact, at these leading 

organizations, not only did program and business managers participate in 

the design and implementation of financial management systems, they 

typically were responsible for driving the effort and played a key role in 

reengineering core business processes. In contrast, at NASA, financial 

managers and accountants have had primary responsibility for the 

implementation of the core financial module, while the other key users 

mentioned above have been largely excluded. 

According to IFMP planning documents, financial managers and 

accountants are considered direct customers responsible for the 

administrative processes that will be reengineered and automated. 

Therefore, these individuals, to date, have been engaged in defining system 

requirements and priorities. On the other hand, stakeholders—including 

program mangers, cost estimators, and the Congress—are described in 

NASA documents as the ultimate beneficiaries of system improvements but 

are not expected to be actively involved in the system’s implementation. 

While NASA has formed teams to reengineer portions of the agency’s 

administrative process, these teams primarily consisted of financial 

managers. As a result, NASA has neither reengineered its core business 

processes nor established adequate requirements of the system to address 

many of its most significant management challenges, including improving 

contract management, producing credible cost estimates, and providing the 

Congress with appropriate visibility over its large, complex programs.

The Core Financial Module 

Will Not Provide the 

Information Needed to 

Manage Contracts

The core financial module is not being implemented to provide program 

managers with the information they need to fully monitor the work being 

performed by contractors. Based on our review of NASA’s three largest

9

GAO/AIMD-00-134.
Page 14 GAO-03-507 NASA's IFMP

space flight programs—Space Launch Initiative,10 ISS, and Space Shuttle—

we found that the core financial module, as currently planned, will not 

accommodate much of the information provided by NASA’s contractors 

and needed by program managers to monitor contractor performance. 

Specifically, the core financial module is not being implemented to 

(1) accommodate the contract schedule information received from 

contractors and needed by program managers to monitor contractor 

performance and (2) maintain cost data at a sufficient level of detail for 

certain contracts. 

Core Financial Module Does Not 

Integrate Cost and Schedule 

Data Needed by Program 

Managers

To adequately oversee NASA’s largest contracts, program managers need 

reliable contract cost data—both budgeted and actual—and the ability to 

integrate these data with contract schedule information to monitor 

progress on the contract. However, because program managers were not 

involved in defining system requirements or reengineering business 

processes, the core financial module is not being designed to integrate cost 

and schedule data needed by program managers. As a result, program 

managers are resorting to using other systems that will result in additional 

cost over and above IFMP costs. 

The primary source of contract schedule information used by program 

managers comes directly from NASA’s contractors in the form of monthly 

hard copy or electronic cost and schedule performance reports. NASA 

tracks contract schedule status by comparing the budgeted and actual cost 

of work planned with budgeted and actual cost of work completed for 

specific time periods. The term “schedule” incorporates both the concept 

of status of work and whether a project or task is being completed within 

planned time frames. Depending on the nature of the work being 

performed, the method of measuring work progress varies. Work is 

measured in terms of tasks when a specific end product or end result is 

10During the time of our review, NASA was pursuing a program—known as the Space 

Launch Initiative—to build a new generation of space vehicles to replace its aging space 

shuttle. This was part of NASA’s broader plan for the future of space travel—known as 

NASA’s Integrated Space Transportation Plan. On October 21, 2002, NASA postponed further 

implementation of the program to focus on defining the Department of Defense’s role, 

determining future requirements of the ISS, and establishing the agency’s future space 

transportation needs. In November 2002, the administration submitted to the Congress an 

amendment to NASA’s fiscal year 2003 budget request to implement a new Integrated Space 

Transportation Plan. The new plan makes investments to extend the space shuttle’s 

operational life and refocuses the Space Launch Initiative program on developing an 

orbital space plane—which provides crew transfer capability to and from the space 

station—and next generation launch technology.
Page 15 GAO-03-507 NASA's IFMP

produced. But when work does not produce a specific end product or 

result, level-of-effort or a more time-oriented method of measurement is 

used. The type of information, level of detail, and reporting format 

provided by contractors are determined during the contract negotiation 

process and vary from contract to contract depending on the size, 

complexity, and duration of the contract. In general, however, these reports 

show contractor progress against cost and schedule targets set by the 

program manager and against which contractor performance can be 

measured. Contractors also report any significant variances from the 

targets and explain how they will be mitigated. 

NASA’s program managers need this contractor information to plan and 

manage their programs effectively. However, the information from cost and 

schedule performance reports is not recorded in the core financial module. 

Instead, NASA uses only data from monthly contractor financial 

management reports, commonly referred to as NASA form 533 reports, to 

update the core financial module. NASA form 533 reports contain 

estimated and actual contractor cost data but, according to NASA program 

managers, do not contain the data needed to adequately assess schedule 

performance. According to IFMP officials, the information needed to 

perform cost and schedule analysis by program managers is outside the 

scope of the core financial module and IFMP. IFMP program officials told 

us that they chose to forgo certain system capabilities to expedite 

implementation of the core financial module and have stated that these 

capabilities can be added at a later date. However, NASA does not currently 

have a plan for maintaining the data contained in cost and schedule 

performance reports in the core financial module or IFMP. 

Because contract schedule information is not currently maintained through 

the core financial module, program managers will continue to rely on hard 

copy reports, electronic spreadsheets, or other means to monitor 

contractor performance. Several of NASA’s programs, including the Space 

Launch Initiative and the ISS, are currently using other systems to monitor 

contract cost and schedule performance, but these are stand-alone efforts 

and have not been part of a coordinated NASA plan. Officials at Marshall 

Space Flight Center have recognized the importance of maintaining 

common cost and schedule performance data in a single integrated system 

that is available to all NASA managers at all locations. As such, these 

officials have proposed that NASA establish the system they currently use 

as a NASA-wide standard. 
Page 16 GAO-03-507 NASA's IFMP

NASA has stated that the core financial module is expected to result in a 

single, integrated financial management system that is intended to serve 

the needs of its program managers. By not including the cost and schedule 

information needed by program managers in the core financial module, 

NASA risks operating with two sets of books—one that is used to report to 

management and the Congress and another that is used to manage NASA’s 

programs.

Core Financial Module Will Rely 

on Legacy Coding Structure

Because NASA has not fundamentally changed the way it operates by 

involving key users in business process reengineering efforts, the core 

financial module is not being implemented to capture cost information at 

the same level of detail that it is received from NASA’s contractors. Instead 

of implementing an accounting code structure that would meet the 

information needs of program managers, NASA has embedded the same 

accounting code structure that it uses in its legacy reporting system in the 

core financial module. As a result, the availability of detailed cost data is 

dependent on the adequacy of NASA’s legacy coding structure. In some 

cases, the cost information received by NASA on monthly contractor 

reports must be aggregated to a higher, less detailed level before it is 

posted against the old accounting code structure. For example, as shown in 

figure 1, program managers for the Space Shuttle receive monthly 

contractor reports on the space flight operations contract that track costs 

related to friction stir weld and propulsion safety upgrades separately. 
Page 17 GAO-03-507 NASA's IFMP

Figure 1: Space Shuttle Flight Operations Contract

Note: Amounts shown are for illustrative purposes only.

However, because the NASA legacy accounting code structure embedded 

in the core financial module only tracks the cost of space shuttle flight 

hardware upgrades, the more detailed costs that program managers need, 

such as friction stir weld and propulsion safety upgrades, are not available 

through the core financial module. According to NASA officials, the core 

financial module is capable of capturing this more detailed cost data; 

however, due to the complexity associated with converting detailed data 

from the centers’ legacy systems, NASA has deferred this capability. While 

this information is available to program managers from the contractor, it is 

not available through the core financial module. In fact, on this particular 

contract, program managers have access to the contractor’s system and, 

therefore, have access to an even greater level of detail than that reported 

by the contractor on hard copy reports. 

On the other hand, in cases where the legacy coding structure adequately 

reflects the programs’ information needs, the cost data received from 

Flight

Hardware

$100

Flight hardware

upgrades

$40

Level captured

in the core

financial module

Level received

from contractors

Flight hardware

safety upgrades

$40

Friction stir weld

upgrades

$10

Propulsion

upgrades

$10

Solid rocket

booster

$60 Data to

IFMP

Space Shuttle

Flight Operations

$100

Source: GAO analysis of NASA cost data.
Page 18 GAO-03-507 NASA's IFMP

contractors do not have to be aggregated prior to posting. For example, 

program officials with the ISS program recently redesigned the program’s 

cost coding structure in order to more precisely identify the cost of specific 

work. This was not done as part of an IFMP reengineering effort, but in 

response to external criticisms of the program’s failure to manage its costs. 

Regardless of the reason, the program’s reengineering effort has to some 

extent improved the usefulness of the cost data being entered into the core 

financial module.

Core Financial Module Will 

Not Provide the Information 

Needed to Prepare Credible 

Cost Estimates

The core financial module, as currently planned, will not provide 

sufficiently detailed data for cost estimators. Although the core financial 

module is technologically capable of maintaining the detailed data required 

by cost estimators, cost estimators were not involved in defining the 

system requirements or reengineering business processes. As a result, 

NASA has not determined the most cost-effective way to satisfy the 

information needs of its cost estimators nor reengineered its business 

process to ensure that their needs are met.

According to members of NASA’s cost estimating community, they typically 

need cost data at an even greater level of detail than that currently being 

provided by NASA’s contractors. The cost estimators we spoke with told us 

that their requests for more detailed cost data are often not satisfied 

through the contract negotiation process. For example, as shown in figure 

2, while program managers may want—and contractors may provide—the 

cost of an engine fan, cost estimators need to know more detailed 

information, including the cost of the various tasks needed to make a rotor 

assembly, which ultimately becomes part of the fan.
Page 19 GAO-03-507 NASA's IFMP

Figure 2: Example of Level of Detail Reported versus That Required by Cost Estimators

The lack of sufficiently detailed information for cost estimators is due to 

NASA’s lack of reengineering efforts for the acquisition management 

process, which should have been done prior to implementing the core 

financial module. Because the core financial module will not contain 

sufficiently detailed historical cost data necessary for projecting future 

costs, cost estimators will continue to rely on labor-intensive data 

collection efforts after a program is completed. These efforts involve 

searching through old hard copy and electronic contractor reports to 

extract all relevant data. NASA pays its contractors extra to provide data 

required but not contained in these reports, usually at a later point in time. 

Contract

Engine

Fan Compressor

Fan assembly

Case assembly Rotor assembly

Work packages

Level of detail

required by cost

estimators

Level of detail

often provided

by contractors

Task 1

Task 2

Task 3

Full scale fan rig Minor fan rig Dual spool

compressor rig

Turbine

Auxiliary

power

Source: NASA Work Breakdown Structure Reference Guide.
Page 20 GAO-03-507 NASA's IFMP

Data collection after the fact is expensive but, according to some NASA 

officials, is more cost effective than requiring contractors to provide 

detailed cost data throughout the course of the contract. However, NASA 

has not done the analysis needed to determine the appropriate mix of 

routinely requiring contractors to provide detailed cost data and capturing 

that data in the core financial module versus purchasing the data after a 

contract is complete. 

Core Financial Module May 

Not Provide Better 

Information for 

Congressional Oversight

NASA has identified the Congress as a key stakeholder and ultimate 

beneficiary of system improvements. However, based on our discussions 

with congressional staffs from NASA’s authorizing committees, the agency 

did not consult with them regarding their information needs. Consequently, 

NASA cannot be sure that it is implementing a system that will provide the 

Congress with the information it needs for oversight. As discussed 

previously, according to IFMP planning documents, financial managers and 

accountants are considered direct customers and are responsible for 

defining system requirements and priorities. On the other hand, NASA 

considered the Congress a stakeholder and, therefore, did not seek input 

from congressional staffs in defining system requirements.

Similar to the problems faced by program managers and cost estimators, 

the core financial module may not address many of the information needs 

of the Congress. To properly assess the agency’s annual budget submission 

and make funding decisions, the Congress needs timely, reliable cost and 

schedule information on the status of large, high-risk programs, such as the 

ISS. As previously described, the module will not provide the type of cost 

and schedule information that program managers need to adequately 

monitor the status of NASA’s major programs and may not maintain 

sufficient information to readily address any special congressional needs 

that arise. 

Nevertheless, the Congress should be able to receive somewhat better 

information about NASA’s finances than it has in the past because, as 

previously described, the core financial module may improve some aspects 

of NASA’s ability to produce reliable financial information. For example, 

the use of a standard general ledger will provide more standardized 

accounting data and general ledger controls. As a result, the core financial 

module should enable NASA to provide timelier, more reliable high-level 

cost information to the Congress on some issues, such as annual spending 

limits. 
Page 21 GAO-03-507 NASA's IFMP

NASA’s Requirements 

Management Process 

for the Core Financial 

Module Is Ineffective

NASA has not effectively implemented a requirements management 

process11 to support the implementation of the core financial module and 

therefore has increased the risk that the agency will not be able to 

effectively identify and manage the detailed system requirements that 

system developers and program managers use to acquire, implement, and 

test a system. Specifically, based on discussions with IFMP officials and a 

review of the process documents related to the core financial module, we 

found that NASA was relying on a requirements management process that 

did not require detailed documentation of system requirements prior to 

system testing. Industry best practices, as well as NASA’s own system 

planning documents, indicate that detailed system requirements should be 

documented to serve as the basis for effective system testing. Instead, 

NASA’s approach relied on the expertise of certain subject matter experts 

to remember the detailed requirements necessary to evaluate the 

functionality actually provided. 

As a result of this approach, we found that (1) for over 80 percent of the 132 

core financial module requirements we reviewed, the functionality to be 

delivered was not adequately described or stated in a manner that allowed 

for quantitative evaluation and (2) the traceability among the various 

requirement documents was not maintained. Accordingly, the potential for 

these requirements defects to result in costly rework is significant and 

increases the risk that the core financial module will not meet its cost, 

schedule, and performance objectives. Because of the direct relationship 

between requirements and testing, the lack of complete and unambiguous 

requirements also significantly impairs the testing phase. Furthermore, 

NASA has not effectively implemented the types of metrics that can help it 

understand the effectiveness of its requirements management process, 

such as identifying and quantifying any weaknesses in its process and then 

developing the corrective actions needed.

11According to the Software Engineering Institute, requirements management is a process 

that establishes a common understanding between the customer and the software project 

manager regarding the customer’s business needs that will be addressed by a project. A 

critical part of this process is to ensure that the requirements development portion of the 

effort documents, at a sufficient level of detail, the problems that need to be solved and the 

objectives that need to be achieved. 
Page 22 GAO-03-507 NASA's IFMP

NASA Requirements 

Management Process Was 

Not Designed to Provide 

Detailed System 

Requirements

Requirements are the specifications that system developers and program 

managers use to acquire, implement, and test a system. Requirements 

should be consistent with one another, verifiable, and directly traceable to 

higher-level business or functional requirements. It is critical that 

requirements be carefully defined and that they flow directly from the 

organization’s concept of operations (how the organization’s day-to-day 

operations are or will be carried out to meet mission needs).12 Improperly 

defined or incomplete requirements have been commonly identified as a 

root cause of system failure and systems that do not meet their cost, 

schedule, or performance goals. Without adequately defined requirements 

that have been properly reviewed and validated, a significant risk exists 

that the system will need extensive and costly changes before it will meet 

NASA’s needs.

As discussed previously, NASA is designing and fielding the core financial 

module without having determined the specific information needs of its 

key stakeholders, including program managers, cost estimators, and the 

Congress. The omission of this critical step increased the risk that the 

project would not effectively include all the detailed system requirements 

that were needed to achieve management’s vision of a core financial 

management module that provides timely, consistent, and reliable cost and 

performance information for management decisions. 

IFMP officials stated that their basic approach to developing the core 

financial module system requirements was (1) defining high-level 

requirements that could be used for making a software selection, 

(2) defining the business processes that the core financial module needed 

to address, (3) linking the requirements that were originally defined for the 

software selection to those business processes, and (4) using subject 

matter experts to determine whether the application met the business 

processes envisioned by the users during their discussions of the needed 

functionality. A key feature of the NASA approach is that the detailed 

requirements covered in the discussion of the business processes are not 

12According to Institue of Electrical and Electronics Engineers Standard 1362-1998, a 

concept of operations document is normally one of the first documents that is produced 

during a disciplined development effort since it describes system characteristics for a 

proposed system from the user's viewpoint. This is important since a good concept of 

operations document can be used to communicate overall quantitative and qualitative 

system characteristics to the user, developer, and other organizational elements. This allows 

the reader to understand the user organizations, missions, and organizational objectives 

from an integrated systems point of view. 
Page 23 GAO-03-507 NASA's IFMP

required to be documented prior to testing. Rather, NASA depends on 

subject matter experts, who are assigned to ensure that the core financial 

module has the needed functionality, to know the detailed requirements 

necessary to evaluate the functionality actually provided. Such an 

approach relies on the subject matter expert being available throughout the 

process and on the expert remembering the undocumented requirements 

completely and consistently. Specifically, an individual assigned to develop 

a test case13 is relied on to understand the detailed requirements associated 

with all facets of that test case and then to ensure that the test will provide 

the information needed to understand whether the functionality was 

actually provided. 

IFMP officials also stated that the current approach was based on 

discussions with their contractors and eliminated the need for detailed 

documented requirements normally associated with efforts such as IFMP. 

They also recognized that this approach was somewhat inconsistent with 

their own Requirements Management Framework, issued in October 2000, 

which stated that “[i]n order to test the software, a more detailed statement 

of a requirement or process may be required to insure [sic] the successful 

completion of a test.” This document also recognized that these detailed 

requirements would be needed for “a more refined testable set of 

requirements . . . and needed to serve as a basis for the testing that will 

occur . . .” In a January 2003 report14 by a contractor on the lessons learned 

on the IFMP effort, the contractor noted that NASA would need to develop 

a set of requirements and design specifications that had been validated by 

the individuals responsible for managing each process. The contractor also 

noted that although such an approach delays the first phase of the project 

design, it reduces the overall implementation time. 

As a result of NASA’s stated approach to requirements management, our 

review of NASA’s system requirements related to the process documents 

for the core financial module found that key attributes of effective 

requirements were missing for many requirements. According to the 

Institute of Electrical and Electronics Engineers (IEEE)—a leading source 

13A test case is a series of actions, performed serially, in parallel, or in some combination, 

that creates the desired test conditions. Rex Black, Managing the Testing Process: 

Practical Tools and Techniques For Managing Hardware and Software Testing (Redmond, 

Wash.: Microsoft Press, 1999).

14Gartner, Inc. 
Page 24 GAO-03-507 NASA's IFMP

for defining the best practices for efforts such as this—good requirements 

have several characteristics, including the following:15

• The requirements document contains all the requirements identified by 

the customer, as well as those needed for the definition of the system.

• The requirements fully describe the software functionality to be 

delivered. Functionality is a defined objective or characteristic action of 

a system or component. For example, a system may have inventory 

control as its primary functionality.

• The requirements are stated in clear terms that allow for quantitative 

evaluation. Specifically, all readers of a requirement should arrive at a 

single, consistent interpretation of it.

• Traceability among various requirement documents is maintained. 

Requirements for projects such as IFMP can be expressed at various 

levels depending on user needs. They range from agencywide business 

requirements to increasingly detailed functional requirements that 

eventually permit the software project managers and other technicians 

to design and build the required functionality in the new system. 

Adequate traceability ensures that a requirement in one document is 

consistent with and linked to applicable requirements in another 

document.

NASA established about 590 requirements for the core financial module.16

We reviewed in detail one business process area of this module—the 

“Manage Accounts Payable” process—that included 132 of these 

requirements. We found that (1) for over 80 percent of the 132 requirements 

the functionality to be delivered was not adequately described or stated in a 

manner that allowed for quantitative evaluation and (2) the traceability 

between the various requirement documents was not maintained. 

Requirements Were Not Specific For over 80 percent of the 132 “Manage Accounts Payable” requirements, 

the process documents lacked the specific information necessary to 

understand the required functionality that should be provided and how to 

15IEEE 830-1998.

16NASA originally identified about 590 requirements. However, 51 of these were deleted and 

86 were deferred.
Page 25 GAO-03-507 NASA's IFMP

determine quantitatively, through testing or other analysis, whether the 

system will meet NASA's needs. The following are examples of core 

financial module requirements that lacked the necessary specificity. 

• One requirement stated that the system must “[a]llow the information 

contained in the system to be queried to present detailed data as 

requested (such as payee information). The capability to perform a Print 

Screen must be available to all user screens.” This requirement did not 

clearly state such items as (1) the data elements that must be supported, 

(2) how the user would obtain the data definitions for the data elements 

that could be used, (3) the tool or process that would be used to perform 

these queries, and (4) the relationship between the ability to perform 

such queries and the requirement to be able to print the screen.

• The core financial module was required to support “multiple payment 

addresses and/or bank information for a single payee.” This requirement 

did not clearly state the maximum number of payment address and bank 

information entries that should be allowed.

• Several requirements called for the core financial module to make 

accounting entries; however, these requirements did not define the 

specific accounting entries that should be made. 

The lack of documented requirements that are complete and unambiguous 

not only increases the risk that the project's functionality goals will not be 

met, but also significantly impairs the testing phase of the system 

implementation efforts, as discussed later in this report. 

Traceability Was Not Maintained NASA has adopted a four-level approach to defining its requirements—

processes, subprocesses, activities, and tasks, with processes stating highlevel requirements and tasks providing the most detailed level. In reviewing 

the various requirement documents, we found that (1) traceability was not 

always maintained through the various documents and (2) the level of 

detail did not provide additional specificity for a given requirement as it 

progressed through the hierarchy. 

Traceability allows the user to follow the life of the requirement both 

forward and backward through these documents and from origin through 

implementation. Traceability is also critical to understanding the 

parentage, interconnections, and dependencies among the individual 

requirements. This information in turn is critical to understanding the 

impact when a requirement is changed or deleted. Without an effective 
Page 26 GAO-03-507 NASA's IFMP

traceability approach, it is very difficult to perform such actions as 

(1) accurately determining the impact of changes and making value-based 

decisions when considering requirement changes, (2) maintaining the 

system once it goes into production, (3) tracking the project's progress, and 

(4) understanding the impact of a defect discovered during testing. 

To illustrate these issues, we attempted to follow the hierarchy of 

requirements for one of the core financial module's seven processes 

through the four levels of requirements utilized by NASA for this project. 

As shown in figure 3, the “Manage Accounts Payable” process area has 132 

requirements associated with nine subprocesses. However, one of the 132 

requirements, "Multiple User Access," contained in the “Manage Accounts 

Payable” process, was not shown in any of the subprocesses, and it was 

unclear where this requirement would be further defined. 
Page 27 GAO-03-507 NASA's IFMP

Figure 3: System Requirements for the “Manage Accounts Payable” Process

Note: The total number of requirements shown for the subprocesses (202) exceeds the number of 

requirements shown for the “Manage Accounts Payable” process (132) because some requirements 

apply to more than one activity.

Our review of the nine subprocesses found that 5 of the remaining 131 

requirements contained in the subprocesses were not linked to any activity. 

For example, a requirement that applies to all federal agencies and is 

designed to ensure compliance with the Internal Revenue Service's 1099 

reporting requirements was not included in any of the activities. Therefore, 

the individuals responsible for implementing the requirements contained in 

the activities would not have the full universe of requirements they must 

address. Conversely, as shown in figure 4, several of the activities for the 

“Validate Payment” subprocess did not contain any related requirements. 

Therefore, it was unclear whether these activities should have been 

associated with this subprocess. For example, the “clear advance” and 

“adjust invoice” activities did not include any requirements related to 

validating payments. Further, the lack of requirements for these activities 

may cause confusion for the individuals assigned to test the functionality 

Manage accounts

payable

Validate payment (17)

Enter invoice (41) Manage travel (33)

Maintain AP master data (11) Process HHS (9)

Execute and manage payments (49) Process IPAC payments (20)

Report on payment activities (14) Recertify payment activities (8)

Source: NASA.
Page 28 GAO-03-507 NASA's IFMP

associated with this subprocess. We were also unable to trace the 

requirements from activities to tasks, which should be the most detailed 

level of requirements because, for the activities associated with the 

“Validate Payment” subprocess, NASA used the same information for the 

activities and tasks. In other words, the requirements for the tasks were 

identical to those listed for the activities and therefore did not provide any 

additional details.

Figure 4: Requirements for “Validate Payment” Subprocess

Note: The total number of requirements shown for the activities (22) exceeds the number of 

requirements shown for the “Validate Payment” subprocess (17) because some requirements apply to 

more than one activity. 

As can been seen in this example, NASA was unable to maintain adequate 

traceability of the requirements for this subprocess as it progressed 

through the hierarchy. More important, the level of specificity associated 

with these requirements did not change. Based on our review, we generally 

found that the wording of a given requirement was identical regardless of 

Validate Payment

(17)

Adjust invoice (1) Validate payment (0)

Clear advance (0) Set flag (2)

Check for other deductions (1) Determine other adjustments (2)

Check for proper approvals (1) Post invoice (11)

Determine holdback status (1)

Adjust payment amount as neccessary (2)

Compute holdback amount (1)

Source: NASA.
Page 29 GAO-03-507 NASA's IFMP

the requirement document reviewed. For example, if a requirement was 

listed in a subprocess area and flowed to an activity, the same wording was 

used and the needed level of specificity to help ensure proper 

implementation was not available. Therefore, although NASA appeared to 

have adopted a requirements hierarchy that would facilitate the needed 

specificity as the requirements flowed from subprocesses to tasks and 

activities, the implementation of this approach did not address the 

specificity problems discussed earlier. Accordingly, this is another factor 

that increases the risk that this project will not meet its schedule, cost, and 

functionality goals. A NASA contractor hired to help evaluate the 

implementation of the core financial module had similar findings. For 

example, the contractor found that NASA had not developed 

documentation that explicitly details the relationship between lower-level 

requirements and requirements of the next level.

Requirements Defects 

Adversely Affect Testing of 

the Core Financial Module

Because requirements provide the foundation for system testing, the 

specificity and traceability defects in the system requirements preclude 

NASA from implementing a disciplined testing process. That is, 

requirements must be complete, clear, and well documented to design and 

implement an effective testing program. Consequently, NASA is taking a 

significant risk that its testing efforts will not detect significant defects 

until after the system is placed into production. Industry best practices 

indicate that the sooner a defect is recognized and corrected, the cheaper it 

is to fix. This is especially true since NASA is depending on the subject 

matter experts’ knowledge, rather than documented requirements, to 

ensure that the application does not have any significant defects before the 

system is placed into production.

As shown in figure 5, there is a direct relationship between requirements 

and testing. 
Page 30 GAO-03-507 NASA's IFMP

Figure 5: Relationship between Requirements Development and Testing

Although the actual testing activities occur late in the development cycle, 

test planning can help disciplined activities reduce requirements-related 

defects. For example, developing conceptual test cases based on the 

requirements derived from the concept of operations and functional 

requirements stages can identify errors, omissions, and ambiguities long 

Source: GAO.

User acceptance testing

Verifies that system operates

correctly with operational hardware

and meets users' needs

Concept of operations

Specifies how system is used in

operation

System acceptance testing

Verifies that the complete system

satisfies functional requirements

Functional requirements

Specifies the high-level functions

of the system

Integration testing

Verifies that units of software, when

combined, work together as

intended

Design requirements

Specifies the tasks each software

component must perform

Unit testing

Verifies that each component of the

software faithfully implements the

detailed design

Detailed design and coding

Specifies the detailed steps for

each software component and

implements those steps

Stages of system development Stages of testing

For projects such

as IFMP, these

items are normally

handled by the

commercial 

software vendor
Page 31 GAO-03-507 NASA's IFMP

before any code is written or a system is configured. Disciplined 

organizations also recognize that planning testing activities in coordination 

with the requirements development process has major benefits. 

We have identified several indications that NASA’s testing program has 

been adversely affected by the lack of complete and specific requirements. 

Although we plan to continue our review of NASA’s testing plan for IFMP 

implementation, we noted (1) significant defects that appeared to be 

related to requirements occurred in the application after it was placed into 

production and (2) several cases where NASA did not ensure that 

modifications made to the application did not cause unintended effects and 

that the system or component still complied with its specified requirements 

after the change. 

Significant Defects Appeared in 

Production System

Our review of the system test defect reports for the core financial module 

disclosed that several defects considered by NASA to have an initial 

severity rating of critical17 or high18 had been identified after the system 

was placed into production at Marshall Space Flight Center and Glenn 

Research Center. Detecting such problems after the system goes into 

production may lead to costly rework due to factors such as having to 

reenter transactions and adjust reports manually. Furthermore, the manual 

processes required to make these adjustments may introduce data integrity 

errors. Our preliminary review indicated that the root cause of many of 

these defects could be linked to the lack of complete requirements. For 

example, see the following:

• Shortly after the system was placed into production, NASA found that 

the core financial module was not properly executing certain business 

rules. An emergency fix was developed, and the defect report noted that 

a long-term solution and requirements would need to be developed. It 

was unclear why the subject matter experts did not include the business 

rules in the test cases used to evaluate the functionality of the 

17NASA defines critical defects as those that “impact the ability to move forward or 

complete an entire business function or task, and impacts multiple business functions, 

multiple users and/or locations. [It] presents a failure that has no workaround or 

alternative.”

18NASA defines high defects as those that have “a significant impact on the completion of a 

business function or task, however, activities can continue as far as the next function. A 

limited number of business functions, business users, or locations are impacted, and may be 

an impact to only one.”
Page 32 GAO-03-507 NASA's IFMP

application. However, we believe one cause is the lack of documented 

requirements that the testers could use to develop effective test cases.

• NASA was unable to process vendor invoices that contained over 200 

line items, which, according to NASA officials, is a common occurrence 

on NASA’s large contracts. It was unclear why the subject matter experts 

responsible for testing this functionality had not developed the test 

cases to ensure that large invoices were properly processed before the 

system was placed into production. IFMP officials recognized that this 

was an oversight in their testing process. If NASA had documented its 

requirements, it would have recognized that a properly developed test 

case had not been designed to cover this necessary functionality. 

• About 3 months after the core financial module was placed into 

production at one center, it was found that when the system produced 

multiple bills for the same customer, only the first bill was sure to have 

the proper account classification code printed. The remaining bills often 

contained incorrect values since the program improperly assumed that 

the account classification code would not change until the customer 

changed. Since the account classification code is critical for these types 

of bills, the center was required to manually make the necessary 

corrections on the bills. 

Adequate Regression Testing Is 

Not Being Performed

Efforts such as the core financial module undergo change constantly at this 

stage of their development as functionality is being added and defects are 

being corrected. However, before the revised application is released, 

testing needs to be performed to ensure that any modifications have not 

caused unintended effects and that the system or component still complies 

with its specified requirements. This practice is commonly referred to as 

regression testing. An effective regression testing program is critical for 

ensuring that the functionality associated with requirements that has been 

validated during previous testing efforts has not been impaired by 

subsequent changes in the application.

Although NASA officials stated that they require regression testing before 

deploying any changes, we found that they do not have an effective method 

to ensure that adequate regression testing is being performed or that a 

consistent approach is being taken in performing such testing. According 

to NASA officials, the individual identifying a defect is responsible for 

ensuring that the defect is corrected and for determining the amount of 

testing necessary to ensure that the defect has been corrected. For 

example, if a defect is identified when executing a test case, the tester may 
Page 33 GAO-03-507 NASA's IFMP

only test the section of the case where the error was originally identified 

rather than performing all the steps in the test case. This approach 

increases the risks that defects introduced into the application during 

enhancements or defect corrections will not be detected until after the 

application is deployed, which results in costly rework. We noted several 

examples where NASA appeared to perform inadequate regression testing. 

These include the following:

• After adding an interface, it was found that an application screen for 

recording advances did not operate as it had before the change was 

made.

• A process for recording transactions provided by the Department of the 

Treasury failed after an update to the application program.

• One center was testing a certain type of invoice and received an error 

message. This error was attributed to a system patch that had been 

applied to the application.

IFMP officials agreed that they did not have a comprehensive regression 

testing program with a consistent approach. However, they told us that 

they believed that any defects would be detected by the centers as the 

application progresses through its releases because they encourage each 

center to completely retest the application before placing the application 

into production. This approach is particularly risky in light of the 

requirements defects discussed previously, which substantially increase 

the risk that the testing conducted by the centers not yet operational may 

not detect any negative impacts associated with a system change. However, 

IFMP officials recognized that, after all centers are in production, a 

regression testing program will be needed.

A NASA contractor monitoring this project also identified potential 

problems relating to regression testing. According to the contractor 

performing this work, NASA has not implemented the testing tools 

necessary to adequately perform the regression testing to provide NASA 

reasonable assurance that changes made in a given software release do not 

have any adverse consequences for future releases. 
Page 34 GAO-03-507 NASA's IFMP

Performance Metrics Could 

Be Used to Assess Potential 

Risks of Identified 

Weaknesses

Without a well-documented set of requirements, it is impossible to place an 

error in context and understand the cause of the defect—for example, 

determining whether the error was caused by the underlying requirements 

or by some other process failure, such as inadequate testing or inadequate 

controls over system configuration. NASA has not effectively captured the 

types of metrics that can help the organization understand the 

effectiveness of its management processes, such as identifying and 

quantifying any weaknesses in its requirements management process. 

Accordingly, NASA is unable to implement a metrics measurement process 

that allows it to understand (1) its capabilities to manage the IFMP effort, 

(2) how its process problems will affect its cost, schedule, and 

performance objectives, and (3) the corrective actions needed to reduce 

the risks associated with the problems identified.

The Software Engineering Institute (SEI) has found that metrics identifying 

important events and trends are invaluable in guiding software 

organizations to informed decisions. Key SEI findings relating to metrics 

include the following:

• The success of any software organization depends on its ability to make 

predictions and commitments relative to the products it produces.

• Effective measurement processes help software groups succeed by 

enabling them to understand their capabilities, so that they can develop 

achievable plans for producing and delivering products and services.

• Measurements enable people to detect trends and to anticipate 

problems, thus providing better control of costs, reducing risks, 

improving quality, and ensuring that business objectives are achieved.19

A critical element in helping to ensure that a project meets its cost, 

schedule, and performance goals is to ensure that defects are minimized 

and corrected as early in the process as possible. Although NASA has a 

system that captures the defects that have been identified during testing, 

we found that the agency did not analyze its identified defects to determine 

their root causes. Understanding the root cause of a defect is critical to 

evaluating the effectiveness of a process. For example, if a significant 

19William A. Florac, Robert E. Park, and Anita D. Carleton, Practical Software 

Measurement: Measuring for Process Management and Improvement (Pittsburgh, Pa.: 

Software Engineering Institute, Carnegie Mellon University, 1997).
Page 35 GAO-03-507 NASA's IFMP

number of defects are caused by inadequate requirements definition, then 

the organization knows that the requirements management process it has 

adopted is not effectively reducing risks to acceptable levels. IFMP officials 

stated that they do not capture the root causes of their defects. 

Our initial observations identified that the root cause of many defects 

appeared to relate directly to the requirements management process. For 

example, see the following:

• About a week after the system was placed into production at a center, 

NASA found that it was making payments to its vendors 1 day earlier 

than required by Treasury regulations. This occurred because NASA 

thought that Treasury would warehouse its payments. If NASA had 

researched and documented the requirements associated with payment 

warehousing for cash management purposes, it would have known that 

Treasury does not warehouse payments such as these.

• About 3 weeks after the system went into production, NASA found that 

one of the payment processing tools was not working as required. It was 

unclear whether this was caused by a requirements defect or failure to 

properly test the functionality. A review of the requirements documents 

relating to this functionality provided a different description of the 

requirement than that included in the defect report.

• In early October 2002, NASA found that the accounting entries for 

certain advance transactions were incorrect. Properly documenting the 

requirements, developing a test case that ensured the requirements were 

met by the application, and executing that test case after the change was 

made should have detected this problem.

• After a patch was applied to the system, it was found that some code 

was duplicated, which caused an error. The apparent reason for the 

duplication of code was that manual adjustments were made to the code 

after the patch had been applied. 

By analyzing the root causes of its identified defects, NASA could 

determine whether the requirements management approach it has adopted 

sufficiently reduces its risks of the system not meeting cost, schedule, and 

functionality goals to acceptable levels. Root cause analysis would also 

help to quantify the risks inherent in the testing process that NASA has 

selected for the core financial module. Because, as discussed previously, its 

approach in both these areas includes elements that are not considered 
Page 36 GAO-03-507 NASA's IFMP

industry best practices, such metrics would be particularly important to 

NASA’s being reasonably assured that its processes will result in a system 

that meets its business needs.

Conclusions NASA has established the right goal for IFMP, and its ongoing 

implementation of several already-acquired system components, 

particularly the core financial module, may provide some improvements to 

NASA’s accounting data. However, implementation of these components 

will only partially address NASA’s information needs related to its complex 

space programs and contracts because NASA has deferred implementation 

of the system capabilities needed to provide this information and has not 

reengineered key business processes such as acquisition management. 

NASA’s long-standing weaknesses in this area have been central to our 

designation of NASA contract management as high risk. Moreover, NASA’s 

approach to acquiring and implementing IFMP components has and will 

continue to introduce risk and increase the chances that the agency will fall 

short of meeting its IFMP goal.

NASA faces serious near-term risks in implementing the commercial 

components that it has already acquired, including the core financial 

module. However, it is too far along in deploying these components to its 

centers, and relying on them to support operations, to stop and first acquire 

and then implement them properly. Instead, NASA will be forced to make 

the best of what it has acquired and implemented, meaning that NASA will 

have to stabilize the components while they are operational by identifying 

and correcting requirements defects and adequately testing the 

components to ensure that completed requirements are met. Such rework 

of already-deployed system components is a much more costly approach to 

implementing systems than adequately defining requirements and 

effectively testing system capabilities before they are deployed. However, 

NASA has left itself no other viable option.

In the longer term, NASA has an opportunity to avoid the mistakes it has 

made to date in acquiring system components, such as the core financial 

module, by first determining whether proposed components are the best 

solutions to meeting the agency’s corporate needs before it acquires them. 

It is critically important that NASA’s acquisition management strategy for 

future components includes a well-defined, risk-based methodology for 

understanding the dependencies among commercial component options 

before it acquires any additional components. Once these components are 

acquired, it is also critically important that NASA employ effective 
Page 37 GAO-03-507 NASA's IFMP

requirements management, testing, and performance metrics practices in 

implementing the components. To do less will increase the risk of IFMP 

becoming NASA’s third unsuccessful attempt to transform its financial 

management and business operations.

Recommendations for 

Executive Action

Given that NASA has already largely deployed and placed into production 

the IFMP commercial components acquired to date, we recommend that 

the NASA Administrator direct the Program Executive Officer for IFMP to 

focus near-term attention on stabilizing the operational effectiveness of 

these deployed commercial components. 

Specifically, to mitigate the risks associated with relying on alreadydeployed IFMP commercial components and to expeditiously stabilize 

these components’ operational capability and performance, we 

recommend that the Administrator direct the Program Executive Officer 

for IFMP to develop and implement a corrective action plan. At a minimum, 

this plan should provide for 

• identifying known and potential risks,

• assessing the severity of the risks on the basis of probability and impact,

• developing risk mitigation strategies,

• assigning accountability and responsibility for implementing these 

strategies,

• tracking progress in implementing these strategies, and

• reporting progress regularly and frequently to relevant congressional 

committees.

Additionally, this plan should provide for 

• developing and properly documenting requirements that are consistent, 

verifiable, and traceable, and that contain the necessary specificity to 

minimize requirement-related defects;

• conducting thorough regression testing before placing modified 

components into production; 
Page 38 GAO-03-507 NASA's IFMP

• implementing a metrics program that will identify and address the root 

causes of system defects;

• reengineering acquisition management processes, particularly with 

respect to the consistency and detail of budget and actual cost and 

schedule data provided by contractors; and

• engaging stakeholders—including program managers, cost estimators, 

and the Congress—in developing a complete and correct set of user 

requirements. 

To mitigate future risks, we further recommend that the Administrator 

require the Program Executive Officer for IFMP to complete the following 

actions before the acquisition of any additional IFMP components:

• establish and implement a methodology for commercial system 

component dependency analysis and decision making, and

• evaluate the suitability of already acquired, but not yet implemented, 

IFMP component products within the context of a component 

dependency analysis methodology.

Agency Comments and 

Our Evaluation

In its written comments on a draft of this report, NASA stated that it 

recognized and was addressing several of the concerns we raised and had 

already implemented some of the recommendations. NASA also stated that 

it disagreed with some of the issues in the report. NASA’s comments on our 

recommendations included the following:

• With regard to our recommendation to establish and implement a 

methodology for IFMP commercial system component dependency 

analysis and decision-making, NASA stated that it agreed with the 

importance of having an approach for acquiring additional IFMP 

components and believes that it has an effective strategy already in 

place. We disagree that NASA has an effective strategy because it did not 

provide convincing evidence to support its position that it is using 

methodologically based dependency analysis—a best practice for 

implementing commercial component-based systems—in acquiring 

IFMP. 

• Although NASA concurred with our recommendation regarding the 

need for a short-term plan to mitigate the risks currently associated with 
Page 39 GAO-03-507 NASA's IFMP

relying on already-deployed IFMP commercial components, it disagreed 

with many of our findings in the areas of (1) its efforts to involve users 

and reengineer its business process to ensure that the core financial 

module would meet the needs of program managers and cost estimators 

and (2) the need for detailed system requirements. We continue to 

believe that any effort that falls short of end-to-end business process 

reengineering will not result in a system that substantially improves the 

data available for contract oversight and decision-making and that 

documented, detailed requirements are necessary to reduce the risks of 

implementing the selected processes to acceptable levels. 

Overall, NASA disagreed with our findings related to three issues—

dependency analysis, user needs, and requirements and testing—which are 

addressed in the following sections. NASA also included several technical 

comments, which we have addressed as appropriate throughout the report.

Dependency Analysis In its written comments on our recommendation to establish and 

implement a methodology for IFMP commercial component dependency 

analysis and decision making, NASA stated that it agreed with the 

importance of having an approach for acquiring additional IFMP 

components but disagreed with our finding that it has not performed such 

dependency analysis to date in acquiring four IFMP commercial 

components. It also disagreed that it lacked a methodology to guide its 

analysis, and subsequent decision making, for future IFMP component 

acquisitions. According to NASA’s comments, the agency already has an 

effective strategy in place and it has followed this strategy to date in 

acquiring four IFMP commercial components. NASA described this 

strategy as consisting of two factors: following an enterprise resource 

planning (ERP) suite integration strategy (i.e, “best of suite” approach) and 

using an enterprise application integration framework and associated tool 

set for integrating current and future IFMP components.

NASA said that prior to receiving our draft report, it had provided us 

detailed documentation describing how it began performing its component 

dependency analysis before selecting the SAP ERP product for IFMP’s core 

financial module. The agency also noted that in a meeting following its 

receipt of our draft report, it had provided us with clear evidence that the 

program began performing dependency analysis before selecting the SAP 

product. Further, NASA’s comments stated that it was using an enterprise 

application integration tool to facilitate product integration and ease the 

associated complexities of integrating multiple products. NASA added that 
Page 40 GAO-03-507 NASA's IFMP

there were “perhaps miscommunications” and “some misunderstanding” of 

its approach, and opined that much of our concern about component 

dependency stems from our belief that NASA is not following a “best of 

suite” approach but rather a “best of breed” approach. To support its view 

that it is following a “best of suite” approach, NASA offered several 

statements, including that it is (1) on record in presentations and letters, 

including responses to congressional inquiries, that it is following “best of 

suite,” (2) developing business cases before implementing an IFMP 

module, (3) working with SAP to extend its ERP product, and (4) following 

a prioritization process when considering how to introduce functionality 

that the SAP ERP product does not provide.

We agree with several of NASA’s comments and disagree with others. 

Collectively, NASA’s comments do not change our finding and 

recommendation. Specifically, we do not question that NASA is using an 

enterprise application integration product and that this product facilitates 

integration of system components, both commercial and legacy 

components. Further, we do not challenge NASA’s statements regarding its 

representation in presentations and briefings that it is following a “best of 

suite” approach, its development of business cases, its interactions with 

SAP, and its use of a prioritization process.

However, we do challenge NASA’s assertion that much of our concern is 

based on our belief that it is following a “best of breed” approach. On the 

contrary, our finding and recommendation do not hinge on the distinctions 

between “best of breed” versus “best of suite,” despite evidence supporting 

our statements in the report that NASA is indeed following a best of breed 

approach. Such evidence includes (1) a report from a NASA contractor 

hired to provide an independent evaluation of IFMP stating that NASA is 

following a “best of breed” approach, (2) NASA’s acquisition of four 

separate commercial products from four vendors to satisfy the first five of 

nine planned IFMP system modules, and (3) NASA’s statement in its 

comments that additional products may be selected in the future. We fully 

appreciate that when implementing an ERP solution, other vendor 

products will likely be needed to fill gaps between agency requirements 

and the ERP product’s capabilities. Accordingly, we state in our report that 

proactive, methodologically based dependency analysis and evaluation is 

needed regardless of whether an agency is following a “best of breed” or a 

“best of suite” approach, although we appropriately recognize that this 

analysis and evaluation is more vital in a “best of breed” effort.
Page 41 GAO-03-507 NASA's IFMP

Instead, our finding and recommendation is based on whether NASA is 

following, and plans to follow, methodologically based dependency 

analysis—a best practice for implementing commercial component-based 

systems—in acquiring IFMP. In this regard, documentation that NASA 

provided us during the course of our review, and that it provided following 

its receipt of our draft report, both of which NASA cited in its comments, 

does not offer convincing evidence that NASA is following this best 

practice. For example, the documentation lacked product descriptions and 

comparisons as well as any analysis of integration requirements. Moreover, 

the Deputy Program Manager responsible for IFMP integration told us 

during the course of our review that proactive analysis of prospective IFMP 

components’ dependencies had not been performed and was not planned, 

and that NASA did not have a methodology for doing such analysis. The 

Deputy Program Manager for IFMP integration added, similar to NASA’s 

comments on a draft of this report, that the agency’s use of an enterprise 

application integration product and its associated tools will make 

integration easy and will negate the need for proactive dependency 

analysis. As noted above, we recognize that this product and tool set 

facilitates integration of multiple system components. However, it does not 

negate the need for dependency analysis and understanding to support 

informed decision-making before integration begins. As we state in our 

report, without such a proactive approach to acquiring system 

components, the risk of component product incompatibilities increases, as 

do the challenges and complexities that integration products and tools 

must overcome in integrating the products. 

User Needs NASA agreed that deployed and in-deployment modules do not yet meet all 

the needs of program managers. NASA indicated that this was the result of 

its “step-wise” approach in implementing the core financial module first 

and then integrating follow-on modules at a later date. As noted in our 

report, however, the deferral of many basic management functions has 

resulted in critical NASA programs, such as the ISS, using other systems to 

monitor contract cost and schedule performance. By not including the cost 

and schedule information needed by program managers in the core 

financial module, NASA risks operating with two sets of books—one that is 

used to report to management and the Congress and another that is used to 

manage NASA’s programs. NASA disagreed with our specific findings 

related to user needs in three key areas: 
Page 42 GAO-03-507 NASA's IFMP

• NASA believes that we have understated its accomplishments and the 

significance of the current capabilities delivered by the core financial 

module.

• NASA took issue with our assessment of the level of detail maintained in 

the core financial module, but did not comment specifically on our 

recommendation that the agency reengineer its acquisition management 

processes, particularly with respect to the consistency and detail of 

budgeted and actual cost and schedule data provided by contractors.

• NASA disagrees with our characterization that key users were not 

actively involved in the implementation of the core financial module or 

defining system requirements, although NASA indicates that better 

coordination was needed between program managers and the financial 

management community. 

First, we acknowledge again the significant effort that NASA has put into 

this project. Moving from 10 separate, incompatible systems to a single 

integrated financial management system is a major, complex undertaking. 

However, as we discussed previously in this report, the core financial 

module falls short of NASA’s own representation of the module’s 

capabilities, which was to provide program managers with the information 

required for day-to-day decision making. Specifically, it does not provide 

integrated cost and schedule performance information needed by program 

managers to oversee many of NASA’s largest and most complex contracts. 

In commenting on a draft of this report, NASA officials stated that it was 

never NASA’s intent to integrate schedule data with the initial core financial 

module implementation. However, IFMP planning documents (including its 

program plan and business case analysis), congressional testimony by 

NASA’s Administrator, and NASA’s own press releases clearly established 

an expectation that the core financial module would remedy many of 

NASA’s long-standing management challenges by providing program 

managers and other users with integrated financial and performance 

information. For example, according to his testimony before the House of 

Representatives Committee on Science on February 27, 2002, the NASA 

Administrator stated that while all components of IFMP are important, the 

successful completion of the core financial project will satisfy the Office of 

Management and Budget requirement that the financial and performance 

management systems supporting day-to-day operations are fully
Page 43 GAO-03-507 NASA's IFMP

integrated.20 NASA responded that it is currently in the process of engaging 

program managers and defining specific requirements related to needed 

cost and schedule performance data.

Second, we recognize that the commercial components NASA has selected 

for its core financial module are technologically capable of capturing and 

maintaining the detailed cost data required by program managers and cost 

estimators. However, the level of detailed cost data currently maintained in 

the core financial module depends on the level of detail provided by NASA’s 

contractors and the coding structure embedded in the core financial 

module. With respect to the level of detail provided by contractors, we 

reported that NASA has not reengineered its acquisition management 

processes to ensure that contractors are consistently providing the detailed 

cost data needed by program managers and cost estimators and 

recommended that NASA do so. 

NASA did not specifically address our recommendation but stated that it is 

incumbent upon program managers and cost estimators to learn and 

understand the capabilities of the new module and take advantage of them 

for their specific purposes. NASA’s comments also indicate that the data 

structure in the core financial module would be extended beyond the 

current legacy capabilities (i.e., the module will be able to record a greater 

level of detail) in fiscal year 2004. However, increasing the module’s 

capacity to store greater detail will not ensure that the information needed 

by program managers and cost estimators is requested and received from 

contractors and subsequently updated in the module. Although NASA 

commented that it would review its current project management process to 

ensure that its contractors provide the appropriate levels of cost data, we 

continue to believe that any effort that falls short of end-to-end business 

process reengineering will not result in a system that substantially 

improves the data available for contract oversight and decision making. 

Third, we acknowledge that the IFMP implementation team made an effort 

to include resource management staff from program management offices in 

various process teams. However, as we discussed previously in this report, 

no effort was made to include the cost estimating community in these 

20The Chief Financial Officers Act of 1990 and subsequent related financial management 

reform legislation, among other things, set expectations for agencies to develop and deploy 

more modern financial management systems, produce sound cost and operating 

performance information, and design results oriented reports on the government’s financial 

condition by integrating budget, accounting, and program information.
Page 44 GAO-03-507 NASA's IFMP

efforts. While program management office staff did participate, these 

efforts did not address the program cost and schedule needs of program 

managers or cost estimators. For example, the program management staff 

with whom we spoke, who worked on three of NASA’s largest programs 

(ISS, Space Shuttle, and Space Launch Initiative), viewed the core financial 

module as an “accounting system” that would be used by the accountants 

but was not necessarily going to change the way that program managers 

manage their programs. With this understanding, it is not surprising that 

the core financial module does not meet the needs of program managers or 

cost estimators. Implementing an integrated financial management system 

that is intended to change the way an organization does business is 

extremely complex and involves cultural, organizational, and process 

improvements. It also means making financial management an agency-wide 

priority. Our work at leading public and private sector organizations has 

shown that implementing a financial management system that meets the 

organization’s business needs takes more than just placing a handful of 

business or line management representatives on the implementation team. 

NASA’s approach has resulted in a core financial module that will be of 

limited value to program managers and cost estimators, who will continue 

to rely on other systems or ad hoc processes to get the data they need. As 

such, implementation of the core financial module to date continues to 

foster the concern that, at NASA, finance is not viewed as an integral part 

of NASA’s program management decision process.

Requirements and Testing NASA generally agreed that improvements were needed in its requirements 

management and testing processes and has stated that it has already begun 

to make improvements. For example, NASA recognized the need to 

implement a more rigorous regression testing methodology and stated that 

by October 2003 it would have an improved regression testing program. 

NASA also recognized that its process for tracing requirements and testing 

needed improvement and stated that it planned to implement improved 

capability and functionality for traceability over the next few months. 

However, according to NASA officials, they are following best practices for 

implementing an ERP solution and have “defined and implemented 

rigorous, closed-loop requirements and testing processes.” Further, 

regarding the applicability of requirements management standards, NASA 

did not agree that IEEE 830-1998 was applicable to the IFMP since it was an 

ERP implementation effort. NASA stated that specifying detailed 

requirements for already-developed software is high risk and that other 

leading industry experts have told them that NASA needed to change its 
Page 45 GAO-03-507 NASA's IFMP

processes to conform to the capabilities of the commercial software 

selected rather than attempt to change the software to conform to the 

existing NASA processes. We agree with NASA’s position that it needs to 

change its business processes to conform to the software; however, we do 

not agree with the agency’s position that detailed requirements are not 

needed. We continue to believe that NASA needs to properly configure the 

software based on detailed requirements in a manner that supports the 

business processes that have been adopted from the selected ERP solution. 

Because it has not done so, we continue to believe that NASA has not 

effectively implemented the types of disciplined processes necessary to 

reduce this project’s risks to acceptable levels. Acceptable levels refer to 

the fact that any systems acquisition effort, such as that being undertaken 

by NASA, will have some requirements-related defects. However, the goal 

is to reduce the risks and prevent significant requirements defects in order 

to limit the negative impact of these defects on cost, timeliness, and 

performance of the project.

During our review, we discussed with IFMP officials our concerns about 

the lack of documented, detailed system requirements for implementing 

the core financial module. In those meetings, we recognized that NASA’s 

approach for developing requirements was based on a business process 

model and did not disagree that this approach could be used to define how 

NASA would implement the necessary functionality. However, we continue 

to believe that once the business processes are defined and selected, 

documented, detailed requirements are necessary to reduce the risks of 

implementing the selected processes to acceptable levels. As NASA noted 

in its comments, its consultants also recommended that NASA needed to 

“determine the requirements while putting together the design process.” 

Therefore, guidance provided by the IEEE standard is applicable to the 

successful configuration and implementation of commercial software 

packages and is useful to help gauge the effectiveness of those efforts. 

As agreed with your offices, unless you announce its contents earlier, we 

will not distribute this report further until 30 days from its date. At that 

time, we will send copies to interested congressional committees, the 

NASA Administrator, and the Director of the Office of Management and 

Budget. We will make copies available to others upon request. In addition, 

the report will be available at no charge on the GAO Web site at 

http://www.gao.gov. 
Page 46 GAO-03-507 NASA's IFMP

If you or your staffs have any questions concerning this report, please 

contact Gregory D. Kutz at (202) 512-9505 or kutzg@gao.gov, Randolph C. 

Hite at (202) 512-6256 or hiter@gao.gov, Allen Li at (202) 512-4841 or 

lia@gao.gov, or Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov. Key 

contributors to this report are acknowledged in appendix III.

Gregory D. Kutz

Director

Financial Management and Assurance

Allen Li

Director

Acquisition and Sourcing Management

Randolph C. Hite

Director

Information Technology Architecture

and Systems Issues

Keith A. Rhodes

Chief Technologist

Applied Research and Methods
Page 47 GAO-03-507 NASA's IFMP

Appendix I

Objectives, Scope, and Methodology AAppenednxidexiIs

To determine whether the National Aeronautics and Space Administration 

(NASA) is effectively managing the Integrated Financial Management 

Program (IFMP) acquisition, we reviewed relevant program-level 

acquisition management documentation to obtain an understanding of 

NASA’s plans and strategy, including the program overview, program- and 

project-level management plans, the acquisition strategy, implementation 

and integration plans, briefing materials on the agency’s plans to develop 

an information architecture, and a report on IFMP lessons learned by 

NASA’s consultant, Gartner, Inc. We also interviewed various program 

officials, including the Program Executive Officer for IFMP, the IFMP 

Program Director, the IFMP Deputy Program Director, the Core Financial 

Project Manager, the Integration Office Deputy Program Manager, and the 

Chief Information Officer to clarify our understanding of the agency’s 

strategy and obtain current information on the status of the agency’s 

efforts. Specifically, we inquired as to NASA’s basis for selecting alreadyacquired commercial products and its plans for selecting future modules. 

We then compared NASA’s plans and activities to relevant best practices, 

Office of Management and Budget (OMB) requirements, federal guidance, 

and NASA procedures and guidance. 

We had also intended to include our assessment of a key element of NASA’s 

acquisition strategy—whether NASA was acquiring IFMP components in 

the context of an enterprise architecture—in this report. However, because 

NASA did not provide the data needed to complete our assessment until 

after the conclusion of our fieldwork, we plan to address NASA’s enterprise 

architecture in a future product.

To determine whether NASA had adequately considered the information 

needs of key users in implementing the core financial module of IFMP, we 

reviewed IFMP documents discussing the business case and properties of 

the core financial module and spoke with IFMP implementers at Marshall 

Space Flight Center—the lead center on this project—and NASA 

headquarters. To determine whether the data requirements, as established 

by the IFMP implementers, would address NASA’s known problems with 

cost control and cost tracking, we spoke with program managers involved 

in three of NASA’s largest programs and other NASA program and business 

management staff at three centers—Marshall Space Flight Center, Johnson 

Space Center, and Glenn Research Center. We also reviewed prior work on 

NASA’s cost problems, including the report by the International Space 

Station Management and Cost Evaluation Task Force, which reviewed the 

recent cost growth in that program and identified causes and necessary 

actions. 
Appendix I

Objectives, Scope, and Methodology

Page 48 GAO-03-507 NASA's IFMP

In addition to speaking with program managers and their staffs, we spoke 

with cost estimators at the three centers mentioned above as well as 

Langley Research Center and NASA headquarters. We also spoke with 

center staff who oversee and support earned value management for 

programs that use that tool, and with the congressional staffs of NASA’s 

authorization committees. We asked them about the extent to which they 

had been asked by IFMP implementers for input on their data needs, the 

extent to which they had been involved in IFMP’s design and 

implementation, and whether they had been briefed by IFMP implementers 

on the capabilities of the core financial module. 

To determine what kind of cost information program managers use to 

oversee their programs, we reviewed selected large, cost-type contracts for 

NASA’s three largest space flight programs—the International Space 

Station, the Space Shuttle, and the Space Launch Initiative project 

(intended to develop technologies for the next generation replacement for 

the Space Shuttle.) For all three of these programs, cost control and cost 

tracking have been issues of concern. The three programs together involve 

most of NASA’s work in the human space flight area, which accounts for 

most of the agency’s spending. These programs range from relatively new 

(Space Launch Initiative) to quite mature (Space Shuttle) programs and 

require the procurement of a wide range of goods and services. Each of 

these programs is being run at multiple centers, involves the work of 

multiple contractors, and uses cost-type contracts1

 that run for multiple 

years. 

For the contracts we selected, we spoke to responsible personnel about 

how costs are tracked and monitored, including the level of detail provided 

by contractors, the format in which cost data are available, and how 

contract cost data reporting requirements are developed. We also obtained 

and reviewed copies of contractor financial management reports and cost 

and schedule performance reports that we compared with contract or 

program work breakdown structures, as well as contract cost data 

reporting requirements and statements of work. We analyzed and discussed 

with agency officials how all these documents and reports related to each 

other and to the work breakdown structure. We also discussed how the 

1

Cost-reimbursement contracts are often the most appropriate type for developmental items 

or those for which the exact price of the goods or services being purchased cannot be 

definitely known prior to contract award. This type of procurement instrument places 

greater risk on the government than contracts based on firm fixed prices. 
Appendix I

Objectives, Scope, and Methodology

Page 49 GAO-03-507 NASA's IFMP

reported information was used by the programs and the extent to which 

that information would be included in the core financial module. We did 

not, however, evaluate whether the information currently received from 

contractors, or represented as needed by program managers and cost 

estimators, was adequate for management purposes.

IFMP’s core financial module is intended to address known problems with 

NASA’s program cost accounting and with its financial reporting. We did 

not, however, review how the core financial module will address the 

agency’s financial reporting issues, including property accounting and 

budgetary information, and whether the module will reduce the time and 

resources needed to close the books each accounting period and reduce 

the number of postclosing adjusting entries. We plan to review and report 

on these issues at a later date.

To assess whether NASA had established and implemented an effective 

requirements management process to support implementation of the core 

financial module, we wanted to determine whether NASA had effectively 

implemented (1) the disciplined processes that can reduce project risks to 

acceptable levels for its requirements management process and (2) the 

types of metrics to identify and quantify any weaknesses in its 

requirements management process. To accomplish these objectives, we 

• reviewed various requirements documents produced for the core 

financial module project, including the over 500 contract requirements 

used to acquire the SAP software;

• performed an in-depth review and analysis of the 132 requirements, 

which represent about 22 percent of the contract requirements, 

developed for the “Managing Accounts Payable” process to determine 

whether they had the attributes normally associated with good 

requirements and whether these requirements traced between the 

various requirements documents;

• reviewed NASA’s procedures for defining its requirements management 

framework and compared these procedures to its current practices;

• reviewed business processes, problem defect reports, test conditions, 

test cases, and test execution logs contained in Accenture’s Method 

Delivery Management system—NASA’s project management tool; and
Appendix I

Objectives, Scope, and Methodology

Page 50 GAO-03-507 NASA's IFMP

• reviewed guidance published by the Institute of Electrical and 

Electronics Engineers (IEEE), Software Engineering Institute (SEI), and 

publications by experts to determine the attributes that should be used 

for developing good requirements and for identifying and quantifying 

performance metrics. 

To augment these document reviews and analyses, we interviewed officials 

from NASA headquarters, Marshall Space Flight Center, and NASA’s 

independent verification and validation contractor—Titan Systems 

Corporation. In addition, we discussed with NASA officials the processes 

they used to measure the effectiveness of their requirements management 

process and compared NASA’s process to those used by disciplined 

organizations. In order to determine the processes that can be used to help 

an organization understand the effectiveness of its processes, we used 

information from IEEE, SEI, and subject matter experts.

We conducted our work at NASA headquarters in Washington, D.C.; 

Marshall Space Flight Center in Huntsville, Alabama; Glenn Research 

Center in Cleveland, Ohio; Johnson Space Center in Houston, Texas; 

Langley Research Center in Hampton Roads, Virginia; and Goddard Space 

Flight Center in Greenbelt, Maryland. We received written comments on a 

draft of this report from the NASA Deputy Administrator. These comments 

are addressed in the “Agency Comments and Our Evaluation” section of 

this report and are reprinted in appendix II. We performed our work from 

April 2002 through February 2003 in accordance with generally accepted 

government auditing standards. 
Page 51 GAO-03-507 NASA's IFMP

Appendix II

Comments from the National Aeronautics and 

Space Administration ApendxiI

See comment 1.

Note: GAO comments 

supplementing those in the 

report text appear at the end 

of this appendix.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 52 GAO-03-507 NASA's IFMP

See comment 1.

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 53 GAO-03-507 NASA's IFMP

See comment 1.

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 54 GAO-03-507 NASA's IFMP

See comment 2.

See comment 3.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 55 GAO-03-507 NASA's IFMP

See comment 4.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 56 GAO-03-507 NASA's IFMP

See comment 1. 
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 57 GAO-03-507 NASA's IFMP

See comment 1.

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 58 GAO-03-507 NASA's IFMP

See comment 5.

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 59 GAO-03-507 NASA's IFMP
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 60 GAO-03-507 NASA's IFMP

See comment 1.

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 61 GAO-03-507 NASA's IFMP

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 62 GAO-03-507 NASA's IFMP

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 63 GAO-03-507 NASA's IFMP

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 64 GAO-03-507 NASA's IFMP

See comment 6.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 65 GAO-03-507 NASA's IFMP

See comment 7.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 66 GAO-03-507 NASA's IFMP

See comment 8.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 67 GAO-03-507 NASA's IFMP

See comment 1.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 68 GAO-03-507 NASA's IFMP

See comment 9.
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 69 GAO-03-507 NASA's IFMP
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 70 GAO-03-507 NASA's IFMP

The following are GAO’s comments on the National Aeronautics and Space 

Administration’s (NASA) letter dated March 25, 2003.

GAO Comments 1. See the “Agency Comments and Our Evaluation” section of this report. 

2. Although NASA indicates that it has extended the SAP suite to include 

Business Warehouse (for reporting) and Asset Management, it did not 

provide us any documentation to support these selections during the 

course of our fieldwork.

3. We did not assess the deployment and operation of the three modules 

to which NASA referred. We understand that the NASA Inspector 

General has recently begun a review of the Travel Management module.

4. The scope of our work did not include a review of the Integrated 

Financial Management Program (IFMP) budget or schedule. We plan to 

address these issues in a future product.

5. As stated in this report, although the core financial module will provide 

some improvement to NASA’s current accounting system environment, 

certain system capabilities have been deferred and will not be available 

when the system becomes an agencywide operation in June. Without an 

effort to reengineer NASA’s acquisition management processes, it is 

unlikely that detailed cost information will be available to meet the 

needs of program managers, cost estimators, and the Congress. Thus, 

NASA’s assertion that it will be able to transition to “full cost 

management practices” in June of 2003 is questionable.

6. NASA’s comments refer to the “Accounts Payable” process illustrated in 

figure 3 of the report. While we did not verify or evaluate the extent to 

which the additional requirements to which NASA refers in its response 

were established or validated, the accounts payable requirements, as 

described, do not provide for quantitative evaluation to determine 

whether the system meets NASA’s needs. Furthermore, the additional 

requirements did not provide the needed clarification for the 

requirement cited in our report related to the ability to query 

information. Moreover, given that NASA added new requirements for 

reporting, it is unclear whether the existing accounts payable 

requirement was to provide some other query functionality not 

included in the other general reporting requirements. 
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 71 GAO-03-507 NASA's IFMP

7. As noted in our report, requirements provide the foundation for system 

testing. In meetings with NASA, we acknowledged that the “processcentric” approach that the agency adopted was an acceptable 

methodology for understanding how the processes supported by the 

selected enterprise resource planning (ERP) solution could be 

implemented at NASA. However, we believe that this approach still 

requires the development and documentation of the necessary 

requirements to fully understand the functionality to be provided by a 

given process. Without such requirements, a disciplined testing process 

is very difficult to implement since requirements are a fundamental 

attribute of an effective testing process. As discussed in our report, we 

continue to believe that the lack of an effective requirements 

management process hampered NASA’s testing efforts since significant 

defects in the production system should have been detected before 

system implementation. 

Although NASA stated that it will repeat its testing efforts at each 

center implementing the system, without adequately documenting its 

requirements and ensuring that the testing process adequately tests 

those requirements, it does not have reasonable assurance that the 

testing process will identify significant defects before a center is 

converted to the production system. For example, NASA stated that it 

had developed over 1,700 test conditions. However, it was not until the 

system was placed into production that NASA identified several 

significant weaknesses, as discussed in our report. We continue to 

believe that NASA will not have reasonable assurance that it has 

adequately tested the system until it (1) documents its requirements 

and (2) develops test conditions that fully test those requirements.

8. As noted in our report, discussions with IFMP officials recognized that 

a test case was not properly developed to test large contracts that 

contained over 200 line items—a common occurrence according to 

IFMP officials—and that this was an oversight in their testing process. 

Had NASA developed and documented a detailed requirement for this 

functionality and then mapped its test conditions against those 

requirements, it would have recognized that it had not developed a test 

condition to properly demonstrate and test the functionality prior to 

the system going into production. Properly processing these types of 

payments may have enabled NASA to reduce the impact of the payment 

backlog. 
Appendix II

Comments from the National Aeronautics 

and Space Administration

Page 72 GAO-03-507 NASA's IFMP

9. As noted in our report, NASA does not have metrics that properly 

analyze the cause of the defects so that it can improve its processes. 

For example, although NASA was able to show the number of defects 

that were related to subsequent implementation, referred to as the 

second wave, it did not have information that could be used to analyze 

whether these defects were caused by, for example, requirements or 

testing problems or by not adequately correcting prior defects. 

Therefore, although NASA states that it has a structured testing and 

problem analysis process in place, we continue to believe that the 

examples provided in NASA’s comments do not provide the data 

necessary to identify the causes of defects or assess the effectiveness 

of processes such as the requirements management and testing 

processes. As noted in our report, these types of data can be used to 

prevent or anticipate problems before they occur, resulting in less 

rework.
Page 73 GAO-03-507 NASA's IFMP

Appendix III

GAO Contacts and Staff Acknowledgments ApendxiI

GAO Contacts Diane Handley, (404) 679-1986

Cynthia Jackson, (202) 512-5086

Chris Martin, (202) 512-9481

Acknowledgments Staff members who made key contributions to this report were Erin Baker, 

Molly Boyle, Felicia Brooks, Francine DelVecchio, Michael Giannone, 

Jamie Haynes, Richard J. Herley, Kristi Karls, LaTonya Miller, Maria Storts, 

Eric Trout, Teresa Tucker, Carrie Wilson, and Jenniffer Wilson.

(120148)
GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal government 

for the American people. GAO examines the use of public funds; evaluates federal 

programs and policies; and provides analyses, recommendations, and other 

assistance to help Congress make informed oversight, policy, and funding 

decisions. GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail this 

list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO 

Mailing Lists” under “Order GAO Products” heading.

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check 

or money order should be made out to the Superintendent of Documents. GAO 

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single 

address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00