Business Modernization: NASA Must Consider Agencywide Needs to Reap the Full Benefits of Its Enterprise Management System Modernization Effort
Published: 2007-07-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1990, GAO has designated the National Aeronautics and Space Administration's (NASA) contract management as an area of high risk in part because it lacked modern systems to provide accurate and reliable information on contract spending. In April 2000, NASA began a system modernization effort, known as the Integrated Enterprise Management Program (IEMP). When GAO last reported on the status of IEMP in September 2005, NASA had begun to implement disciplined processes needed to manage IEMP, but had yet to implement other best practices such as adopting business processes that improve information on contract spending. This GAO report addresses (1) actions taken by NASA to effectively implement the disciplined processes needed to manage IEMP and (2) the extent to which NASA has considered the strategic issues associated with developing a concept of operations and defining standard business processes. GAO interviewed NASA officials and obtained and analyzed documentation relevant to the issues.
section_title: What GAO Found
paragraphs: Since GAO last reported on NASA's IEMP efforts, NASA implemented its IEMP contract management module and upgraded the software used for its core financial module. NASA has also taken steps to improve its processes for managing IEMP--including implementing improved requirements management and testing processes, enhancing its performance metrics related to tracking system defects, and developing an IEMP risk mitigation strategy. Further, NASA has developed quantitative entry and exit criteria for moving from one phase of an IEMP project to another--a recognized industry best practice. However, NASA has not yet addressed weaknesses in the areas of requirements development and project scheduling, which ultimately caused the agency to assume a greater risk that it would not identify significant system defects prior to implementation of the core financial upgrade. Despite these difficulties, NASA financial managers have stated that the core financial upgrade is now functioning as expected for most transactions. As of the end of GAO's audit work in May 2007, NASA was working to correct a number of system errors, including posting errors for certain types of transactions. Because NASA was still working to stabilize the system, GAO was unable to determine the significance of these weaknesses. Further, NASA has not yet fully considered higher-level strategic issues associated with developing an agencywide concept of operations and defining standard business processes. With a planned investment of over $800 million for IEMP, NASA must immediately and effectively address these strategic building blocks if IEMP is to successfully address long-standing management challenges--including overseeing contractor performance and properly accounting for NASA's property, plant, and equipment. NASA officials stated that they have begun developing a concept of operations to describe how all of its business processes should be carried out. According to NASA officials, they expect to complete the concept of operations by the summer of 2008. Ideally, a concept of operations should be completed before system development begins so that it can serve as a foundation for system planning and requirements development. Nonetheless, while NASA's IEMP efforts are already well under way, the completion of such a document remains essential for guiding the development of the remaining IEMP modules as well as any future upgrades. As part of developing a concept of operations, NASA should also define standard business processes that are supported by its IEMP software. NASA needs to ensure that its business processes and the information that flows from those processes support the enterprise's needs. Efforts that primarily focus on the parochial needs of a specific organizational unit, such as accounting, do not provide reasonable assurance that NASA's agencywide management information needs are addressed.

section_title: Letter
paragraphs: As we and others have reported in the past, the National Aeronautics and  Space Administration (NASA) has fundamental problems with its financial  management operations that undermine its ability to effectively manage its  major programs and to report externally on its financial operations. Since  1990, we have designated NASA’s contract management as an area of high  risk, in large part because NASA has lacked a modern financial  management system to provide accurate and reliable information on  contract spending. In April 2000, NASA began a program expected to  address many of its financial and management challenges. This program,  now known as the Integrated Enterprise Management Program (IEMP),  has been focused on implementing a new integrated financial management  system. Specifically, NASA has invested in an enterprise resource planning  (ERP) solution—a business system that is intended to meet the  information needs of both internal and external customers and to promote  standardization and integration of business processes and systems across  the agency. NASA plans to complete IEMP by 2009 for a total cost of over  $800 million.
In April and November 2003—3 years into the IEMP implementation effort  and with significant investment already made in the program—we issued a  series of four reports that detailed weaknesses in NASA’s acquisition and  implementation strategy for IEMP. Specifically, we reported that NASA  had not followed key best practices for acquiring and implementing IEMP  and, therefore, was at risk of making a substantial investment in a financial  management system that would fall far short of its stated goal of providing  meaningful, reliable, and timely information to support effective day-to- day program management and external financial reporting.
NASA is not alone in its struggle to successfully implement an integrated  financial management system. Billions of dollars have been spent  governmentwide to modernize financial management systems that have  often exceeded budgeted cost, resulted in delays in delivery dates, and did  not provide the anticipated functionality when implemented. In our  previous report on government financial management systems failures,  we provided our views on actions that can be taken to help improve the  management and control of agency financial management system  modernization efforts. Based on industry best practices, we identified  three key practices, or building blocks, that are needed to ensure a solid  foundation for agencies’ successful system implementation efforts:  (1) developing a concept of operations that would define how an agency  will carry out its day-to-day operations in order to meet mission needs;   (2) defining standard business processes that result in streamlined  operations, rather than simply automating old ways of doing business; and  (3) effectively implementing the disciplined processes necessary to  manage the project.
When we last reported on NASA’s IEMP effort, in September 2005, NASA  had begun to implement a number of recommendations from our earlier  reports, including taking steps toward implementing the disciplined  processes necessary to manage IEMP—one of the key building blocks  discussed above that is needed to ensure a solid foundation for agencies’  system implementation efforts. For example, we reported that NASA had  implemented new requirements management and testing processes and  had developed metrics to evaluate the effectiveness of its system  implementation processes. However, at that time, the agency had not  implemented our recommendation to properly define and document  system requirements for already-deployed IEMP modules, including the  core financial module. This was important not only because it would affect  the way the core financial module functions but also because it would  affect NASA’s ability to implement future upgrades and other modules  expected to interface with the core financial module. In addition, we  reported that additional enhancements could be made in the area of  regression testing and performance metrics. Finally, NASA had yet to  reengineer its business processes so that the commercial off-the-shelf  (COTS) software products it had selected for IEMP could support these  processes.
Since we last reported, in September 2005, on NASA’s efforts to implement  IEMP, NASA implemented its IEMP contract management module and  upgraded the version of the COTS software that is used for the core  financial module of IEMP—which was expected to enhance the  functionality of the core financial module. Because of your continued  interest in NASA financial management, you asked us to provide periodic  updates on the status of NASA’s financial management improvement  efforts—including its effort to implement IEMP. Specifically, this report  addresses (1) actions taken by NASA to effectively implement the  disciplined processes necessary to manage IEMP and (2) the extent to  which NASA has considered the higher-level strategic issues associated  with developing an agencywide concept of operations and defining  standard business processes—two key building blocks critical to NASA’s  ability to successfully implementing its planned financial management  system.
To achieve these objectives, we interviewed the appropriate NASA  officials and obtained and analyzed documentation supporting the process  improvements that were cited by NASA. We performed our work from  January 2006 through June 2007 in accordance with U.S. generally  accepted government auditing standards. Details on our scope and  methodology are included in appendix I.
subsections: section_title: Background
paragraphs: For more than a decade, we have identified weak contract management  and the lack of reliable financial and performance information as posing  significant challenges to NASA’s ability to effectively run its largest and  most costly programs. While NASA has made some progress in addressing  its contract management weaknesses through improved management  controls and evaluation of its procurement activities, NASA has struggled  to implement a modern, integrated financial management system. NASA  made two efforts in the past to improve its financial management  processes and develop a supporting system intended to produce the kind  of accurate and reliable information needed to manage its projects and  programs and produce timely, reliable financial information for external  reporting purposes. However, both of these efforts were eventually  abandoned after a total of 12 years and a reported $180 million in  spending.
subsections: section_title: IEMP Implementation Status
paragraphs: In April 2000, NASA began its third attempt at modernizing its financial  management processes and systems. With its current financial  management system effort, known as IEMP, NASA has invested in an ERP  solution that is intended to meet the information needs of both internal  and external customers and to promote standardization and integration of  business processes and systems across the agency. NASA plans to  complete IEMP by 2009 for a total cost of over $800 million.
As of March 2007, NASA had deployed the following nine IEMP functional  components: core financial, Travel Manager, ERASMUS, resume  management, position description management, budget formulation,  Agency Labor Distribution System, Project Management Information  Improvement, and contract management. Early in fiscal year 2007, NASA  also implemented an updated version of the core financial software, which  includes several critical enhancements to the previous core financial  software. According to NASA, the core financial upgrade provided the  opportunity for it to leverage the best practices inherent in the new  version and allowed it to redesign or enhance business processes. NASA  updated its core financial system in order to improve compliance with  Federal Financial Management System Requirements, Federal Accounting  Standards, and the Federal Financial Management Improvement Act, and  to respond to GAO recommendations. According to NASA, the software  upgrade has enabled it to implement critical process changes related to  financial tracking and reporting, support the goal of achieving financial  management integrity, and provide better project management  information. NASA claims that the updated software has also provided  other enhancements, which should contribute to NASA’s goals of  achieving a clean audit opinion and achieving a “Green” rating on the  President’s Management Agenda scorecard for “improved financial  performance.” Other IEMP modules that NASA plans to implement in the  future include aircraft management and asset management.
section_title: Prior Reporting on IEMP
paragraphs: As discussed previously, we issued a series of four reports in April and  November 2003 that detailed weaknesses in NASA’s acquisition and  implementation strategy for IEMP in general and the core financial module  in particular. The core financial module, which utilizes SAP software and  is considered the backbone of IEMP, was implemented in June 2003.  Because NASA did not follow key best practices or disciplined processes  for acquiring and implementing IEMP, we reported that NASA had made a  substantial investment in a financial management system that fell far short  of its stated goal of providing meaningful, reliable, and timely information  to support effective day-to-day program management and external  financial reporting. We noted problems in the areas of requirements  development, requirements management, testing, performance metrics,  risk management, and business process reengineering.
Neither program managers nor cost estimators were involved in the  process of defining requirements for the core financial module. As a  result, the module was not designed to maintain the level of detailed  cost information needed by program managers to perform contract  oversight and by cost estimators to develop reliable cost estimates.
The requirements management methodology and tools used to  implement the core financial module did not result in requirements that  were consistent, verifiable, and traceable or that contained enough  specificity to minimize requirement-related defects. Because NASA had  not effectively implemented disciplined requirements management  processes, we reported that it had increased the risk that it would not  be able to effectively identify and manage the detailed system  requirements necessary to properly acquire, implement, and test the  core financial module.
NASA’s ability to effectively test the core financial module was limited  because of the lack of complete and specific requirements. Industry  best practices, as well as NASA’s own system planning documents,  indicated that detailed system requirements should be documented to  serve as the basis for effective system testing. Because the link  between these two key processes was not maintained, NASA had little  assurance that all requirements were properly tested.
NASA also did not effectively capture the type of metrics that could  have helped the agency understand the effectiveness of its IEMP  management processes. For example, NASA did not employ metrics to  help it identify and quantify weaknesses in its requirements  management processes. Because of its lack of performance metrics,  NASA was unable to understand (1) its capabilities to manage IEMP  projects; (2) how its process problems affected cost, schedule, and  performance objectives; and (3) the corrective actions needed to  reduce the risks associated with the problems identified.
NASA did not consistently identify known and potential risks for the  core financial module. Risk management processes are needed to  ensure that a project’s risk is kept at an acceptable level by taking  actions to mitigate risk before it endangers the project’s success.
NASA did not use the implementation of IEMP to fundamentally  change the way it did business. Instead of reengineering its business  processes, NASA automated many of its existing ineffective business  processes. First, NASA did not design the system to accommodate the  information needed to adequately oversee its contracts and programs  and to prepare credible cost estimates. Second, NASA did not  reengineer its contractor cost reporting processes and therefore, did  not always obtain sufficient contract cost information needed by  program managers to oversee contracts and needed by financial  managers for external financial reporting.
When we last reported on NASA’s IEMP effort, in September 2005, NASA  had begun to implement a number of recommendations from our earlier  reports—including steps toward implementing the disciplined processes  necessary to manage IEMP. For example, we reported that NASA had  engaged program managers to identify program management needs,  implemented new requirements management and testing processes, and  developed metrics to evaluate the effectiveness of its system  implementation processes. However, at that time, the agency had not  implemented several of our other recommendations, including the  following:    Properly define and document system requirements for already- deployed IFMP modules, including the core financial module. This is  important not only because it would affect the way the core financial  module functions but also because it would affect NASA’s ability to  implement future upgrades and other modules expected to interface  with the core financial module.
Enhance regression testing processes and performance metrics.
Develop a risk mitigation plan.
Reengineer its business processes so that the commercial off-the-shelf  software products selected for IEMP could support these processes.
At the time of our last report, NASA was making plans to reengineer some  of its business processes. However, because the agency was in the very  early planning stage of implementing this recommendation, the details for  how NASA would accomplish its objectives were still vague. Overall, our  September 2005 report concluded that it was not possible to assess  whether NASA’s plans would accomplish its stated goal of enhancing the  core financial module to provide better project management information  for decision-making purposes.
section_title: Progress Made in Developing Disciplined Project Management Processes, but Some Problems Remain
paragraphs: Since September 2005, when we last reported on NASA IEMP  implementation efforts, NASA has implemented some of the disciplined  processes needed to manage IEMP. Specifically, NASA has, as we  previously recommended, implemented more effective requirements  management and testing processes, improved its performance metrics  program related to tracking system defects, and developed an IEMP risk  management strategy. In addition, NASA has developed quantitative entry  and exit criteria for moving from one phase of an IEMP project to  another—a recognized industry best practice. However, weaknesses in the  areas of requirements development and project scheduling have  undermined some of the progress made in other key areas. As a result,  NASA struggled to complete required systems testing and deliver the  agency’s core financial upgrade. Ultimately, through the heroic efforts of  the core financial upgrade team, NASA delivered the upgrade within about  2 weeks of the October 30, 2006, planned completion date. According to  NASA officials, the system is functioning as expected for most  transactions. However, until the end of March 2007, the upgrade was in a  “stabilization” phase as NASA worked on correcting a number of system  errors, including posting errors for certain types of transactions. Because  the upgrade was still quite new and NASA was continuing to stabilize the  system, we were unable to determine the significance of these  weaknesses.
subsections: section_title: Improvements Made to NASA Requirements Management and Testing Processes
paragraphs: Since our September 2005 report, NASA has used its new requirements  management process—which documents sufficiently detailed  requirements that are traceable from the highest (most general) level to  the lowest (most detailed) level in NASA’s requirements management  system—for both the core financial upgrade and the contract management  module. For example, we selected several requirements for both the core  financial module and the contract management module and validated that  the requirements management process (1) clearly linked related  requirements consistent with industry standards and (2) contained the  information necessary to understand how each requirement should be  implemented and tested in a quantitative manner.
Because NASA developed and is now using a disciplined requirements  management process, it has the quantitative information necessary to  support disciplined testing processes. NASA’s disciplined testing  processes include (1) documentation of the scenarios that need to be  tested to obtain adequate test coverage, (2) requirements that are traced to  the test cases to ensure that all requirements are tested, (3) instructions  and other guidance for the testers, and (4) an effective regression testing  program. Although NASA had disciplined requirements management and  testing processes in place for the implementation of both the contract  management module and the core financial upgrade, difficulties related to  requirements development and project scheduling, discussed later, forced  NASA to compress the testing phase of its core financial upgrade  implementation. As a result, according to NASA officials, completion of  testing for the core financial upgrade required an extraordinary effort on  the part of NASA’s implementation team.
section_title: NASA Has Implemented an Effective Metrics Program and Risk Management Strategy
paragraphs: Since we last reported, in September 2005, NASA has also enhanced its  metrics measurement program, which is used to evaluate the effectiveness  of its project management processes by identifying the causes of process  defects. Understanding the cause of a defect is critical to evaluating the  effectiveness of an organization’s project management processes, such as  requirements management and testing. For example, if a significant  number of defects are caused by inadequate requirements definition, then  the organization knows that corrective actions are needed to improve the  requirements definition process. When we last reported, NASA had made  progress in this important area by collecting information on the causes of  system defects it identified in its regression testing efforts but was not  collecting similar information on defects identified by users and lacked a  formal process for fully analyzing the data related to system defects by  identifying the trends associated with them. Since that time, NASA has  developed additional metrics to track and analyze such things as the  number of changes made to requirements while a system is under  development. In addition, NASA has developed processes for tracking and  analyzing defects identified by IEMP users. For example, since  implementation of the core financial upgrade, NASA has maintained  spreadsheets showing specific information on each service  request submitted by users, including the type of defect involved and the  status of the request.
Finally, NASA has also developed a comprehensive risk management  strategy. Specifically, NASA now has an IEMP Risk Management Plan that  outlines the standard processes and techniques for identifying, analyzing,  planning, tracking, and controlling risks as well as defining the roles and  responsibilities for each level of project risk management. In applying  these techniques to the core financial upgrade, NASA officials documented  the risks that they identified for the project, as well as their mitigation  strategies, likelihood, consequence, and criticality. According to NASA  officials, their risk management process worked well and was one of the  key reasons for the success of the core financial upgrade. For example,  using the metrics information discussed previously, NASA officials said  they were able to assess the risks of changing requirements late in the  project and then mitigate those risks by performing additional testing.
section_title: NASA Uses a Recognized Industry Best Practice to Move from One Project Phase to the Next
paragraphs: In addition to the disciplined processes discussed above, NASA has also  taken action to establish the use of quantitative entry and exit criteria to  move from one phase of an IEMP project to another. The use of such  criteria is considered an industry best practice. Entry criteria are the  minimum essential items considered necessary to enter into a given  project phase, while exit criteria are the minimum essential items  necessary to consider a given project phase successfully completed. For  example, the NASA entry criterion for moving into the regression testing  phase requires that all remaining significant defects from the integration  testing phase be resolved and successfully retested before regression  testing can begin. NASA demonstrated application of this criterion when it  implemented the contract management module. About 3 weeks before the  scheduled start date of regression testing, the project had not yet  successfully completed all test scenarios, and several significant defects  had not been fully resolved. In addition, a series of critical corrections  from the software vendor had not yet been delivered, and the project team  agreed that there would not be adequate time to test the corrections prior  to beginning the scheduled regression testing. Consequently, the team  decided to push back the scheduled date for the contract management  module to begin operating.
For the core financial upgrade, NASA officials said that they used entry  and exit criteria as one of the management tools to determine whether the  project should move forward. However, rather than adopt a “hard stop”  approach when criteria were not met, they used the criteria to make sure  that all appropriate factors were considered before moving forward,  including the risks of not meeting certain criteria. Any instances in which  the project team thought exceptions to the criteria were warranted were  ultimately reviewed and decided on by higher levels of NASA  management, which helped ensure that such decisions were adequately  considered.
section_title: Improved Requirements Development and Project Scheduling Needed
paragraphs: Weaknesses in the areas of requirements development and project  scheduling offset some of the benefits associated with NASA’s improved  requirements management and testing processes—causing NASA to  assume a greater risk that it would not identify significant system defects  prior to implementation. Weaknesses in requirements development and  project scheduling processes resulted in NASA having to compress the  testing phase of its core financial upgrade implementation. As a result,  according to NASA officials, NASA’s ability to complete testing for the  core financial upgrade within the planned implementation time frames  ultimately depended on the extraordinary effort put forth by NASA’s  project implementation team.
Because of weaknesses in NASA’s requirements development process, it  did not have reasonable assurance that it identified all appropriate  requirements for the core financial upgrade when the project began.  Consequently, NASA continued making changes to the requirements very  late in the project’s development, resulting in increased risks, delays, and a  compressed testing schedule. Improperly defined or incomplete  requirements have commonly been identified as a root cause of system  failure. Although NASA made a concerted effort, as part of its core  financial system upgrade, to involve program managers and other key  stakeholders in the requirements development process, it did not follow  standard industry practices for identifying and documenting user  requirements.
According to the Software Engineering Institute (SEI), to help ensure that  critical requirements are identified, an organization should have a well- documented, disciplined requirements development process that, among  other things, (1) defines how customer needs will be elicited, developed,  and validated; (2) specifies how to identify and ensure involvement of  relevant stakeholders; and (3) ensures that people involved in the  requirements development process are adequately trained in such topics  as requirements definition and analysis. In addition, it is critical that  requirements flow from an organization’s business requirements or its  concept of operations. However, as discussed later, NASA has not yet  completed a concept of operations.
In developing its core financial upgrade requirements, NASA established a  task force, consisting of both financial and program managers, whose  primary objective was to “review, assess, and document Program/Project  Management requirements as they relate to financial management.” In  addition, other groups of program managers were asked to review the  requirements and provide input to the task force. However, according to  NASA officials, they have not yet documented and institutionalized  requirements development procedures as recommended by SEI. Lacking  documentation, NASA cannot ensure that appropriate procedures are  followed and that all appropriate stakeholders are included in the process  so that all requirements are identified. Moreover, the requirements that  were addressed by the task force and user groups were at a very high or  general level and therefore, lacked a level of specificity that is needed to  ensure that users’ needs are met.
Because it did not have a well-documented, disciplined requirements  development process in place to provide reasonable assurance that all  requirements had been identified, NASA delayed finalizing the system’s  expected functionality until April 2006—about 6 months before the  upgrade was expected to be implemented—and continued to change some  requirements for several months after that. Delays in finalizing the  requirements contributed to delayed testing and a compressed testing  schedule. To meet the planned October 30, 2006, implementation date, the  three rounds of system testing for the core financial upgrade were  scheduled to occur from mid-June through September 22, with less than a  week between each round. A less compressed schedule could have  allowed more time between the testing cycles to perform necessary  actions, such as additional development work and testing to adequately  address the defects that had been identified. This, in turn, could have  reduced the risk that significant system defects would not be detected  prior to implementation.
One key to developing a realistic project schedule is determining the  sequence of activities, which requires identifying and documenting the  dependencies among the various project activities. For example, testing  activities cannot be completed before the software being tested is  developed, and software should not be developed until requirements have  been defined. However, NASA did not document the dependencies among  the detailed project tasks for the core financial upgrade and therefore, did  not have reasonable assurance that the project schedule established at the  start of the project was realistic. According to NASA officials, they  recognized this risk and adopted several processes to identify and mitigate  the weakness, such as having knowledgeable project officials review the  schedule and holding weekly status meetings to determine whether the  tasks were on schedule.
While the techniques used by NASA to constantly evaluate and adjust the  schedule are considered best practices and allowed NASA to gain  confidence in the schedule as the core financial upgrade project  progressed, they were not sufficient to ensure that the original schedule  was reasonable because they relied on ad hoc processes rather than a  formal task dependency analysis. If NASA had also identified the task  dependencies for the core financial upgrade, it would likely not have had  to rely on extraordinary efforts to complete the project. Rather, project  management would have been in a better position to assess the difficulty  in meeting the planned schedule and to take further steps to reduce this  risk, such as scaling back some aspects of the project or adding more  resources to the project.
According to NASA officials, through the heroic efforts of IEMP staff— their knowledge and experience with past projects and a considerable  amount of overtime invested—the core financial project team was able to  complete testing and other work within about 2 weeks of the planned  implementation date. Although NASA has made significant improvements  in its project management processes, NASA management recognizes that  weaknesses in its requirements development and project scheduling  processes have undermined some of the progress made. Despite the  implementation difficulties, NASA financial managers have indicated that  the core financial upgrade is now functioning as expected for most  transactions. Through the end of March 2007, the upgrade was in a  “stabilization” phase as NASA continued to work on correcting a number  of system errors, including posting errors for certain types of transactions.  Because NASA was continuing to stabilize the system during most of our  audit period, we were unable to determine the significance of these  weaknesses.
section_title: NASA Has Not Yet Fully Considered an Enterprise View of Its Operations and Processes
paragraphs: Although NASA has significantly improved its processes for implementing  IEMP projects, these improvements are directed at implementing the  desired functionality for an individual project. NASA has not yet fully  considered the higher-level strategic issues that affect how useful IEMP  will be in addressing long-standing management challenges—including  problems associated with stovepiped systems and parochial interests of  individual NASA components as well as problems in overseeing contractor  performance and properly accounting for its property, plant, and  equipment. NASA envisions IEMP to be a leading-edge business system  that will provide management information needed for mission success,  meet the needs of internal and external customers, and promote  standardization and integration of business processes and systems across  NASA. To achieve this vision, it is critical that NASA develop an  agencywide concept of operations and adopt standard business processes  that are supported by its software.
subsections: section_title: Concept of Operations Would Provide an Important Foundation for IEMP
paragraphs: NASA officials stated that they have undertaken a critical first step to  achieving their vision for IEMP—they have begun developing a concept of  operations to describe how all of its business processes should be carried  out. NASA created a framework for developing a concept of operations in  fiscal year 2006 and plans to complete it by the summer of 2008, according  to NASA officials. Ideally, a concept of operations should be completed  before system development begins so that it can serve as a foundation for  system planning and requirements development. Nonetheless, the  completion of such a document even at this late stage in NASA’s IEMP  effort would be beneficial for the development of the remaining IEMP  modules as well as any future upgrades to the core financial module. In  addition, once a concept of operations is complete, NASA could reassess  the modules that are already implemented and determine whether and  how they might need to be modified to best meet its agencywide needs.
A concept of operations defines how an organization’s day-to-day  operations are (or will be) carried out to meet mission needs. The concept  of operations includes high-level descriptions of information systems, their  interrelationships, and information flows. It also describes the operations  that must be performed, who must perform them, and where and how the  operations will be carried out. Further, it provides the foundation on  which requirements definitions and the rest of the systems planning  process are built. Normally, a concept of operations document is one of  the first documents to be produced during a disciplined development  effort and flows from both the vision statement and the enterprise  architecture. According to Institute of Electrical and Electronics  Engineers (IEEE) standards, a concept of operations is a user-oriented  document that describes the characteristics of a proposed system from the  users’ viewpoint. The key elements that should be included in a concept of  operations are major system components, interfaces to external systems,  and performance characteristics such as speed and volume.
For NASA, an effective concept of operations would describe, at a high  level, (1) how all of the various elements of NASA’s business systems  relate to each other and (2) how information flows among these systems.  Further, a concept of operations would provide a useful tool to explain  how business systems at the agency can operate cohesively. It would be  geared to a NASA-wide solution rather than individual stovepiped efforts.  Further, it would provide a road map that can be used to (1) measure  progress and (2) focus future efforts. While NASA’s enterprise architecture  efforts, when fully completed, can be used to help understand the  relationships between the various systems, a concept of operations  document presents these items from the users’ viewpoint in nontechnical  terms. Such a document would be invaluable in getting various  stakeholders, including those in the programs and administrative  activities, to understand how the business systems are expected to operate  cohesively and how they fit into “the big picture.”
section_title: Adopting Enterprise Business Processes Would Help NASA Transform the Way It Does Business
paragraphs: As part of an agencywide concept of operations, to best leverage its  investment in IEMP, NASA should also analyze the agency’s current  business processes and determine how these processes can be made more  efficient and effective. Specifically, NASA needs to ensure that the  business processes supported by this system are developed and  implemented to support the enterprise’s needs rather than primarily  focusing on the needs of a specific organizational entity. For example,  system efforts targeted only at addressing accounting or external financial  reporting needs—as was done during the initial implementation of the  core financial module—do not provide reasonable assurance that the  needs of the mission managers or other support organizations are  addressed as well. Our review identified an important opportunity for  NASA to leverage its investment in IEMP by using the system’s inherent  business processes to meet the enterprise’s needs.
Agencies such as NASA that invest in ERP solutions to meet their  enterprise needs often face difficulty in shifting from the stovepiped  processes of the past to the enterprise processes that underlie the ERP  concept. According to technical experts, a key benefit of an effective ERP  system is that the system provides the entire entity consistent data  regardless of which entity component generates a request or for what  purpose; the system maintains data based on the concept of “one truth.” In  other words, in non-ERP environments, one system may have one amount  for an agency’s obligations while another system has another amount for  the same obligations. While either of these systems may be the “official  system,” actions and plans may be based on information in the other  system. In order for all of an organization’s actions and plans to be  consistent, the same information needs to be available and used by all  segments of that organization. Under the ERP concept, it does not matter  whether an individual is in budget, accounting, procurement, or any other  organizational component; the answer to the question of “how much  money has been obligated and how much is still available” is consistent.
One example of an opportunity for NASA to use enterprise processes to  accomplish multiple needs is in the area of program oversight and  accounting for PP&E. NASA typically spends about 85 percent of its  budget procuring goods and services from its contractors each year.
Therefore, much of the cost information NASA needs to oversee its  programs and compile its external financial reports resides with its  contractors. For its larger contracts, NASA generally obtains cost data  from monthly contractor financial management reports, commonly  referred to as NASA Form 533s. NASA Form 533 captures planned and  actual contract costs and, according to NASA officials, is used for  budgeting, monitoring contract costs, and controlling program resources.  The Office of the Chief Financial Officer (OCFO) also uses NASA Form  533 to capture the costs reported on the agency’s financial statements.  However, NASA Form 533 does not contain information related to the  status of work performed on a contract. Therefore, for all major  acquisitions and for development or production contracts and  subcontracts valued at $20 million or more, in addition to NASA Form  533s, NASA’s contractors are also required to provide monthly contract  cost performance reports. Each of these reports is treated as a stovepiped  activity; that is, they provide cost information for a given contract in two  different formats and are used by different organizations and for different  purposes within NASA.
For those contracts for which NASA receives contract cost performance  reports in addition to Form 533s, program managers use the cost  performance reports to monitor contract performance, while the OCFO  uses NASA Form 533 to accrue costs that, among other things, are  reported on the agency’s financial statements. Although NASA Form 533  and the cost performance report reflect cost data pertaining to the same  contract, the level of detail provided in each report may vary considerably  depending on the contractor cost reporting requirements negotiated as  part of the contract. For example, the cost data required by program  managers to manage major acquisitions are often more detailed than those  required by the OCFO. In addition, because neither the cost performance  report nor NASA Form 533 contains the information needed by the OCFO  to properly account for equipment and other property acquired from  contractors, NASA also relies on periodic, summary-level information  provided by its contractors to report property amounts on its financial  statements.
When NASA initially implemented its IEMP core financial module in June  2003, it did not adequately consider program managers’ needs and did not  design the system to accommodate the more detailed cost data contained  in contractor cost performance reports. Since that time, NASA has  redesigned the coding structure embedded in the core financial module to  be more consistent with the work breakdown structure (WBS) coding  used by program managers. However, NASA continues to use cost data  from NASA Form 533—generally reported by contract line items—to  populate the core financial module. As a result, as shown in figure 1, NASA  uses a complex, NASA-specific process to allocate the costs reported on  NASA Form 533 to the WBS codes in IEMP based on available funding.
In a very simplified example, if NASA received a Form 533 showing  $1,000 of cost incurred for a particular contract line item and two WBS  codes pertained to that line item, NASA would allocate the costs to those  two WBS codes. Assuming WBS 1 had more funding available than WBS 2,  NASA might allocate $600 to WBS 1 and $400 to WBS 2. However, the  contract cost performance report might show that the actual costs were  $500 for WBS 1 and $500 for WBS 2. Although this allocation process  reorganizes cost data reported on NASA Form 533 into the same reporting  structure that is used by program managers, it still results in different  costs, maintained in different systems, used for different purposes.  Accordingly, these separate processes do not result in the “one truth” that  is provided when an ERP view is taken.
Further, this dual reporting approach has not addressed one of NASA’s  long-standing financial reporting weaknesses: reporting on its PP&E. For  example, NASA’s processes do not allow the agency to identify capital  costs—that is, those that should be recorded as assets—as they are  incurred. Instead, as we recently reported, the agency performs a  retrospective review of transactions entered into its property system to  determine which costs should be capitalized. This subsequent review is  labor-intensive and error-prone, and therefore increases the risk that not  all related costs will be properly captured and capitalized.
Figure 2 provides an example of how NASA could use IEMP to implement  an enterprise process that (1) provides the necessary data for the  enterprise operations and (2) reduces the burden on NASA and contractor  officials.
As shown in figure 2, if NASA received only one monthly report containing  contract cost data reported in sufficient detail for both program  management and financial reporting purposes, then it could record these  costs directly in IEMP without first going through an allocation process as  it does now. All individuals and components throughout NASA could then  use the same cost data that reside within IEMP for a given contract; IEMP  could provide different arrays of cost information based on each user’s  needs, but all cost information for a given contract would come from one  source. For example, as shown in figure 2, the program manager could use  the cost data from IEMP along with other supplemental contractor  performance information, such as labor hours used, to see if the project is  meeting expectations. In addition, if discrete WBS codes were established  to identify the costs associated with the acquisition of property, then IEMP  could automatically capitalize those costs and financial managers could  readily determine how much cost has been recorded for property. The  key is that under the enterprise process concept, single data entry is used  for multiple purposes. Since the enterprise view provides “one truth,” an  adequate audit trail over the data used to report property can be  maintained simply by reviewing the cost reports that were provided by the  contractors. Thus, NASA can take advantage of the efficiencies inherent in  an ERP solution by allowing the data needed for external financial  reporting to be produced as a by-product of the processes it uses to  manage its mission.
section_title: Conclusions
paragraphs: NASA has made significant strides in developing and implementing more  disciplined processes for supporting its IEMP efforts since our last report  in 2005. NASA has recognized the need for the disciplined processes  necessary to reduce risks to acceptable levels, as evidenced by its  implementation of several of our recommendations. More importantly,  NASA officials recognize that improving system implementation processes  is a continuous effort and that certain processes—particularly  requirements development and project scheduling—may need more  attention. However, the real key to realizing NASA’s IEMP vision is for  NASA’s management to develop an overarching strategy for managing its  agencywide management system development effort. We are encouraged  that NASA has begun to develop a concept of operations. As part of the  development of this document, it will be critical for NASA to define (1) the  agency’s business processes and information needs and (2) the types of  systems that will be used to carry out these processes and produce the  necessary information. Another critical factor in developing a concept of  operations will be analyzing the agency’s current business processes and  determining how these processes can be made more efficient and  effective. For example, NASA can take advantage of the efficiencies  inherent in the solution it has selected by utilizing an enterprise view to  produce the data needed for external financial reporting as a by-product of  the processes it uses to manage its mission. Unless NASA devotes  immediate, focused attention to taking these critical strategic planning  steps, it will continue to face the risk that its planned $800 million  investment in IEMP will not achieve the transformational changes  necessary to provide NASA with the information needed to make well- informed business decisions and to effectively manage its operations.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that disciplined processes are effectively implemented for  future IEMP modules, upgrades, or other business systems, we  recommend that the NASA Administrator direct the IEMP Program  Director to take the following two actions.
Establish requirements development policies and procedures regarding  (1) how customer needs will be elicited, developed, and validated;   (2) how to identify and ensure the involvement of relevant  stakeholders; and (3) required training in such topics as requirements  definition and analysis to be provided to people involved in the  requirements process.
Develop policies and procedures that require project schedules to  include the identification and documentation of dependencies among  various project tasks.
To help ensure that future IEMP projects are designed to carry out NASA’s  mission in an efficient manner that meets the needs of all users, we  recommend that the NASA Administrator establish as a high priority the  completion of a concept of operations that addresses NASA’s business  operations for both its mission offices and administrative offices (such as  financial management and human capital) before any new implementation  efforts begin.
Once the concept of operations is complete, we recommend that the NASA  Administrator review the functionality of previously implemented IEMP  modules for the purpose of determining whether enhancements or  modifications are needed to bring them into compliance with the concept  of operations.
To help ensure that NASA receives the maximum benefit from its reported  $800 million investment in IEMP, we recommend that the NASA  Administrator establish policies and procedures requiring approval to  establish or maintain business processes that are inconsistent with the  processes inherent in the COTS solutions selected for IEMP. The reasons  for any decisions made to not implement the inherent COTS processes  should be well-documented and approved by the Administrator or his  designee. At a minimum, approved documentation should address any  decisions to maintain current contractor cost reporting processes rather  than revise these processes to facilitate the use of one consistent source of  cost data.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from NASA, which  are reprinted in appendix II. NASA agreed with our recommendations and  described the approach and steps it is taking or plans to take to improve  its enterprise management system modernization efforts. We are  encouraged that a number of these steps are already under way, including  the establishment of an IEMP advisory body representing NASA’s missions  and centers. As NASA progresses in addressing our recommendations, it is  important that it focuses on the concepts and underlying key issues we  discussed, such as considering the need to reengineer key business  processes to support agencywide needs and to take full advantage of its  ERP solution. We continue to believe that careful consideration of all of  the building blocks and key issues we identified will be integral to the  success of NASA’s efforts. NASA also provided technical comments, which  we incorporated as appropriate.
As agreed with your offices, unless you announce its contents earlier, we  will not distribute this report further until 30 days from its date. At that  time, we will send copies to interested congressional committees, the  NASA Administrator, and the Director of the Office of Management and  Budget. We will make copies available to others upon request. In addition,  the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact McCoy Williams at (202) 512-9095 or williamsm1@gao.gov or Keith  Rhodes at (202) 512-6412 or rhodesk@gao.gov. Key contributors to this   report are acknowledged in appendix III. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine whether the National Aeronautics and Space Administration  (NASA) has improved its management processes for implementing the  Integrated Enterprise Management Program (IEMP), we reviewed project  management documentation for several IEMP projects, including the core  financial upgrade and the contract management module. The  documentation we reviewed for these projects included requirements  management documents, detailed testing plans, project schedules, risk  management plans, and metrics documentation. We also interviewed  numerous IEMP officials, including the IEMP Director, the Director and  Assistant Director at the IEMP Competency Center, and the Manager of  IEMP Application Development and Software Assurance. In addition, we  interviewed the leader of a NASA team that provided an independent  assessment of the core financial upgrade project to obtain his views of  IEMP management processes.
To assess NASA’s implementation of disciplined processes, we reviewed  industry standards and best practices from the Institute of Electrical and  Electronics Engineers, the Software Engineering Institute, and the Project  Management Institute. To assess the effectiveness of NASA’s requirements  management processes, we performed a traceability analysis of several  requirements for both the contract management module and the core  financial upgrade, which demonstrated that there was traceability among  the different levels of requirements and with testing documentation. To  determine whether NASA had adequately and systematically determined  the information needs of key users of IEMP data when developing system  requirements, we reviewed documentation of NASA’s requirements  identification effort for the core financial upgrade and interviewed a  number of program managers and staff who worked on various space and  science programs at three NASA centers—Marshall Space Flight Center,  Johnson Space Center, and Goddard Space Flight Center. We also met  with officials from the Office of the Chief Financial Officer (OCFO),  including the Deputy Chief Financial Officer, and with officials from the  Office of the Chief Engineer to obtain their opinions regarding the  requirements of the core financial upgrade. In addition, we discussed the  requirements development methodology with IEMP management.
To determine the results of the implementation of the core financial  upgrade, we met with both IEMP and OCFO officials. We reviewed data on  the amount and types of system defects that were identified by users  during the project’s stabilization phase. We also obtained written  responses to specific questions about the results of the implementation  from three NASA centers.
To determine the extent to which NASA has considered the higher-level  strategic issues associated with developing an enterprisewide concept of  operations and defining standard business processes, we met with senior  management from IEMP and the OCFO. In addition, we also discussed  these issues with senior officials in the Office of the NASA Administrator.  We also interviewed IEMP officials about NASA’s current processes for  recording contract costs. We also discussed this issue with officials from  the OCFO, the Office of the Chief Engineer, and the Office of Program and  Institutional Integration. In addition, we obtained documentation of  NASA’s plans for reengineering processes related to the costs of capital  assets. We briefed NASA officials on the results of our audit, including our  findings and their implications. On May 25, 2007, we requested comments  from NASA and we received them on June 21, 2007. NASA also separately  provided technical comments. Our work was performed from January  2006 through June 2007 in accordance with U.S. generally accepted  government auditing standards.
section_title: Appendix II: Comments from the National Aeronautics and Space Administration
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the contacts named above, staff members who made key  contributions to this report were Diane Handley, Assistant Director;   J. Christopher Martin, Senior Level Technologist; Francine DelVecchio;  Kristi Karls; and Lauren Catchpole.