RESULTS

Defect Management Was Not Adequate

12

The SVU Project Office did not establish adequate controls for managing defects.
NPR 7150.2, paragraph 3.4.5, requires a project office to document defects identified
during testing and track closure of defects. “Defect severity level” refers to the
importance assigned to a defect found during testing. Test Plan, paragraph 9.1, defines
the severity levels of defects:

e Low severity level defect — a noncritical problem that may require a change to be
made.

¢ Medium severity level defect — a materially damaging problem; it requires that a
change be made by a requested date.

¢ High severity level defect — a problem that would materially affect testing, causing
an immediate and substantial impact; it requires that a change be made by a
specific date.

e Very high severity level defect — a problem that cannot be circumvented, i.e., there
is no workaround available, and that impacts the operation of the affected
application; it requires that a change be made on an immediate-response basis.

Test Plan, paragraph 9.1, also requires automatically elevating the severity of any defect
from high to very high if the defect is not resolved (fixed or closed status) by a “specific
date.” According to the Test Plan, test team members were responsible for identifying
and documenting test defects and recommending a defect severity level. However,
according to the SVU Project Office defect and testing managers, test team members
often did not have the necessary expertise to determine appropriate defect severity levels
because test team members were functional users. When they encountered a defect, they
often rated the defect based on the impact on the function rather than rating the defect
based on the impact on continued testing, as required by the Test Plan. In addition, the
Test Plan did not include guidance requiring review of the defect severity level after the
initial recommendation. Consequently, SVU Project management determined that test
team members assigned inappropriate severity levels to defects, and changes in severity
levels had to be made afterward. However, the reason for the change in the severity level
of a defect was not always recorded.

We reviewed all 1,117 defects identified in Quality Center during the three SIT Passes
and found that the severity levels were changed for 147 defects (13.2 percent). However,
SVU Project staff adequately documented the reason for the change for only 21 of the
147 defects (14.3 percent).

In addition, the SVU Project Office identified a total of 86 high severity defects—

60 during SIT Pass One, 15 during SIT Pass Two, and 11 during SIT Pass Three—none
of which were fixed or closed during their respective SIT Passes. Consequently, testing
associated with those defects was delayed and the defect was not resolved by a “specific
date.” Therefore, the SVU Project Office should have elevated all 86 defects to very high

Report No. IG-07-031


APPENDIX B

our recommendation. The SVU Project Manager stated management had already defined
contingency test plans for areas of highest risk from an integration perspective, outlined a
process in the Test Plan that dealt with issues as they occurred during testing, and defined
very specific exit criteria for each SIT Pass. However, as noted in the body of this report,
the SVU Project Office did not always follow its own exit criteria for each SIT Pass.

3. Adequacy of Test Procedures. On June 9, 2006, we informed the IEM Program
Director that the SVU test procedures were not adequate and did not trace to project

requirements. We recommended that the IEM Program Director require the SVU Project
Manager to

solidify all project requirements and test procedures prior to proceeding into
testing,

* adequately develop test procedures and trace them to project requirements prior to
entering a test readiness review and prior to actual testing, and

© provide the IA contractor the necessary inputs to conduct a requirements
traceability analysis test.

On June 19, 2006, the IEM Program Director agreed, stating that the actions management
was taking met the intent of our recommendations. Specifically, they were working to
have all low-level design requirements (Level Vs) solidified and test plans created prior
to the start of testing on June 26. However, there were still a few remaining items to be
worked out in the detailed design of some custom developed objects, as well as in some
process areas (e.g., reimbursable); therefore, it may not be realistic to expect zero
volatility in the detailed design requirements as SVU moves into SIT Pass One. The IEM
Program Director stated that he felt that it was reasonable to proceed into testing because
they had clearly defined exit criteria that will help reduce risk as SVU moves from one
testing pass to the next. In addition, the IEM Program Director concurred that it was a
sound process to have all test plans developed and traced to project requirements prior to
entering the TRR. While this may no longer be possible for SVU due to the schedule that
has already been established, the IEMP Program will take the action to adjust its standard
Test Plan Approach to ensure that it requires test plan development and tracing to
requirements to be completed prior to a project’s TRR. The IEM Program Director also
stated that management was working with the IA contractor to provide all the information
needed to conduct the traceability analysis and to verify where SVU was in terms of the
test plan coverage for project requirements. Although management agreed with our
assessment, test procedures were never adequately documented in any of the SIT Passes.
In addition, the Requirements Traceability Report was issued on May 12, 2006, without
including the Level V requirements.

4. Review of Test Steps. On June 9, 2006, we informed the IEM Program Director that a
documented review and analysis of the test procedures was not being performed prior to
the TRR. We recommended that functional teams document and submit all test
procedures for independent verification and review at least 3 to 5 working days prior to

Report No. IG-07-031 33


APPENDIX A

“Information Technology: Architecture Needed to Guide NASA’s Financial Management
Modernization” (GAO-04-43, November 21, 2003)

“Business Modernization: Improvements Needed in Management of NASA’s Integrated
Financial Management Program” (GAO-03-507, April 30, 2003)

National Aeronautics and Space Administration

“Synopsis of Management Issues Associated with NASA’s Integrated Financial
Management Program” (IG-05-020, June 3, 2005)

“Audit of the Implementation of Integrated Financial Management Program (IFMP)
Audit Recommendations” (IG-05-008, February 11, 2005)

“NASA’s Travel Module Lacks Management Control Structure and Compliance With
Federal Requirements” (IG-04-027, September 24, 2004)

“Integrated Financial Management Program Budget Formulation Module” (IG-04-017,
March 30, 2004)

“Integrated Financial Management Program Core Financial Project Information
Technology Security Planning and Implementation” (IG-04-016, March 31, 2004)

“Summary Report on Audit of Integrated Financial Management Program Core Financial
Module” (IG-03-028, September 29, 2003)

“Integrated Financial Management Program Core Financial Module Conversion to Full
Cost Accounting” (IG-03-015, May 30, 2003)

Gartner Research and Consulting

Gartner Research and Consulting, “Core Financial Module Implementation Assessment”
(June 15, 2005)

Report No. IG-07-031

29


APPENDIX B

We received management’s response on October 3, 2006. The Competency Center
Director partially agreed, stating that Competency Center staff would provide explicit risk
management documentation for any high severity defect that did not have a workaround
and include the likelihood and impact of the defect on SVU functionality. However, the
Competency Center Director did not agree to test workarounds in SIT Pass Three because
he believed the staff had made ongoing efforts to obtain closure for all high severity
defects. In addition, the SVU Project Office would be providing a status report for all
SVU issues, including explicit risk management documentation for each unresolved
defect, during the ORR. The Competency Center Director stated that ORR participants
would provide input to SVU Project Office management, IEM Program management, and
leaders of the governing bodies to confirm or recommend changes to the original
assessment. At the conclusion of the ORR, a decision would be made to either exercise
the contingency plan and continue using the current version of SAP or proceed as
planned. If the decision was made to proceed, the SVU Project Office would continue its
efforts to finalize remaining workaround processes and accompanying procedures in
preparation for go-live.

11. Testing and Defect Management. On September 29, 2006, we informed the IEM
Program Director that the SVU Project Office did not develop or implement adequate
testing and defect management policies and procedures. We recommended that the IEM
Program Director require the SVU Project Manager to

¢ develop and document policies and procedures to control the modifications of test
procedures;

¢ document and implement policies and procedures establishing the management,
assignment, and change controls of defect severity levels;

¢ develop procedures to establish specific automatic escalation date of high defects
to very high; and

© conduct a review and report those high defects that should have been escalated to
very high for visibility purposes.

We received management’s response on October 19, 2006. The Competency Center
Director did not clearly specify whether he agreed with our recommendations but
provided the following information:

In an effort to improve our software implementation processes, the Competency
Center is piloting a form of Agile software development project management called
Scrum. The Scrum approach advocates short cycles of iterative development, called
sprints, during which an integrated team of functional and technical resources works
on a pre-determined set of system features. The aim of each sprint, which typically
last thirty days, is to deliver a fully tested and working solution. Both unit and full
functional tests are executed as part of each sprint.

The major benefit that we see in the Scrum model versus the traditional waterfall
approach is that it no longer relies on static requirements that are clearly and

REPORT No. IG-07-031


RESULTS

NPR 7123.1, Table G-6, establishes entry criteria for a Test Readiness Review (TRR) and
Table G-9 establishes entry criteria for the ORR. According to Table G-6, a TRR is
intended to ensure that the system, facility, personnel, and procedures are ready for
testing. According to Table G-9, an ORR should result in reporting the actual system
characteristics and procedures used in the system and that all system and support
hardware, software, personnel, procedures, and user documentation accurately reflect the
deployed state of the system.

Software Requirements Had Not Been Stabilized

The SVU Project Office did not stabilize software requirements before developing the
Test Plan or test procedures.'' According to NPR 7150.2, paragraph 5.2.1.1, software
requirements are the specifications that system developers and program managers use to
acquire, implement, and test the software within a system. According to NPR 7150.2,
paragraph 3.4, testing verifies the code against the requirements and the design to ensure
that the requirements are implemented. Because testing determines a system’s viability,
verifying test procedures against requirements that are not stable is not effective and
cannot ensure the system will fulfill users’ needs.

SVU Project management established five levels of requirements, addressing both system
requirements and software requirements. System requirements are the high-level
requirements that primarily entail (1) what the system or process must do to fulfill
business needs and (2) how well—in terms of performance measures such as speed,
security, or usability—the system or process fulfills those business needs. Software
requirements, at a lower level, describe in detail the specifications that system developers
and program managers use to acquire, implement, and test a system. For the purposes of
this report, system requirements are Levels I through III and software requirements are
Levels IV and V. The five levels of requirements are as follows:

e Level I — Agency drivers (IEMP guiding principles);

e Level II- functional drivers (critical business needs of NASA stakeholders for the
specific IEMP project);

¢ Level Ill — high-level requirements (processes and software features needed to
satisfy Level II and regulatory requirements);

e Level IV — acquisition requirements (detailed functional and technical

requirements that specify how the application should work and that link to
Level III requirements); and

"' The detailed steps used for testing software during each SIT.

6 Report No. IG-07-031


48

APPENDIX D

entrance and exit criteria for each SIT pass and will continue to use this criteria going
forward.

7. The IEM Program Director should require the project manager to ensure
that all IA analysis is performed and all TA reports are complete.

MSEC Response: Concur. We agrce that the IA contractor missed several opportunities
to formally document how schedule pressures were impacting SVU requirements stability
and increasing risk, though these concerns were verbally communicated to the project
team. Recent MOA adjustments have shifted the [A analysis more toward the Agile
‘Scrum approach. Due to its iterative nature, the Scrum process is making the Level V
requirements and their traceability available for inspection much earlier in the process.
As a result, the IA resource has more of an opportunity to review the requirements,
provide comments, and alert the project to schedule or scope risks. This change in
process should address the intent of the recommendation, even though the IA work will
continue to focus primarily on ensuring adherence to the process and not on subject
matter level verification and validation. In addition, the IA contractor is now formally

documenting findings in the IEMP Issues Tracking Database and in the NASA IV&V
Issues Tracking System.

The enclosure to this response includes additional comments and recommendations
regarding the draft audit report.

Please contact me if you have any questions or require additional information regarding

this yepponse.

/

Enclosure

ce:

Director, Marshall Space Flight Center

Deputy Director, Integrated Enterprise Management
Program/Competency Center Manager

SVU Project Manager

ALR Representative, Marshall Space Flight Center

REPoRT NO. IG-07-031


46

APPENDIX D

MANAGEMENT COMMENTS

Renly to Aan et

National Aeronautics and
Space Administration

Headquarters
Washington, OC 20546-0001

September 14, 2007
Integrated Enterprise Management Program

TO: Assistant Inspector General for Auditing
FROM: —_ Director, Integrated Enterprise Management Program

SUBJECT: Management Response to Office of Inspector General Draft Audit Report
entitled “System Integration Testing of the Systems, Applications and

Products Version Update Project Needed Improvement”. (Assignment No,
A-06-017-00)

Thank you for the opportunity to respond to the Draft Audit Report, Assignment A-06-
017-00, provided on August 16, 2007. Below is NASA management’s response to each
of the recommendations set forth in the draft report. This is a consolidated response
which includes inputs compiled from various members of the Integrated Enterprise
Management Program (IEMP) staff, including the SAP Version Upgrade (SVU) Project
Manager, the Director of the IEMP Competency Center (CC) and the IEMP Program

Director. In addition, MSFC inputs were coordinated with and reviewed by the MSFC
Associate Center Director.

Recommendations:

1, The IEM Program Director should require the project manager to stabilize
software requirements prior to beginning testing.

MSEC Response: Concur. Current project efforts are utilizing a process (Agile Scrum)
that is designed to help drive out requirements during recurring iterations of design,
development, and testing. The IEMP Competency Center is using this process on new
projects and is tracking the volatility of requirements through the iterations to verify that
the approach leads to stable requirements prior to the start of SIT. The approach also
consists of full-scale functional testing of the components delivered during each iteration,
further helping to validate and lock down requirements.

2. The IEM Program Director should require the project manager to establish

a baseline of test procedures to be completed during each SIT before beginning
testing.

MSFC Response: Concur. The IEMP Competency Center has always made a concerted
effort to establish a baseline of test procedures prior to the start of each SIT pass.

Report NO. IG-07-031


APPENDIX E

REPORT DISTRIBUTION

National Aeronautics and Space Administration

Administrator
Deputy Administrator
Chief of Staff
Associate Administrator, Program Analysis and Evaluation
Chief Financial Officer
Chief Information Officer
Integrated Enterprise Management Program Director
Director, Marshall Space Flight Center
Director, Competency Center
Supervisor, IEMP Business Process & Application Support Office

Non-NASA Organizations and Individuals

Office of Management and Budget
Deputy Associate Director, Energy and Science Division
Branch Chief, Science and Space Programs Branch
Government Accountability Office
Director, Defense, State, and NASA Financial Management, Office of Financial
Management and Assurance
Director, NASA Issues, Office of Acquisition and Sourcing Management

Congressional Committees and Subcommittees, Chairman and
Ranking Minority Member

Senate Committee on Appropriations

Senate Subcommittee on Commerce, Justice, and Science
Senate Committee on Commerce, Science, and Transportation

Senate Subcommittee on Science and Space
Senate Committee on Homeland Security and Governmental Affairs
House Committee on Appropriations

House Subcommittee on Commerce, Justice, Science, and Related Agencies
House Committee on Oversight and Government Reform

House Subcommittee on Government Management, Organization, and Procurement
House Committee on Science and Technology

House Subcommittee on Investigations and Oversight

House Subcommittee on Space and Aeronautics

52 RePorRT No. IG-07-031


APPENDIX C

Quality Center. An automated tool the SVU Project Office uses to document test
procedures including test steps, the expected test results, and actual test results. Quality
Center is also used to link system requirements to test procedures.

Regression Test. A type of testing used to reevaluate software requirements and
software design issues whenever any significant code change is made. Regression testing
involves retesting to verify that the modified software still meets its specified
requirements. Regression testing is performed when any changes to the product are made
during installation to verify that the basic software requirements and software design
assumptions affecting other areas of the program have not been violated.

Requirements Traceability Analysis. Identifying relationships of system requirements
to software requirements for correctness, consistency, completeness, and accuracy.

Software Requirement. Specifications that system developers and program managers
use to acquire, implement, and test a system.

System Integration Testing (SIT). A type of testing that validates that the software
correctly implements the software requirements and design as each software component
is incrementally integrated with each other.

System Integration Testing (SIT) Pass. For the SVU Project, iterations of testing, with
each pass conducted independent from the other and within its own system environment.

System Requirement. What the system, process, or product/service must do in order to
fulfill the business needs and how well it must do it, in terms of performance
measurements such as speed, security, or usability.

Test Plan. A document that describes the plans for software component level testing,
software integration testing, software qualification testing, and system qualification
testing of software systems. The plan describes the software test environment to be used
for testing, identifies the tests to be performed, and provides schedules for environment,
development, and test activities. The plan provides an overview of software testing, test
schedules, and test management procedures.

Test Procedure. The detailed steps used for testing software during each SIT.
Test Readiness Review (TRR). A review that is intended to ensure that the test article
(hardware/software), test facility, support personnel, and test procedures are ready for

testing and data acquisition, reduction, and control.

Test Set. A set of related business events. For SVU, a test set consists of the name of the
general area that would be tested during each SIT.

Test Traceability Analysis. Identifying relationships in the test plans, requirements, and
test procedures for correctness and completeness.

REPorRT NO. IG-07-031


RESULTS

Independent Assessment Was Not Adequate

16

The independent assessment (IA)'° of the SVU was not adequate. IEMP and the Goddard
Space Flight Center (Goddard) Independent Verification and Validation (IV&V) Facility
entered into a memorandum of agreement (MOA), January 4, 2006, for IA of various
systems including the SVU. The MOA, valued at $480,000,'¢ outlined work to be
performed and deliverables for the SVU project, the Contract Management Module,
e-Travel, and the Aircraft Maintenance Module. However, the contractor hired to
conduct the IA (1A contractor) submitted inadequate and incomplete reports to the IV&V
Project Manager who, in turn, submitted those reports to the SVU Project Office.

Requirements Traceability Report Incomplete. The Requirements Traceability
Report, dated May 12, 2006, was not complete because it did not provide a determination
of the adequacy of requirements traceability from Levels I through V or whether all
requirements were objectively testable, as required by the MOA, paragraph 6. The IA
contractor could not complete the requirements traceability analysis or the testability
determination because the SVU Project Office did not provide him with the final Level V
requirements. The SVU Project Office did not have stable Level V requirements until
September 2006. However, the IA contractor did not report the impact of not having the
Level V requirements documented before the development of test procedures in the
conclusions presented in the Requirements Traceability Report.

What the contractor reported in the Requirements Traceability Report was that the
traceability of the Level V requirements would be postponed until the Test Analysis
Report was completed. The IA contractor did perform a traceability analysis of the
Level I through Level IV requirements. However, the IA contractor only traced each
system requirement to one software requirement and each software requirement to one
system requirement. The process the IA contractor used did not consider all possible
links that requirements could have. Without considering all possible links, the IA
contractor could not show that system requirements were completely addressed by the
software requirements. In addition, by not having the Level V requirements stabilized
before the development of test procedures, there was an increased risk that testing would
be inadequate and an increased risk that the SVU Project would not meet schedule
milestones. According to “IEMP Independent Assessment Task Plan,” February 17,
2006, paragraph 4.3, the IA contractor should have identified the increased risk to testing
and the schedule risk in the Requirements Traceability Report, but did not.

'S NASA IV&V Facility IVV 09-2 “Independent Assessments,” September 2002, paragraphs 3 and 6, state
that an IA is an analysis of the software development and products to identify critical software risks that
could jeopardize mission safety and success. IAs can range from the analysis of a specific subsystem
performed at a specific time during the life cycle to assessments of several segments of the system over the
entire life cycle of the development activity. IA includes system level assessments to evaluate specific
requirements, testing, and review processes and associated risk management.

'S IEMP provided $350,000, and $130,000 came from the Corporate general and administrative overhead
account.

Report No. IG-07-031


APPENDIX B

the TRR. All issues and modifications to the test baseline should be tracked. On

June 19, 2006, the IEM Program Director agreed, stating that it is a sound process to have
test plans available for review prior to the TRR. Though SVU has been diligent in
ensuring that test plans are reviewed, this process was not fully completed prior to TRR.
The IEM Program Director agreed that completing the test plans earlier would provide
more time to locate testing gaps and correct them prior to the start of a SIT. He stated
that they were taking action to adjust their standard test plan approach to ensure that test
procedures were developed and reviewed before the TRR for future projects. In addition,
they planned to consider implementing a process that enables baselining test procedures
for future projects and then tracking changes that occur after the baseline date.

5. Delta TRR for SIT Pass One. On June 21, 2006, we informed the IEM Program
Director that the SVU Project Office, with input from other key decision makers, decided
to proceed with SIT Pass One based on an inadequate Delta TRR, held June 12-13, 2006.
Several entrance and exit criteria were not met before and after conducting the TRR. We
recommended that the IEM Program Director require the SVU Project Manager to
conduct another Delta TRR after meeting the following conditions:

* solidify and conduct a documented review of test procedures to include tracing
requirements,

¢ resolve technical issues surrounding data migration, and

* provide TRR participants with briefing charts, whitepaper status, and access to

test procedures, allowing them adequate review time prior to conducting the Delta
TRR.

We received a preliminary response from the IEM Program Director on June 21, 2006,
requesting that we revise the issue statement to note that the IEM Program Director also
received input from a Marshall Space Flight Center-led independent review team. We
received another response to the issue statement from the Competency Center Director on
June 30, 2006, stating that, although they agreed with the issues identified, action had
already been taken to address our concerns; consequently, they did not agree with the
recommendation to conduct a follow-up TRR. The Competency Center Director also
stated that they had executed follow-on actions that addressed the specific conditions as
prerequisites to a Delta TRR. Although management stated they executed follow-on
actions that addressed our concerns, test procedures were not adequately documented
prior to testing and a review of test procedures never occurred. In addition, some
technical issues concerning data migration were not resolved before SVU was
implemented.

6. Delta TRR for SIT Pass Two. On July 26, 2006, we informed the IEM Program
Director that the SVU Project Office, with input from other key decision makers, decided
to proceed with SIT Pass Two for the SVU based on an inadequate Pass Two Delta TRR,
held on July 19, 2006. Several entrance and exit criteria were not met before or after
conducting the TRR. We recommended that the IEM Program Director require the SVU

Report No. IG-07-031


OVERVIEW

¢ The SVU Project Office did not develop an adequate process for assigning and
changing defect severity levels or accurately report the number of very high
defects, as required by NPR 7150.2, paragraph 3.4.5, and the Test Plan,
paragraph 9.1.

¢ SIT Pass Three—regression testing—was not successfully completed before the
ORR, as required by NPR 7123.1, Table G-9. SIT Pass Three could not be
considered a successful regression test because only 66.5 percent of the
requirements had been successfully tested before the start of SIT Pass Three and
because of the number of defects—at least 49—-found during SIT Pass Three that
required code changes to correct.

¢ The independent assessment performed by the Independent Verification and
Validation Facility contractor was not adequate. Specifically, the contractor did
not determine whether requirements adequately traced to test procedures; did not
conduct sufficient analysis to provide confidence that all appropriate requirements
were adequately tested, as required by the memorandum of agreement,>
paragraph 6; and did not identify the increased risk to the project schedule, as
required by the “IEMP Independent Assessment Task Plan,” February 17, 2006,
paragraph 4.3. The contractor only analyzed a judgmental sample of 10 test
procedures out of a total of 4,127 test procedures (0.002 percent) and concluded
that “all selected tests appeared to adequately test the specified requirements.”

Because requirements were not stabilized, a baseline of test procedures could not be
developed. Also, SIT was adversely impacted because defect management was
inadequate, regression testing was unsuccessful, and the requirements were not
adequately traced to test procedures. The combination of these factors delayed SVU
system implementation until November 20, 2006,‘ and resulted in significant software
deficiencies that had to be corrected after implementation. As of September 2007, the
SVU Project Office was resolving system issues, and at least one Business Warehouse
report was not expected to be corrected until October 2007.

Management Action

Throughout the course of our audit, management considered all of our recommendations
and took action to improve the planning and execution of testing procedures for the SVU
Project as they deemed appropriate. Given that the opportunity to address all of the issues
we identified in the SVU Project has passed, the recommendations contained in this

* The IEM Program Office and the Goddard Space Flight Center Independent Verification and Validation
Facility entered into a memorandum of agreement, January 4, 2006, for the independent assessment of
various systems, including the SVU.

* The Core Financial system was available to a limited number of users for limited processing on
November 15, 2006, but was not available to all users for all types of processing until November 20,
2006.

Report No. IG-07-031 iti


APPENDIXES

APPENDIX A

Scope and Methodology

We performed the fieldwork at NASA Headquarters, Goddard Space Flight Center, Ames
Research Center, and Marshall Space Flight Center. We performed this audit from May
2006 through September 2007 in accordance with generally accepted Government
auditing standards. To gain an understanding of NASA’s test planning for the SVU, we
obtained and reviewed applicable documents including, but not limited to

¢ NPR 7120.5C, “NASA Program and Project Management Processes and
Requirements,” March 22, 2005;

¢ NPR 7123.1, “NASA Systems Engineering Processes and Requirements,”
March 13, 2006;

¢ NPR 7150.2, “NASA Software Engineering Requirements,” September 27, 2004;

¢ NASA IV&V Facility [VV 09-1, “Independent Verification and Validation,”
March 16, 2006;

¢ Industry standards such as the Institute of Electronic and Electrical Engineers

(IEEE) Standard 1012, “IEEE Standard for Software Verification and Validation,”
June 8, 2005;

e TEMP Release 7.1 System Integration Test Plan, May 18, 2006;

¢ National Institute of Standards and Technology Special Publication 500-234,
“Reference Information for the Software Verification and Validation Process,”
March 29, 1996; and

¢ Memorandum of Agreement Between Goddard Space Flight Center IV&V
Facility and the IEM Program Management, January 4, 2006.

To determine whether the SVU Project Office prepared adequate test plans, we reviewed
the test plans, the performance measures developed by the Project Test Team that
establishes criteria used when determining pass or fail of specific tests, requirements
documentation, and design documentation. In addition, we met with program and project
management personnel.

To determine whether appropriate testing was performed, we reviewed system and
software requirements, test procedures, and test results.

Report No. IG-07-031 27


RESULTS

severity and notified key decision makers but did not do so. For example, on

September 22, 2006, we identified defect 1106 as a defect that should have been elevated
from high to very high, because the defect was not fixed by the end of SIT Pass Two, and
reported to key decision makers. However, the IEM Project Director disagreed with our
assessment and did not follow Project guidance established in the Test Plan. Defect 1106
was not fixed during testing. After SVU was implemented, SVU Project management
determined that defect 1106 was concealing other software defects. Defect 1106 and the
concealed additional software defects resulted in an unusually large number of transaction
errors occurring in Travel Manager, which ultimately delayed reimbursement of travel
expenses to NASA employees.

The Defect Manager stated that automatic promotion of high severity defects to very high
was not done because the entry and exit criteria from the SIT Passes provide equivalent
visibility and reporting of both the high and very high defects. While visibility is an
important factor in reporting status, prioritizing resources to fix defects is also an
important factor associated with defect severity. Although the SVU Project Office
reported high and very high severity defects with equivalent visibility, reporting a defect
as high severity did not communicate the same level of urgency as properly reporting that
defect as very high severity. As SVU Project management reported during the SIT Pass
Three Delta TRR, defect levels were often changed after developers had spent time trying
to fix them. The lack of strict management of defect severity classification resulted in
unresolved software defects that required costly and cumbersome manual workarounds
after SVU implementation, or “go-live.”

During a meeting with the SVU Project Manager on August 23, 2006, we stated that it
appeared that high and very high severity defect levels were being lowered in order to
meet entry and exit criteria outlined in the Test Plan. Test Plan, paragraph 5.2, requires
that all high or very high defects be in the “ready to retest” status before beginning SIT
Pass Three. The SVU Project Manager agreed that the severity level could be improperly
changed to meet entry and exit criteria. Subsequently, the SVU Project Manager
presented proposed changes to defect severity levels to key decision makers for approval.

We notified the SVU Project Manager about inadequate defect management in an e-mail
on September 29, 2006. The SVU Project Office took action by implementing additional
informal procedures to assign and control defect severity levels. Specifically, the SVU
Project Office made an informal change in the way defect severity levels were assigned
and in who was authorized to change defect severity. The new process required that the
tester label the severity level as “TBD” [To Be Determined]. Then, defect managers and
test team managers met daily to discuss and assign the severity level for each new defect
as well as discuss the status of existing defects. The severity level for each defect
discussed was then updated in Quality Center. In addition, the SVU Project Office
restricted access to changing the severity level. However, the SVU Project Manager did
not agree with our concern about not accurately reporting the number of very high defects
to key decision makers. The SVU Project Manager stated that the SVU Project Office
reviewed the high and very high defects and reported all those defects during each SIT
Pass. The SVU Project Manager did not implement a procedure to automatically elevate

REPORT No. IG-07-031 13


RESULTS

* of the time spent waiting for a software fix prior to making the decision to
implement the manual workaround and input the travel data, and

* of additional down-time'” of SAP interfaces.

On January 18, 2007, the SVU Project Office fixed the interfaces affecting payment of
travel vouchers.

Business Warehouse Testing and Reporting. The SVU Project Office did not identify
software deficiencies involving data integrity issues within Business Warehouse.
Business Warehouse is a Web-based application that provides reports using data from the
Core Financial system database. Reports from Business Warehouse are used by NASA
managers to assess and manage financial resources of NASA programs.

At an SAP industry conference in April 2007, the Competency Center Director stated that
NASA had experienced a problem with Business Warehouse. Specifically, “Reliable
capabilities available in 4.6C used to link FM [Funds Management] documents to
originating documents in other modules (FI [Financial Accounting], MM [Materials
Management], etc) were broken in 2005/ECC 6.0 . . . Required extensive effort by NASA
to design and implement workaround (deployed 3/25).” The Competency Center Director
acknowledged that this issue had a significant reporting impact and that SAP was
investigating a long-term solution.

On January 18, 2007, Business Warehouse was not generating accurate reports for funded
programs and cost centers.'* Funded programs capture financial information by project,
and cost centers capture financial information by organization. For both funded programs
and cost centers, obligations and disbursements were not being reported correctly in
Business Warehouse. Without accurate reporting, managers do not have accurate and
timely financial information to manage costs. One organization at Goddard used
QuickBooks to provide accurate information about their fund balances because they could
not obtain needed reports from Business Warehouse. Another organization at Goddard
used an Excel spreadsheet to control their accounts payable functions because they said
they could not rely on the Payment History Report generated by Business Warehouse.

In February 2007, the SVU Project Office applied a fix and corrected labor and travel
reporting in Business Warehouse. In March 2007, the SVU Project Office applied a fix
and corrected procurement. However, according to the Competency Center Director,

these fixes were temporary workarounds and SAP was still in the process of developing a
final solution.

"’ Atthe end of the fiscal year, SAP normally shuts down for approximately 2 weeks; however, SVU
increased the down time to approximately 4 weeks. In addition, to the 4 weeks of down-time, the SVU
interfaces were also shut down from December 12, 2006, through December 19, 2006, because the NASA
Office of the Chief Financial Officer wanted to review the appropriation data.

' According to the Competency Center Director, this was fixed in March 2007.

Report No. IG-07-031 19


RESULTS

regression test because of the significant number of defects fixed during the SIT Pass.
We recommended that the IEM Program Director require the SVU Project Manager to
conduct a regression test to verify that the software modifications resulting from fixing
the defects during SIT Pass Three did not introduce unintended errors and that the system
still complied with its specified requirements.

On November 16, 2006, the Competency Center Director stated that he did not concur
with conducting additional SIT Passes, stating that appropriate action had been taken to
ensure that the SVU was ready for use and all known risks and issues had been clearly
documented and accepted.

ORR Held Prematurely. The SVU Project held the ORR on October 11, 2006, even
though testing was not complete and some defects were still unresolved. NPR 7123.1,
Table G-9, establishes entry criteria for the ORR. Specifically, an ORR should report on
the actual system characteristics and procedures used in the system and report whether all
system and support hardware, software, personnel, procedures, and user documentation
accurately reflect the deployed state of the system. At the time of the ORR, SIT Pass
Three was still in progress, 32 test procedures had not been successfully run, and not all
of the workarounds needed for go-live had been provided. Those workarounds were not
provided until November 16, 2006. Without a successful execution of a valid ORR, key
stakeholders might not be aware of or agree with the SVU Project Office’s intended
defect resolution, operational status, or new processes and procedures to be implemented.
Lack of stakeholder awareness of or agreement with defect resolution could result ina
lack of user confidence in the updated system and users not being aware of system
limitations, operational controls, and/or user workarounds needed to perform required
tasks.

Although the SVU Project Office held two Authority to Proceed decision meetings after
the ORR, those meetings only presented potential or partial system characteristics. The
purpose of the ORR was not fully realized because the SVU never presented an entire
view of the system as it was to be implemented. That approach allowed the SVU Project
Office more time to correct known defects; however, a presentation of the entire view of
all actual system characteristics to be implemented was not accomplished.

On October 31, 2006, we notified the IEM Program Director and SVU Project Manager
that the SVU Project Office had not completed testing and had unresolved defects as of
October 26, 2006. We recommended that they conduct a second ORR.

The Competency Center Director stated that the SVU Project Manager had presented a
complete status of open defects including severity, functionality impacted, business
impact, and workaround in the event the defect was not fixed during the ORR. In
addition, the team provided the status on a weekly basis leading up to the Authority to
Proceed decision meetings on October 25, 2006, and November 10, 2006. The
Competency Center Director stated that the Authority to Proceed decision meetings met
the intent of the additional ORR that we recommended.

Report No. IG-07-031 15


APPENDIX A

To determine whether procedures to address and resolve test failures were established and
to ensure all failures were addressed and corrected, we reviewed test procedures and
policies for resolution of test failures, reviewed test results, and met with program and
project management personnel.

Use of Computer-Processed Data. We used data from Quality Center to view the SVU
Project’s requirements, test plans, and test procedures during the audit. Quality Center
basically organizes and stores the testing data. The only time Quality Center manipulates
data is when summaries of the testing information are generated. During the audit, we
did not use summary information generated by Quality Center. Also, we did not rely
solely on the information within Quality Center to draw conclusions.

Review of Internal Controls

We reviewed NASA management controls for determining whether testing for the SVU
Project was adequately planned and performed. We identified weaknesses in the testing
process as described in the results section of this report.

Prior Coverage

The Government Accountability Office (GAO) and the NASA Office of Inspector
General (OIG) have issued reports of particular relevance to the subject of this report.
Unrestricted reports can be accessed over the Internet at http://www.gao.gov (GAO) and
http://www.hq.nasa. gov/office/oig/hq/audits/reports/F Y07/index.html (NASA). In
addition, a private firm has produced a report relevant to this topic.

Government Accountability Office

“Business Modernization: Some Progress Made Toward Implementing GAO

Recommendations Related to NASA’s Integrated Financial Management Program”
(GAO-05-799R, September 9, 2005)

“Information Technology Management: Government wide Strategic Planning,
Performance Measurement, and Investment Management Can Be Further Improved”
(GAO-04-49, January 12, 2004)

“Business Modernization: NASA’s Challenges in Managing Its Integrated Financial
Management Program” (GAO-04-255, November 21, 2003)

“Business Modernization: NASA’s Integrated Financial Management Program Does Not
Fully Address Agency’s External Reporting Issues” (GAO-04-151, November 21 , 2003)

“Business Modernization: Disciplined Processes Needed to Better Manage NASA’s
Integrated Financial Management Program” (GAO-04-118, November 21, 2003)

28 Report No. IG-07-031


10

RESULTS

completing the test . . . [procedures] earlier would provide more time to locate testing
gaps and correct them prior to the start of a System Integration Test. The IEM
Program will take the action to adjust its standard Test Plan Approach to ensure that it
requires test plan development and review to be completed prior to the TRR for future
projects.

The IEM Program Director did not agree to improve either the documentation of test

procedures baseline or the review of test procedures during SVU testing because of time
constraints.

Deviations from Test Plan During SIT Pass One Not Adequately Reported. The
SVU Project Office did not report deviations to the Test Plan during SIT Pass One.

NPR 7150.2, paragraph 5.3.2.1, requires that detailed software test reporting must include
deviations from test procedures in the Test Plan and the rationale for each deviation.

During SIT Pass One, the SVU Project Office marked 52 test procedures “N/A” within
Quality Center but did not report those test procedures as deviations. In addition,
although the name of the test was left in Quality Center, the text describing detailed test
steps and results was deleted for 35 of the 52 test procedures marked N/A. The following

table lists the test procedures that were marked N/A and detailed test steps that were
deleted.

Number of Test Procedures Deleted or Marked “N/A” from
Established Test Sets for SIT Pass One

Number of Number of Number of Total
Test Set Test Procedures Procedures = Number of
Database Procedures Deleted and = Only Marked Procedures
Number Title in Test Set Marked N/A N/A Not Run
24 OPTIN-CCR-REIM 21 1 0 1
32 Deobs-Converted Lines 15 5 0 5
36 Bankcard 12 6 0 6
NSSC Working Capital
52 Fund Activities with the 9 1 1 2
Reimbursable Process
SGL Upward and
55 Downward Adjustment 26 4 7 11
Processing
59 Travel 67 14 7 21
72 AP Only 28 2 1 3
Data Migration
TM Validation R ° ' '
BW Data Migration
B Validation 4 2 o 2
Total Total 195 35 17 52

Report No. IG-07-031


SEPTEMBER 28, 2007
AupIT Report No. IG-07-031

OFFICE OF AUDITS

OFFICE OF INSPECTOR GENERAL
et

ADDITIONAL COPIES

Visit www.hq.nasa.gov/office/oig/hq/audits/reports/FY 07/index.html to obtain additional copies of this
report, or contact the Assistant Inspector General for Auditing at 202-358-1232.

COMMENTS ON THIS REPORT

In order to help us improve the quality of our products, if you wish to comment on the quality or
usefulness of this report, please send your comments to Ms. Jacqueline White, Director of Quality

Assurance, at Jacqueline. White@nasa.gov or call 202-358-0203.

SUGGESTIONS FOR FUTURE AUDITS

To suggest ideas for or to request future audits, contact the Assistant Inspector General for Auditing.
Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing
NASA Headquarters
Washington, DC 20546-0001

NASA HOTLINE

To report fraud, waste, abuse, or mismanagement, contact the NASA OIG Hotline at 800-424-9183 or
800-535-8134 (TDD). You may also write to the NASA Inspector General, P.O. Box 23089, L’Enfant
Plaza Station, Washington, DC 20026, or use http://www.hq.nasa.gov/office/oig/hq/hotline.html#form.
The identity of each writer and caller can be kept confidential, upon request, to the extent permitted
by law.



INTRODUCTION

According to both Scope Documents, those benefits will contribute to NASA’s financial
tracking and reporting and its goal of achieving an unqualified audit opinion. The SVU
Project Manager estimated that the SVU cost about $70.1 million—$48.2 million for the
SVU Project and $21.9 million for implementation at the Centers.

System Integration Testing. Software testing helps identify the correctness,
completeness, security, and quality of developed and purchased computer software.’ A
subset of software testing is system integration testing (SIT),* a vitally important phase of
the testing process. The SVU Project Office developed the 7.1 System Integration Test
Plan (Test Plan), May 18, 2006, to provide guidance on conducting the SITs. SIT tests
business processes, system functions (standard and custom), application security, and user
procedures. Paragraph 5.1 of the Test Plan sets up each SIT in three “Passes” or
iterations of testing to validate that the software correctly implements the software
requirements and design as each software component is incrementally integrated with
each other. The objectives of the SITs were to ensure that related systems (such as the
Core Financial system, Contract Management Module, Travel Manager, Business
Warehouse, and the Bank Card system) operated in accordance with the process design
specifications for all business functions and that the parts of the system worked together
as an integrated whole. According to the Test Plan, each SIT Pass is to be conducted
independently within its own testing environment.

¢ SIT Pass One — integrated, end-to-end business functions. This pass covers the
day-to-day key business activities that represent the heart of NASA’s business.
This pass includes testing of custom development components (all types), based
on availability. During SIT Pass One, test defects’ are identified and documented,
but execution does not stop and wait for resolution. Execution continues using
workarounds, if necessary, except in the situation where the error prevents any
further action from taking place completely.

¢ SIT Pass Two — integrated, end-to-end business functions. As in Pass One, Pass
Two covers the day-to-day key business activities that represent the heart of
NASA’s business and includes testing of custom development components (all
types). In Pass Two, also, test defects are identified and documented and
execution does not halt and wait for test defect resolution; execution continues
with open test defects using workarounds where possible. In Pass Two, however,
if execution cannot continue with workarounds, if the workaround does not
support the objectives of the test, or if the error prevents any further action from
taking place completely, execution is halted. However, by the end of SIT Pass
Two, all significant defects should be ready to retest during SIT Pass Three.

7 NASA Technical Standard 8739.8, “Software Assurance Standard,” July 28, 2004.

* IEEE Standard 610.12-1990, “IEEE Standard Glossary of Software Engineering Technology,”
September 28, 1990.

° A defect is a flaw in a system or system component that causes the system or component to fail to perform
its required function. A defect, if encountered during execution, might cause failure of the system.

Report No. IG-07-031


RESULTS

Evaluation of Management’s Response. Management’s comments are not responsive
because they did not agree to comply with the current policy when conducting future
IEMP projects. NPR 7123.1A requires that test failures and anomalies be resolved prior
to the ORR. However, at a minimum, all high?! and very high” severity defects should
be resolved or have documented workarounds that have been successfully tested. We
request that the IEM Program Director reconsider his position and provide additional
comments in response to this final report.

Recommendation 7. The IEM Program Director should require the project manager to
ensure that all IA analysis is performed and all IA reports are complete.

Management’s Response. The IEM Program Director concurred, stating they agree that
the IA contractor missed several opportunities to formally document how schedule
pressures were impacting SVU requirements stability and increasing risk, although those
concerns were verbally communicated to the project team. The IEM Program Director
said recent MOA adjustments have shifted the IA analysis more toward the Agile Scrum
approach. The IEM Program Director also said that the IA contractor is now formally
documenting findings in the IEMP Issues Tracking Database and in the NASA IV&V
Issues Tracking System.

Evaluation of Management’s Response. Management’s comments meet the intent of
the recommendation. We will validate implementation of this recommendation when we
perform future audits of IEMP projects. We consider this recommendation closed.

*1A high severity defect is a problem that would materially affect testing, causing an immediate and
substantial impact; it requires that a change be made by a specific date.

* A very high severity defect is a problem that cannot be circumvented, i.e., there is no workaround
available, and that impacts the operation of the affected application; it requires that a change be made on
an immediate-response basis.

REPORT No. IG-07-031

25


RESULTS

was October 30, 2006. For full implementation, November 20, 2006, is a more
accurate date. Although the system was opened up on November 15, 2006, for
existing funds, the users experienced problems processing transactions; therefore,
corrections had to be made before the system could be opened to all users. Once
the corrections were made and tested, the system was opened for all users on
November 20, 2006. During SIT Pass One and SIT Pass Two (June to September
2006), the SVU Project experienced testing failures in migration activities, data
validation, and sales orders. Because of those problems and the inability to
resolve the issues by the scheduled implementation date, the SVU Project
Manager asked the Financial Steering Group for approval to implement alternate
solutions that ultimately delayed implementation.

¢ While it is IEMP’s standard practice to establish a baseline of test plans and test
sets prior to SIT, this was not done as required by NPR 7123.1. Prior to SIT Pass
One, during the Delta TRR in June 2006, the SVU Project Manager informed
participants that they did not have all the test plans and sets developed.

e Although the IEM Program Director stated that all of the workarounds were
provided to the project teams prior to go-live, at least two Centers said they never
received them. Specifically, we met with the SVU transition teams at Marshall
Space Flight Center and Ames Research Center on October 27, 2006, and
November 14, 2006, respectively. Both teams stated that they had not been
provided with the workarounds. During a briefing on November 30, 2006, the
SVU Project Manager informed the Financial Steering Group that the Centers
were provided the workarounds on November 20, 2006.

¢ The MOA is specific about the tasks to be performed by the IA contractor. The
MOA directs the IA contractor to “Review requirements to provide confidence
that requirements exhibit adequate traceability between levels. ... Review
requirements to help ensure that they are objectively testable... . Review
traceability of requirements from level I through level V.” To accomplish that,
the MOA requires the IA contractor to review requirements, test plans, test cases,
traceability of requirements, and other tasks. The MOA does not require the IA
contractor to ensure that the SVU Project adhered to the IEMP quality assurance
process. The Competency Center currently has a staff dedicated to that function.

Recommendations, Management's Response, and Evaluation of
Management’s Response

Recommendation 1. The IEM Program Director should require the project manager to
stabilize software requirements prior to beginning testing.

22 REPORT NO. IG-07-031


SEPTEMBER 28, 2007

CONTENTS

INTRODUCTION

Background 1

Objectives 3
RESULTS

System Integration Testing Was Not Adequate 4
APPENDIX A

Scope and Methodology 27

Review of Internal Controls 28

Prior Coverage 28
APPENDIX B

Issues Communicated During the Audit 30
APPENDIX C

Glossary 42
APPENDIX D

Management Comments 46
APPENDIX E

Report Distribution 52

Report No. IG-07-031


OVERVIEW

report are intended to improve the planning and execution of testing procedures for future
IEMP projects.

For future IEMP projects, we recommend that the IEM Program Director and the Project
Manager comply with NPR 7150.2, NPR 7123.1, and the Independent Assessment Task
Plan by (1) stabilizing software requirements prior to beginning testing; (2) establishing a
baseline of test procedures to be completed during each SIT before testing begins;

(3) implementing policies and procedures establishing the management, assignment, and
change controls of defect severity levels; (4) reporting all deviations from test procedures;
(5) successfully completing a regression test: (6) not holding the ORR until testing is
completed and defects are resolved; and (7) ensuring that all independent assessment
analysis is performed and all independent assessment reports are complete. (See the
Results section for our evaluation and disposition of management’s comments.)

In response to a draft of this report, the IEM Program Director concurred with the
recommendations to stabilize software requirements prior to beginning testing; establish a
baseline of test procedures to be completed during each SIT Pass before testing begins;
implement policies and procedures for managing, assigning, and changing defect severity
levels; report all deviations from test procedures; complete a successful regression test;
and ensure all Independent Assessment analysis is performed and all independent
assessment reports are complete.

The IEM Program Director partially concurred with Recommendation 6 to complete all
testing and resolve all defects prior to holding the ORR, stating that he agrees that an
ORR should not be held until testing is completed but disagreed with the NPR 7123.1
requirement to resolve all defects prior to entering the ORR. Specifically, the
Competency Center understands the intent of the regulation but believes it is not feasible
or cost effective to force resolution of every individual defect related to an administrative
software application prior to conducting the ORR. Although we agree that it may not be
cost effective to resolve all defects prior to the ORR, we maintain that all high’ and very
high® severity defects should be resolved or have documented workarounds that are
successfully tested prior to the ORR, as required by NPR 7123.1, Table G-9.

We have closed Recommendations 1, 2, 4, 5, and 7. Recommendation 3 will remain
open until all actions have been completed and verified. We request that the IEM
Program Director provide us with additional comments for Recommendation 6.

*A high severity defect is a problem that would materially affect testing, causing an immediate and
substantial impact; it requires that a change be made by a specific date.

SA very high severity defect is a problem that cannot be circumvented, i.e., there is no workaround
available, and that impacts the operation of the affected application; it requires that a change be made on
an immediate-response basis.

Report No. IG-07-031


RESULTS

We interviewed the testing managers to understand their rationale for marking the test
procedures N/A, because reasons were not reported or documented.

« Seven test procedures from test sets 24-OPTIN-CCR-REIM, 32-Deobs-Converted
Lines, and 74-Data Migration Validation were not run because they were
dependent on resolution of a defect identified in a prior test procedure.

¢ Two test procedures from test set 75-BW Data Migration Validation were not run
because the Business Warehouse analysis was not completed.

The SVU Project Office completed at least 5 of the 52 test procedures marked N/A during
SIT Pass One; however, those 5 test procedures failed and a defect report should have
been generated for each failure. The other 47 test procedures marked N/A were not run
and should have been reported as deviations to the test plan, as required by NPR 7150.2,
paragraph 5.3.2.1. According to the SVU Project Office database administrator for
Quality Center, he was directed by SVU Project Office management not to include any
test procedures marked N/A when providing summary information to the SVU Project
Office on the results of SIT Pass One. The effect of this omission is that inaccurate and
incomplete test results for SIT Pass One were reported to key decision makers during the
Delta TRR for SIT Pass Two.

We discussed the lack of effective controls over reporting deviations from the Test Plan
during testing and the use of N/A with the SVU Project Manager during a meeting on
August 23, 2006. We also identified these issues in an e-mail on September 29, 2006.
The SVU Project Manager agreed to consider implementing controls over test procedures
in future IEMP projects, and the SVU Project Office took action to eliminate the practice
of marking test procedures N/A. Specifically, starting with SIT Pass Two (July 28
through September 12, 2006), the SVU Project Manager reported testing deviations and

directed the SVU testing managers not to use the N/A status within Quality Center for
SIT Pass Two or Three.

Contingency Plans that Included a Backout Plan Not Developed. During SIT Pass
One, we found that the SVU Project Office had not developed test contingency plans that
included a backout plan. NPR 7123.1, Table G-6, states that test contingency plans must
be completed and documented before the TRR. Without test contingency plans or a
backout plan, key personnel might not know what procedures to follow in the event of
unanticipated scenarios or failures—for example, if the SVU experiences a major delay or
cancellation. When we notified SVU Project management officials on June 5, 2006, they
promptly added a backout plan, incorporated contingency planning into the technical
review processes and modified the Competency Center Release 7.1 Cutover Plan to
address steps to be taken in the event an issue prevented the SVU from being
implemented.

Report No. IG-07-031


SEPTEMBER 28, 2007

INTRODUCTION

Background

Over the last decade, NASA has invested significant resources to modernize its business
systems and to implement the Integrated Enterprise Management Program (IEMP).
Through incremental implementation of commercial software packages and related
hardware and software components, IEMP is expected to produce an integrated
NASA-wide business and financial management system. (See Appendix C for a glossary
of system and programmatic terminologies.) Systems Applications and Products’ (SAP)
R/3 is the software foundation for NASA’s Core Financial system. The Core Financial
system is the IEMP backbone for implementing standard processes and systems needed to
support NASA’s financial management activities.

President’s Management Agenda. The President’s Management Agenda was initiated
in 2001 to improve management and performance in Federal agencies. The Agenda
includes five Government-wide goals—one of which is to improve financial performance.
Improved financial performance would provide Federal managers accurate and timely
financial information to use to manage costs. The Office of Management and Budget
updates the status of the five Government-wide goals quarterly. Since June 2002, NASA
has had an unsatisfactory rating in the “improve financial performance” goal. By
updating the Core Financial system, NASA hopes to improve compliance with Federal
regulations and change its rating to satisfactory.

SAP Version Update Project. The Integrated Enterprise Management (IEM) Program
Office initiated the SAP Version Update (SVU) Project in September 2005 to update
from SAP R/3 SAP 4.6c to mySAP Enterprise Resource Planning 2005/Enterprise Core
Component 6.0 (mySAP ERP 2005/ECC 6.0). MySAP ERP 2005/ECC 6.0 contains
source code fixes and redesigns that have been based on issues seen in previous versions
of SAP, including SAP R/3 SAP 4.6c. According to the SVU “Initial Scope Document,
Baseline,” December 15, 2005, and the SVU “Scope Document, Revision A,” Draft,
April 5, 2006, benefits of the SVU, once implemented, are expected to include

¢ improved data integrity based on SAP funds management redesign;

¢ additional automation of adjustment accounting entries;

¢ improved support for the budget distribution process;

¢ improved program/project management information for decision making: and

¢ streamlined year-end processing (starting with FY 2007).

Report No. IG-07-031 1


RESULTS

During our initial interview with the SVU Quality Assurance Manager and the IA
contractor in September 2006, they stated that they did not determine the adequacy of
requirements traceability between levels or whether all requirements were objectively
testable. In a later interview in October 2006, the IV&V Project Manager stated that the
IA contractor fulfilled the intent of the MOA. However, the documentation provided by
the IV&V Project Manager did not support the assertion that the requirements were
adequately traced from Levels I through V because the IA contractor only performed a
traceability analysis of the Levels I through IV requirements. In addition, the IA
contractor could not have determined whether all requirements were objectively testable
because 4,065 of the 7,898 Level V requirements (51.5 percent) were either created or
modified after the Requirements Traceability Report was issued.

Test Analysis Report Inadequate. The Test Analysis Report, dated September 22,
2006, and revised on October 2, 2006, was not adequate because the IA contractor did not
conduct sufficient analysis to provide confidence that all appropriate requirements were
adequately tested as required by the MOA, paragraph 6. The IA contractor only analyzed
a judgmental sample of 10 test procedures out of a total of 4,127 test procedures

(0.002 percent) and concluded that “all selected tests appeared to adequately test the
specified requirements.”

During interviews in September 2006, the LA contractor stated, he did not determine
whether all appropriate requirements were adequately tested and the IV&V Project
Manager agreed that the sample taken did not provide adequate confidence that all
appropriate requirements were adequately tested. In a later interview in October 2006,
the IV&V Project Manager stated that the IA contractor fulfilled the intent of the MOA.

Management Action on IA Issues. During the course of our audit, we discussed our
findings with the SVU Project Office. In addition, in an e-mail on September 7, 2006, we
notified the IEM Program Director that the Requirements Traceability Report was
incomplete and that the IA contractor was not planning to perform all analyses needed to

produce the Test Analysis Report. We recommended that the SVU Project Office ensure
that the IA contractor

¢ complete the analysis that should have been included in the Requirements
Traceability Report—specifically, determine the adequacy of requirements
traceability from Levels I through V and whether all requirements were
objectively testable; and

e complete the analysis for the Test Analysis Report—specifically, conduct
sufficient analysis to provide confidence that all appropriate requirements were
adequately tested.

On September 18, 2006, the SVU Project Office responded that

e the IA contractor was including the review of traceability between Levels IV and
V requirements in the Test Analysis Report,

Report No. IG-07-031


SEPTEMBER 28, 2007

AuDIT REPORT

OFFICE OF AUDITS
i

SYSTEM INTEGRATION TESTING OF THE SYSTEMS,
APPLICATIONS, AND PRODUCTS VERSION UPDATE
PROJECT NEEDED IMPROVEMENT

OFFICE OF INSPECTOR GENERAL

National Aeronautics and
Space Administration

Report No. IG-07-031 (ASSIGNMENT No. A-06-017-00)


APPENDIX B

team continued to provide current status on a weekly basis leading up and subsequent to
the first Authority to Proceed checkpoint on October 25, 2006. The Competency Center
Director stated that the Authority to Proceed checkpoint met the intent of a Delta ORR.
The next and final Authority to Proceed checkpoint prior to go-live would be conducted
on November 10, 2006.

13. Regression Testing. On October 31, 2006, we informed the IEM Program Director
that the SVU Project Office had not completed a successful regression test. Specifically,
as of October 30, 2006, the SVU Project Office had identified 58 high or very high
severity level new defects during SIT Pass Three, some of which required code changes
to correct. According to the Test Plan, SIT Pass Three was to be a regression test for
previously identified defects. The significant number of new defects identified and fixed
during that Pass indicated that SIT Pass Three was not a successful regression test. We
recommended that the IEM Program Director require the SVU Project Manager to
conduct a regression test.

We received management’s response on November 16, 2006. The Competency Center
Director did not concur with our recommendation if our recommendation would require
them to conduct, in essence, additional formal passes of testing. The Competency Center
Director stated that given the SVU Project Office’s follow-on testing activities through
the Authority to Proceed review on November 10, 2006, to close remaining defects,
management believed appropriate action and diligence had been demonstrated to ensure
the delivered system was ready for implementation and all known risks and issues had
been clearly documented and accepted.

14. Testing, Unresolved Defects, and Data Migration. On October 31, 2006, we
informed the IEM Program Director that the SVU Project Office had not completed
testing, had defects that had not been resolved, and that data migration was not complete.
The SVU Project Office was to provide satisfactory resolution of risks and issues voiced
by the Centers, including end-to-end testing of data migration, before it could complete a
valid ORR. On October 25, 2006, the SVU Project Office held an Authority to Proceed
review. According to the SVU Project Office, the purpose of the Authority to Proceed
review was to provide an intermediate project milestone to assess project and
environment readiness to proceed, provide an update on open issues from the ORR,
review new issues found since the ORR, and assess overall release 7.1 risk. We
recommended that the IEM Program Director require the SVU Project Manager to
conduct an ORR after completion of all testing and resolution of all defects.

We received management’s response on November 15, 2006. The Competency Center
Director stated that they concurred with our recommendation, based on the alignment of
their efforts with our expectations. The Competency Center Director stated that SVU
project management continued to resolve and test remaining defects beyond the
completion of SIT Pass Three and that they presented a complete status of open defects
including severity, functionality impact, business impact, and workaround at the ORR. In
addition, the Competency Center Director stated that the team continued to provide
current status on a weekly basis leading up to the first Authority to Proceed review on

REPoRT No. IG-07-031


RESULTS

SYSTEM INTEGRATION TESTING
Was NoT ADEQUATE

System integration testing was inadequate because NASA managers did not take a
sufficiently disciplined approach to testing SVU software. The SVU Project
Manager did not

stabilize software requirements before developing the Test Plan and test
procedures, as required by NPR 7150.2, “NASA Software Engineering
Requirements,” September 27, 2004, paragraph 3.4;

prepare adequate test plans to include a baseline of test procedures to be
tested during each SIT and test contingency plans that incorporated a backout
plan, as required by NPR 7123.1, “NASA Systems Engineering Processes and
Requirements,” March 13, 2006, Table G-6, and NPR 7150.2,

paragraph 5.3.2.1, nor adequately report on deviations from the Test Plan
during SIT Pass One, as required by NPR 7150.2, paragraph 5.3.2.1;

develop an adequate process for assigning and changing defect severity levels
or accurately report the number of very high defects, as required by
NPR 7150.2, paragraph 3.4.5, and Test Plan, paragraph 9.1;

complete SIT Pass Three—regression testing—before the Operational
Readiness Review (ORR), as required by NPR 7123.1, Table G-9; or

ensure that reports from the independent assessment (IA) contractor were
adequate and complete.

As a result, system implementation was delayed until November 20, 2006,'° and the
SVU Project Manager did not identify significant software deficiencies that had to be
corrected after the system was implemented. As of September 5, 2007, the SVU
Project Office was still resolving system issues.

Concerns Raised During the Audit, and Action Taken

During the audit, we maintained continuous communication with the IEM Program
Director, the SVU Project Manager, and the Competency Center Director and informed
them of issues we identified as the SVU went through its three SIT Passes. Specifically,
between June and October 2006, we provided the IEM Program Director, the SVU

'° The Core Financial system was available to a limited number of users for limited processing on
November 15, 2006, but was not available to all users for all types of processing until November 20,

2006.

REPORT No. IG-07-031


INTRODUCTION

e SIT Pass Three — regression testing for test defects plus sequenced “close”
activities (month-end/year-end). This pass is used to re-execute any test plan from
Pass One or Pass Two that was completed (all steps/functions executed) with test
defects outstanding. In this situation, the test condition status would be “failed”
and the Test Plan status would note that execution was completed, but test defects
were outstanding. As such, this test condition would need to be retested through
one of its mapped test plans.

Test Plan, paragraph 5.3, states that test procedures will be organized by test sets. Test
sets summarize a set of related business events that, when performed in sequence,
represent an actual business function or transaction from start to finish. A test set will
have one or more test procedures and each test procedure may have one or more detailed
test steps. Test procedures were developed by the Integrated Process Team leads for the
different functional areas. The Integrated Process Team leads are also generally the
testing managers that organize and conduct the testing.

Objectives

The overall objective of the audit was to determine whether testing for the SVU was
adequately planned and performed. Specifically, we determined whether the SVU Project
Office prepared adequate test plans, performed all testing in the test plans, and established
procedures to address and resolve test failures. We also reviewed internal controls as
they related to the overall objective. See Appendix A for details of the audit’s scope and
methodology, our review of internal controls, and a list of prior coverage.

RePorT No. IG-07-031


Major Contributors to the Report:
Catherine Schneiter, Director, Financial and Institutional Management Directorate
Susan Aggen, Project Manager
Rhodora Posey, Auditor
Janice Smith, Auditor
Trevor Strickland, Auditor
Troy Zigler, Auditor
Mike Beims, Computer Engineer, Technical Advisor
Janet Overton, Report Process Manager

Report No. IG-07-031

53


APPENDIX B

the issues surrounding the migration of sales order data until just prior to the SIT Pass
Two Delta TRR held on July 19, 2006. At that review, SVU Project Office management
presented several possible solutions and a recommendation to the Financial Steering
Group, and later to the Financial Executive Round Table, for consideration. These key
decision makers agreed that SIT Pass Two should proceed as planned but that a parallel
test environment was needed to validate the solution endorsed by the governing bodies
and then implement the solution into the testing environment for SIT Pass Three. While
the decision was made on July 19, 2006, to continue with testing, SVU Project Office
management stated that the parallel approach to resolving open issues involving
migration of sales order data and the progress made addressed our recommendation
without taking a serial approach that would cause critical delays in overall testing.

8. IA Analysis. On September 7, 2006, we informed the IEM Program Director that the
IA contractor was not planning to perform all agreed-to analysis. In addition, issues that
the IA contractor identified did not receive high visibility from the SVU Project Office.
We recommended that the NASA IV&V Facility Project Manager for IEMP require the
IA contractor for the SVU to conduct a statistical sample of requirements to ensure that
the requirements were objectively testable (and include testing results in a formal report)
and adequately tested (and include testing results in a formal report) and also develop, in

coordination with the SVU Project Manager, a method to formally report and resolve
IV&V issues.

We received management’s response on September 18, 2006. The Competency Center
Director partially agreed with our recommendations, stating that

e the IA contractor was including the review of traceability between Levels IV and
V requirements in the Test Analysis Report due on September 22, 2006;

¢ the IA contractor’s test plan review was sufficient to ensure that they adequately
covered requirements;

¢ there was “little value for further review to determine objective testability of
requirements, since at the current time the project has now entered the final test
cycle”; and

¢ aclose, proactive working relationship with the IV&V representative provided the
greatest value to their efforts.

The Competency Center Director also stated that, in the future, the IV&V contractor will
use an IEMP issues-tracking system to officially log IA issues and track their progress
through closure. After reviewing management's response, we reiterated our concerns in
an e-mail on October 6, 2006, and suggested that the IEM Program Director verify how
much money was paid for work that the IA contractor did not perform and determine
whether any of those funds could be recouped.

REPORT No. IG-07-031


RESULTS

Evaluation of Management’s Response. Management’s comments are responsive. We
validated the existence of a new security role established in Quality Center. However,
this recommendation will remain open until we evaluate the policies to be used for defect
severity management.

Recommendation 4. The IEM Program Director should require the project manager to
report all deviations from test procedures.

Management’s Response. The IEM Program Director concurred, stating that the
Competency Center Director agreed to include any test procedures marked as “Not
Applicable” in the test metrics and report on deviations from the test baseline.

Evaluation of Management’s Response. Management’s comments are responsive. We
will validate implementation of this recommendation when we perform future audits of
IEMP projects. We consider this recommendation closed.

Recommendation 5. The IEM Program Director should require the project manager to
successfully complete a regression test.

Management’s Response. The IEM Program Director concurred, stating that the
Competency Center Director will establish a baseline of test procedures prior to each SIT
pass and will ensure that all procedures have been successfully tested, or ensure that a
documented workaround exists for any failed tests, prior to system implementation.

Evaluation of Management’s Response. Management’s comments are responsive. We
will validate implementation of this recommendation when we perform future audits of
IEMP projects. We consider this recommendation closed.

Recommendation 6. The IEM Program Director should require the project manager to not
hold the ORR until testing is completed and defects are resolved.

Management’s Response. The IEM Program Director partially concurred, stating that
the Competency Center Director agrees that an ORR should not be held until testing is
completed. However, the IEM Program Director does not agree that all defects must be
resolved prior to entering the ORR. NPR 7123.1A,”° establishes entrance criteria for the
ORR which specifies that “Test failures and anomalies from validation testing have been
resolved and the results incorporated into all supporting and enabling operational
products.” The IEM Program Director stated that, while the Competency Center Director
understands the intent of this statement, particularly with respect to space flight systems,
it does not seem feasible or cost effective to force resolution of every individual defect
related to an administrative software application prior to conducting the ORR. The IEM
Program Director pointed out that the Competency Center utilizes a set of rigorous
entrance and exit criteria for each SIT pass and will continue to use this criteria going
forward.

?° NPR 7123.1 is an update of NPR 7123.1 and was issued on March 26, 2007, after SUV testing was
completed. There were no significant changes concerning entry criteria for the ORR.

24 Report No. IG-07-031


APPENDIX B

October 25, 2006, and the second Authority to Proceed review on November 10, 2006.
The Competency Center Director stated that he determined that the Authority to Proceed
reviews met the intent of our recommendation to conduct a valid ORR.

Report No. IG-07-031

41


RESULTS

Management’s Response. The IEM Program Director concurred, stating that they have
incorporated into their current project efforts a process called Agile Scrum,!? which is
designed to help identify requirements during recurring iterations of design, development,
and testing. According to the IEM Program Director, the Competency Center Director is
using this process in developing new projects and is tracking the volatility of
requirements through the iterations to verify that the approach leads to stable
requirements prior to the start of SIT. In addition, Agile Scrum includes full-scale
functional testing of the components delivered during each iteration, further helping to
validate and lock down requirements.

Evaluation of Management’s Response. Management’s comments are responsive. We
will validate implementation of this recommendation when we perform future audits of
IEMP projects. We consider this recommendation closed.

Recommendation 2. The IEM Program Director should require the project manager to

establish a baseline of test procedures to be completed during each SIT before beginning
testing.

Management’s Response. The IEM Program Director concurred, stating that the
Competency Center has always made a concerted effort to establish a baseline of test
procedures prior to the start of each SIT pass. However, management agreed that
schedule pressures had sometimes negatively impacted a team’s ability to completely
finalize all test procedures in a timely manner. The IEM Program Director also said that
the Competency Center will examine its release planning procedures and, in conjunction
with the benefits of testing early-and-often afforded by Agile Scrum, work on improving

its ability to complete all test procedures and lock down the testing baseline prior to the
start of each SIT pass.

Evaluation of Management’s Response. Management’s comments are responsive. We
will validate implementation of this recommendation when we perform future audits of
IEMP projects. We consider this recommendation closed.

Recommendation 3. The IEM Program Director should require the project manager to
implement policies and procedures establishing the management, assignment, and change
controls of defect severity levels.

Management’s Response. The IEM Program Director concurred, stating that the
Competency Center has taken steps to improve the controls surrounding the management
of defect severity levels. Specifically, a new security role was established in Quality
Center that permits only specific management personnel to change defect severities. The
IEM Program Director stated that the Competency Center is in the process of establishing
a set of standard policies to be used for defect severity management.

° Scrum, according to the Competency Center, is a best practices methodology that has frequent
intermediate deliveries enabling continuous customer reviews, interactive feedback, and more efficient
testing. This methodology is intended to result in a more integrated, customer-focused product with
greater overall quality.

Report No. IG-07-031




RESULTS

the severity of any defect from high to very high if the defect was not resolved (fixed or
closed status) by the “specific date,” as required by paragraph 9.1 of the Test Plan.

Testing Was Not Completed Before the ORR

14

Regression Testing Not Successfully Completed. The SVU Project Office did not
complete a successful regression test. IEEE Standard 610.12-1990, “IEEE Standard
Glossary of Software Engineering Terminology,” September 28, 1990, defines regression
testing as selective retesting of a system to verify that modifications have not caused
unintended effects and that the system still complies with its specified requirements.
Regression testing is used to reevaluate software requirements and software design issues
whenever any significant code changes are made.

According to Test Plan, paragraph 5.1, SIT Pass Three was intended to be a regression
test for previously identified defects. Test Plan, paragraph 5.2, contains entry criteria for
SIT Pass Three testing requiring that, “Open defects in very high or high severity should
be in ‘Ready to Retest’ status.” Test Plan, paragraph 5.6, contains exit criteria for SIT
Pass Two and SIT Pass Three that requires that all test procedures be executed. However,
the SVU Project Office did not complete 57 test procedures during SIT Pass Two and

32 test procedures during SIT Pass Three. Therefore, the exit criteria for SIT Pass Two
and SIT Pass Three was not met.

In addition, according to the “Test Analysis Report (Revision A),” dated October 2, 2006,
on September 11, 2006, after the end of SIT Pass Two and before the start of SIT Pass

Three, the status of the testing of the 7,361 Level V requirements in Quality Center was
as follows:

Fifty-one (51) requirements were not covered by test procedures.
Test procedures for 492 requirements were not run.

Test procedures for 4,897 requirements were successfully run.
Test procedures for 921 requirements failed.

Test procedures for 1,000 requirements were not completed.

During SIT Pass Three, the SVU Project Office identified 182 defects,'* which included
57 new high and very high severity defects, some of which required software coding
changes to correct. Because only 66.5 percent of the Level V requirements had been
successfully tested before the start of SIT Pass Three and because at least 49 defects
found required code changes to correct, SIT Pass Three could not be considered a
successful regression test.

On October 31, 2006, we informed the IEM Program Director that the SVU Project
Office had not completed a successful regression test. Specifically, we stated that SIT
Pass Three, which ended on October 25, 2006, could not be considered a successful

'S These defects include defects fixed or closed before SVU was implemented.

Report No. IG-07-031


APPENDIX D

“We agree that little value would have been gained by the IA contractor determining
whether requirements were objectively testable in September 2006. ..."

The report states on page 17 that “However, the IA contractor only traced cach system
requirement to one sofiware requirement and each software requirement to one system
requirement, Without considering all possible links, the independent assessment (IA)
contractor could not show that system requirements were completely addressed by the
software requirements.” Additionally, on page 19, the report states “That the IA
contractor did not determine objective testability of the requirements before the project
entered the final test cycle is an example of an IA benefit that was not realized.)”

The report indicates a wide gap between how the OIG interprets the [A memorandum of
agreement (MOA) and how that document is interpreted by the NASA Independent
Verification and Validation (IV&V) PM for IEMP, the IEM Program Office, and the
TEMP Competency Center Director. We understand the importance and value of the IA
and agree that all [A analysis should be performed and all reports should be complete.
However, the OIG appears to interpret the IA SVU MOA to require a detailed review of
the contents of all test procedures and all Level V requirements; that an in-depth NASA
business process and IEMP software application subject matter expertise be applied to
verify that every test adequately addresses the requirements to which it is linked; and that
every Level V is objectively testable. The amount of funds available for IA associated
with IEMP projects is not sufficient to allow for a full IV&V of this type. Ina full IV&V
Process, it might be possible to leverage several NASA business processes and [EMP
applications experts to review all test procedures for subject and system accuracy;
however, the costs of such a review may outweigh the benefits. The intent of the MOA
established for the SVU project was to leverage [A resources to ensure that the SVU
Project adhered to IEMP quality assurance processes. This includes, but is not limited to,
recording requirements traceability for all requirements within the appropriate tool,
creating test plans in the appropriate tool sufficient to cover all testable requirements, and
following test execution and defect Management procedures. We never expected that the
subject matter content of all tests and all requirements be cross-validated,

Report No. IG-07-031

See page 22,
third bullet.

51


APPENDIX B

Date of
E-mail

Issue

NASA
Response

Action Taken on SVU

5 June 21, 2006

Entry and exit criteria
were not met before
conducting Delta TRR
for SIT Pass One

Agreed

No Action Taken - Although
management stated they executed
follow-on actions that addressed our
concerns, test procedures were not
adequately documented prior to testing
and a review of test procedures was not
completed.

6 | July 26, 2006

Entry and exit criteria
were not met before
conducting Delta TRR
for SIT Pass Two

Agreed

No Action Taken - They relied on the
Agency’s subject matter experts to
document appropriate test procedures,
ensure traceability of test procedures to
SVU requirements, and provide expert
consultation on an ongoing basis to
members of their governing bodies so
that they may make informed decisions
at key decision points, primarily the
TRRs.

7 | August 7, 2006

Migration of Sales
Order Data

Agreed

No Action Taken - Although they
established a parallel test environment
to test migration of sales order data, the
data migration process for migrating
sale orders to the new version of SAP
was not completely validated.

8 September 7,
2006

IA analysis was
incomplete

Partially
agreed

No Action Taken - In the future, the IA
contractor will use an LEMP issues-
tracking system to officially log IA
issues and track their progress through
closure.

9 | October 6,
2006

Follow Up on IA
Analysis

Did not agree

No Action Taken - The current MOA
specifies that reviews of requirements
coverage and test plans be delivered
before the start of testing. In addition,
they plan to perform structured testing
throughout the project life cycle, not
just toward the end of the project.

10 | September 18,
2006

Entry and exit criteria
were not met before
conducting TRR for
SIT Pass Three

Partially
agreed

Partial Action Taken - Although they
stated that they would provide explicit
risk management documentation for any
high severity defect that did not have a
workaround, the risk assessment
concerning travel manager was not
correct. In addition, the Competency
Center Director did not agree to test
workarounds in SIT Pass Three.

ReEPoRT NO, IG-07-031

31


APPENDIX B

Date of NASA
E-mail Issue Response Action Taken on SVU
11 | September 29, | Adequate testing and Partially Partial Action Taken - Although not
2006 defect management agreed stated in the response to the issue, the
policies and procedures SVU Project Office took partial action
not developed or on this issue. The response stated, in
implemented the future, they are piloting a form of
Agile Scrum software development
project management called Scrum and
creating a Software Development
Operations Procedures document.
12 | October 18, ORR was held Agreed No Action Taken - Although the SVU
2006 prematurely Project Office held two Authority to
Proceed decision meetings, a
presentation of the entire view of all
actual system characteristics to be
implemented was not accomplished.
13 | October 31, Regression testing not | Did not agree | No Action Taken - Although SVU
2006 completed Project Office’s continued testing
through November 10, 2006, to close
remaining defects, a successful
regression test was not accomplished.
14 | October 31, Testing and data Agreed No Action Taken - Although the SVU
2006 migration not complete Project Office held two Authority to
and effects still Proceed decision meetings, a
unresolved presentation of the entire view of all
actual system characteristics to be
implemented was not accomplished.

32

1. Backout Plan. On June 5, 2006, we informed the IEM Program Director that the
SVU Project Office had not prepared a backout plan. We recommended that the SVU
Project Manager develop a Business Continuity Plan outlining specific procedures to
resume operations should the SVU project encounter a major setback. On June 15, 2006,
the Competency Center Director concurred with our recommendation and stated that
management added a backout plan to the existing Competency Center Release 7.1
Cutover Plan that addressed steps to be taken in the event something occurred during the
upgrade that prevented go-live. In addition, the Competency Center Director had created
a new 7.1 Cutover Plan that outlined steps to be taken to perform year-end close activities
on the current version of the Core Financial system should the contingency option be
invoked.

2. Preparation of Test Contingency Plans. On June 5, 2006, we informed the IEM
Program Director that the SVU Project Office had not prepared test contingency plans.
We recommended that the SVU Project Office develop test contingency plans to ensure
that risks are identified for each phase of testing and establish procedures to address each
tisk. The contingency plans should establish the criteria to be used to determine whether
the risks identified can be mitigated or sufficiently corrected to allow testing to continue.
On June 15, 2006, the Competency Center Director partially agreed, stating that
management believed they had already established a sound process that met the intent of

Report No. IG-07-031


APPENDIX D

However, we agree that schedule pressures have sometimes negatively impacted a team’s
ability to completely finalize all test procedures in a timely manner, The Competency
Center will examine its Release Planning procedures and, in conjunction with the benefits
of testing early-and-often afforded by Agile Scrum, work on improving its ability to

complete all test procedures and lock down the testing baseline prior to the start of each
SIT pass,

3. The IEM Program Director should require the project manager to
implement policies and procedures establishing the management, assignment, and
change controls of defect severity levels.

MSFC Response: Concur. The IEMP Competency Center has already taken steps to
improve the controls surrounding the management of defect severity levels, A new
security role was established in Quality Center (EMP’s testing tool) that permits only
specified management to change defect severities. The Competency Center is in the

Process of establishing a set of standard policies to be used for defect. severity
management.

4. The IEM Program Director should require the project manager to report all
deviations from test procedures.

MSEC Response: Concur. The IEMP Competency Center agrees to include any test

procedures marked as not applicable in the test metrics and report on deviations from the
test baseline.

5. The IEM Program Director should require the Project manager to
successfully complete a regression test.

MSFC Response: Concur. The IEMP ‘Competency Center will establish a baseline of
test procedures prior to cach SIT pass and will ensure that all procedures have been
Successfully tested, or ensure that a documented workaround exists for any failed tests,
prior to system deployment.

6 The IEM Program Director should require the project manager to not hold
the ORR until testing is completed and defects are resolved.

MSEC Response: Partially concur. The IEMP Competency Center agrees that an ORR
should not be held until testing is completed. We do not agree that all defects must be
resolved prior to entering the ORR. NASA Procedural Requirement 7123.1A, “NASA
‘Systems Engineering Processes and Requirements” entrance criteria for the ORR
specifies that “Test failures and anomalies from validation testing have been resolved and
the results incorporated into all supporting and enabling operational products.” While the
TEMP Competency Center understands the intent of this statement, particularly with
Tespect to space flight systems, it does not seem feasible or cost effective to force
resolution of every individual defect related to an administrative software application
Prior to conducting the ORR. The IEMP Competency Center utilizes a set of rigorous

REPORT NO. IG-07-031

47


20

RESULTS

Other Issues. As of September 2007, the SVU Project Office was still stabilizing the
SVU and deploying system patches to resolve issues known before implementation and
issues that surfaced after implementation.

¢ The process for transitioning approximately 9,500 sale orders to the new version
of SAP was not completely validated prior to go-live. Center personnel stated that
additional corrective actions were scheduled to take place during the third quarter
of the fiscal year (FY 2007) to correct the reimbursable data. As of September 5,
2007, sales orders and other accounts receivable issues continued to be
problematic. For example, bills with multiple line items were not posting
correctly.

e As of July 18, 2007, Goddard, Langley Research Center, and Stennis Space
Center could not distribute budget authority due to missing Budget Control
System documents.

e Resource managers for one project at Goddard reported that, for several months
after go-live they could not readily identify the amount of funds available for their
project due to the delayed establishment of FY 2006 and FY 2007 reimbursable
orders in SVU. The project’s funds reconciliation was not completed until
May 2007. As of September 2007, the project’s funds were approximately
$328,000 out of balance. The project’s resource managers were told by
accounting personnel that the funds would remain out of balance because the Core
Financial system was unable to adjust the prior year sales order.

Testing Inadequacies Increased Risk. By not stabilizing requirements prior to
developing test procedures, the SVU Project Office increased the risk that software
requirements were not validated during testing and that software and interfaces might not
perform as anticipated. By not having polices and procedures to adequately manage
defects and corrective actions, the SVU Project Office increased the risk that defect
correction resources were inappropriately prioritized, which left performance-critical
defects not fixed. Also, by not documenting the testing results and identifying key risk
areas, such as deviations from the Test Plan, the SVU Project Office did not always
present an accurate status of testing and the risks of implementing the system.
Inadequacies of procedures, policies, documentation, and testing increased the risk that
as-yet unidentified software defects could impact financial reporting and management’s
ability to rely on the financial information.

SAP Escalation Team. In January 2007, as the number of issues increased and their
impact became apparent, the Competency Center Director and the IEM Program Office
notified SAP that NASA was having significant problems with the upgrade. On

March 15, 2007, SAP deployed an on-site escalation team to the Competency Center with
the objective of better understanding NASA’s issues and marshaling corporate resources
to accelerate the development and delivery of solutions. The SAP Escalation Manager
stated that SAP provides mission critical support whenever there are serious problems
with serious impacts. According to the SAP escalation team, when they arrived on site at

Report No. IG-07-031


APPENDIX D

+ _ improved program/project management information for decision making; and
+ streamlined Year End processing (starting with FY2007 year end processing).

(4) Page 2, ,"Introduction/Background”, footnote number 6:
“An error, flaw, mistake, failure, or fault ina computer program that prevents...”

This definition of test defect does not adequately depict the defect categories from an
Integrated Enterprise Management Program (IEMP) perspective. The definition
described in the report implies we only track computer program errors, while the IEMP
tracks many other types of defects/root causes (i.e., user procedures, user errors,
Tequirement changes, security, and environment issues, ctc.). The definition, along with
the number of defects reflected in the report presents a false assumption that all issues
were catastrophic and coding/software in nature.

(5) Page 9, “The Results” section, third paragraph:

“During an August 23, 2006 meeting held with the Competency Center Director and the
SVU Manager, the SVU Project Manager stated it was their practice...""

It is IEMP’s standard practice to establish a baseline of test plans and sets prior to System
Integration Testing (SIT), as well as steps associated with each test plan. The SVU
Project Manager’s (PM) statement was intended to communicate that testers are
encouraged to execute tests beyond the baseline if time permits.

(6) Page 16, “The Results” section, third paragraph:

“ORR Held Prematurely. The SVU Project had the ORR on October 11, 2006, even
though testing was not complete and some defects were still unresolved. ..”"

The report states that “not all of the workarounds needed for go-live had been provided.
Those workarounds were not provided until November 16, 2006." The OIG's assertion is
not correct, because the project team provided the referenced workarounds prior to
November 16, 2006, as they were developed in the various reviews and certainly at the
Authority to Proceed checkpoints. The team performed extensive analysis on the impacts
of the open items against specific criteria, Perhaps the OIG is referring to the “job aid” or
“one pager” that we provided to the Centers that listed all the workarounds. This was an
action item intended to assist Change Management with their communications and was
not a pre-November 16, 2006, requirement.

(2) Page 17, “The Results” section, fourth paragraph:

“What the contractor reported in the Requirements Traceability Report was that the
traceability of the Level V requirements would be postponed..." and

Page 19, “The Results” section, first full paragraph:

A definition
of “defect”
was added
(see footnote
9 and page
42).

See page 22,
first bullet.

See page 22,
second
bullet.

Report No. IG-07-031


OVERVIEW

Competency Center Director! and informed them of issues we identified as the SVU went
through its three SIT Passes. Specifically, between June and October 2006, we informed
the IEM Program Director, SVU Project Manager, and the Competency Center Director
of 14 issues that might negatively impact SVU testing and implementation. Of the

14 issues identified, they agreed or partially agreed with 12 issues.

Of the 12 issues agreed with, they took action on 1 issue and took partial action on

2 issues. Specifically, the SVU Project Office took action to incorporate a backout plan”
into their contingency planning requirements. The SVU Project Office took partial action
by reporting deviations in testing to stakeholders and reporting test procedures that were
not completed. The SVU Project Office did not take action on the other 9 issues because
there was either insufficient time or they did not fully agree with our concerns.

The IEM Program Director, the SVU Project Manager, and the Competency Center
Director did not agree with 2 issues. Specifically, they did not agree with our
assessments concerning the adequacy of the independent assessment performed by the
Independent Assurance contractor or whether a successful regression test was conducted.
(Appendix B provides a more detailed explanation of the issues and management’s
responses.)

In summary, we found that although the SVU Project Office developed a Test Plan in
May 2006 to provide guidance for conducting SITs, that plan was inadequate and the
execution of the plan was undisciplined. Specifically, we found the following:

¢ SVU software requirements were not adequately stabilized before the SVU
Project Office developed the Test Plan and test procedures. According to NASA
Procedural Requirements (NPR) 7150.2, “NASA Software Engineering
Requirements,” September 27, 2004, paragraph 3.4, testing should verify code
against requirements and design to ensure that the requirements are implemented.
Although a baseline of SVU requirements was established in May 2006, those
software requirements were not stabilized until September 2006, 4 months after
SIT began in May 2006.

The Test Plan did not include a baseline of test procedures to be tested during
each SIT, as required by NPR 7123.1, “NASA Systems Engineering Processes
and Requirements,” March 13, 2006, Table G-6, and NPR 7150.2, paragraph
5.3.2.1. The SVU Project Manager never established a baseline of test procedures
to be executed during each SIT. Because a baseline of test procedures was not
established for each SIT, we were not able to determine whether all testing that
should have been completed was completed during each SIT Pass.

' The Competency Center Director’s official title is Deputy Director, Office of the Marshall SpaceFlight
Center Chief Information Officer.

> A backout plan is a documented plan to restore the system to the state it was in prior to installation of new
software, including assuring the restoration is complete and accurate. The plan usually involves
re-installing the previous versions of the software.

Report No. IG-07-031


APPENDIX B

completely defined early in the project lifecycle. In our IEMP projects, we are
unlikely to get all the requirements during blueprint, which means that we are always
playing catch-up as the project progresses. As a result, once the project plan indicates
that it is time to begin the detailed software configuration and/or custom development
effort, there is often a realization that detailed requirements had not yet been finalized
for all process areas. Work begins on the areas where requirements are more
solidified, but is postponed for areas of greater uncertainty. By the time SIT 1 begins,
it is not possible to have all test plans completed, because lower level requirements,
and/or functionality of the COTS [commercial off the shelf] solution, are still being
discovered. The project team makes its best effort to finalize test plans to ensure
testing of known functionality during SIT 1 and during subsequent SIT passes.
However, as seen with SVU, the team is typically not able to fully complete this task
before heading into the testing cycle.

Our intent is to use the Scrum approach to improve our ability to deliver fully tested
software that better meets the needs of the user. As part of our pilot effort, we are
creating a Software Development Operations Procedures document. This document

describes the overall design and development process, the team structure, testing
procedures, and periodic reviews.

During the development of our Software Development Operations Procedures
document, we will review the policy and procedure recommendations provided by the
IG in Recommendations 1 — 3 to determine if they should be included in that
document.

In response to recommendation No. 4, the project team has reviewed the high and
very high defects that remain open for the SVU project and reported all such defects
in detail during the project Operational Readiness Review and will continue to do so
at follow-on Authority to Proceed (ATP) checkpoints leading up to go-live.

12. ORR. On October 18, 2006, we informed the IEM Program Director that the SVU
Project Office held the ORR on October 11, 2006, even though review materials were not
supplied in a timely fashion, testing for the SVU was not completed, and test defects were
not resolved. We recommended the following:

¢ The IEM Program Director should require the SVU Project Office to continue
testing until all test procedures have been completed and defects have been
resolved. The defect resolution should include documented fixes, related system
limitations, needed operational controls, and related user workarounds.

¢ The IEM Program Director should require that the SVU Project Manager conduct,
after completing all testing and resolution of all defects, a Delta ORR with the
complete set of ORR documentation materials provided to participants

sufficiently in advance of that review to allow participants adequate time for
preparation.

We received management’s response on November 7, 2006. The Competency Center
Director agreed, stating that the SVU Project Office continued efforts to resolve and test
remaining defects beyond the completion of SIT Pass Three. At the ORR, the team
presented a complete status of open defects including severity level, functionality impact,
business impact, and workarounds in the event the defect remained open at go-live. The

RePorT No. IG-07-031

39


APPENDIX B

9. Follow-Up on IA Analysis. On October 6, 2006, we informed the IEM Program
Director that the IA contractor did not complete or report on agreed-to tasks in support of
key deliverables and did not identify SVU Project Office test planning weaknesses. We
recommended that the IEM Program Director verify how much money was paid for work
that the IA contractor did not perform and determine whether any of those funds could be
recouped. We also recommended that the NASA IV&V Facility Director conduct a
review of all IEMP projects to determine whether policies and procedures were being

implemented and followed to ensure the independence of the work being performed by
IV&V personnel.

We discussed those issues with the IV&V Facility Program Director and IV&V Project
Manager during a teleconference on October 19, 2006. On October 23, 2006, the IV&V
Facility Project Manager responded directly to the OIG, stating that we should contact the
IEM Program Director and confirm whether he believed work was not performed by the
IA contractor. The IV&V Facility Project Manager stated that if the IEM Program
Director believed that the IA contractor did not perform the required work, then IV&V
Facility would pursue recouping the cost. The IEM Program Director did not comment
on this issue and has made no effort to recoup costs associated with the IA.

We received management’s response on October 23, 2006. The Competency Center
Director disagreed with our conclusions and recommendation, stating that the IA
contractor noted that on several IEMP projects, the project teams struggle to have all
requirements covered prior to the start of SIT and that test plans tend to be less than
stable as testing approaches. Based on the IA Team’s observations and the Competency
Center’s own efforts to improve processes, several steps were being taken to address
these inherent project challenges. Specifically, the latest MOA between the IEM Program
and the NASA IV&V Facility specifies that reviews of requirements coverage and test
plans be delivered earlier, prior to the start of testing. With earlier delivery, project teams
would have better and more timely insight into whether they are following procedures. In
addition, the Competency Center was in the process of attempting to shift toward agile
methodologies that will reduce risk by using proven requirements solicitation techniques

and by performing structured testing throughout the project life cycle, not just toward the
end of the project.

10. TRR for SIT Pass Three. On September 18, 2006, we informed the IEM Program
Director that SVU Project Office management, with input from key decision makers,
decided to proceed with SIT Pass Three for SVU even though the SVU Project Office did
not meet established testing entry and exit criteria concerning data migration and open
defects. We recommended that the IEM Program Director require the SVU Project
Manager to test and document workarounds for unresolved high severity software defects
as part of SIT Pass Three, where possible. We also recommended that the IEM Program
Director require the SVU Project Manager to provide explicit risk management
documentation for any high severity defect that did not have a workaround and include
the likelihood and possible impact of the defect on SVU functionality.

Report No. IG-07-031 37


RESULTS

¢ Level V — implementation requirements (requirements related to individual
system activities or information delivery components, and these requirements are
defined in the use cases or functional specifications).

Test procedures—the specific steps for testing software—are based on Level V
requirements, i.e., software requirements.

The SVU Project Office did not stabilize the Level V requirements until SIT Pass Three.
On May 26, 2006, the SVU Project Office established a baseline of software requirements
and tracked the changes to the baseline on a weekly basis. Although some volatility in
requirements is expected, even after a successful technical requirements definition
process, high volatility indicates that the system and its attributes are not well understood.
From May 26, 2006, approximately 3 weeks before SIT Pass One began, through
November 3, 2006, approximately 2 weeks after SIT Pass Three ended, the SVU Project
Office created 3,321 new Level V requirements, changed 744 Level V requirements, and
deleted 2,278 Level V requirements. On November 3, 2006, the SVU Project Office had
a total of 7,898 Level V requirements documented. The figure shows the volatility of the
Level V requirements on a weekly basis.

Requirements Volatility

16.00%

14.00%

12.00%

10.00%

8.00%

Level V Requirements Volatility

6.00%
|
| 4.00%
i 2.00%
soo re -
ese2s 38 23 eee2ees 2
8238 gesesee2e2e82eee8e888
| ag 8 SESSSERSSSSSESEES
° ss ERE SFSSSF°SSS2SS8 8 =
Date of Snapshot

The volatility trend (sloped line in the figure) shows that the SVU Project Office was still
identifying requirements well into testing and the requirements had not stabilized until
SIT Pass Three. Stabilization should have occurred before SIT Pass One began.
According to NPR 7150.2, paragraph 3.4, it is critical to have software requirements
traced to test procedures to ensure that requirements are tested. By tracing software
requirements to testing procedures, every time a requirement is changed due to

Report No. IG-07-031


APPENDIX D

Enclosure

Additional NASA comments and recommendations on the OIG Draft Audit Report
entitled “System Integration Testing of the Systems, Applications and Products Version
Update Project Needed Improvement” (Assignment No. A-06-017-00):

() Page iii, “The Issue” section, last paragraph:

“As a result, SVU system implementation was delayed until November 20, 2006, ..."" and

Page 4, “Introduction/Objectives” section, last paragraph:

“As a result, system implementation was delayed until November 20, 2006, .."*

The conclusion that issues with testing caused the change to the Release 7.1 cutover
baseline timeline is inaccurate. We discussed the explanations for changes from the
baseline Release 7.1 cutover timeline with project governance as part of the Delta Test
Readiness Review September 7-8, 2006, and provided the explanations to the OIG
previously on this matter.

(2) Page iii, “The Issue” section, last, ‘Paragraph:
“As a result, SVU system was delayed until November 20, 2006..." and
Page iii “The Issue” section, footnote number 3:

“The Core Financial System was available to a limited number of users for limited
Processing on November 15, 2006, ...”

SVU implementation occurred on November 15, 2006, for existing funds; November 20,
2006 was the implementation date for new fiscal year funding transactions. The footnote
is irrelevant because these dates were part of the planned implementation approach and

occurred in accordance with the established plan.

(3) Page 1,” Introduction/Background” section, third paragraph:

SSAP Version Update Project. The IEMP initiated the SAP Version Update in September
2005 t0 update from ...""

The benefits listed in the report are not consistent with the benefits in the SVU Scope

Document, Revision A, dated April 5, 2006, and should be noted as follows, The SVU
Project key benefits include

+ _ improved data integrity based on SAP Funds Management (FM) redesign;
* additional automation of adjustment accounting;
+ improved support for the Budget Distribution process;

Report No. IG-07-031

See page 21,
last bullet.

See page 21,
last bullet.

See page 21,
last bullet.

See page 21,
last bullet.

Text was
revised in
response to
this
comment
(see page 1)

49


RESULTS

During an August 23, 2006, meeting held with the Competency Center Director and the
SVU Manager, the SVU Project Manager stated it was their practice not to establish a
baseline of test procedures to be executed during each SIT. Because of that practice, we
were not able to determine whether the SVU Project Office finalized the planning for
each SIT Pass or whether all testing that should have been completed was completed
during each SIT Pass. After the SVU was implemented, we were able to determine the
actual number of test procedures that were tested during each SIT Pass. For example, we
found that on June 26, 2006, the day that SIT Pass One was scheduled to start, only

56 percent of the 693 test procedures that were eventually completed during SIT Pass One
were documented in the Quality Center,'? thus preventing the SVU Project Office from
establishing a baseline.

As a result of not establishing a baseline of test procedures before conducting each SIT
Pass, SVU Project management could not ensure that testing adequately covered the
system requirements and could not perform an objective review of the results of the
testing. In addition, a comprehensive review of the test procedures could not be
performed. In a comprehensive review, functional teams review other functional teams’
test procedures to identify testing inadequacies or gaps that involve interfaces and mutual
dependencies. A comprehensive review will also identify if the same test has been
developed multiple times by various functional teams. Industry best practices support
performing a comprehensive review of test procedures.

Also, the SVU Project Office could not identify all possible testing dependencies between
test procedures that had not yet been completely defined. Identifying testing
dependencies involves recognizing that some test procedures are dependent on the
completion of a prior test. If testing dependencies had been fully identified, the SVU
Project staff could have more accurately prioritized resources to fix test defects and
rescheduled test procedure that were dependent on fixing a specific defect.

We notified the IEM Program Director about inadequately documenting the baseline of
test procedures in an e-mail on June 9, 2006. In his June 19, 2006, response, the IEM
Program Director agreed and stated,

We concur that it is a sound process to have all test . . . [procedures] developed and
traced to project requirements prior to entering the TRR. While this may no longer be
possible for SVU due to the schedule that has already been established, the IEM Program
will take the action to adjust its standard Test Plan Approach to ensure that it requires test
plan development and tracing to requirements to be completed prior to a project’s TRR.

In another e-mail on June 9, 2006, we notified the IEM Program Director about the
inability to adequately review the test procedures because the baseline of test procedures
was not adequately documented. In his June 19, 2006, response, the IEM Program
Director agreed and stated

* Quality Center is an automated tool used by the SVU Project Office to document detailed test procedures
that included steps and the expected results. Quality Center is also used to document the results of
testing.

ReEPorT No. IG-07-031


SEPTEMBER 28, 2007

OVERVIEW

SYSTEM INTEGRATION TESTING OF THE SYSTEMS,
APPLICATIONS, AND PRODUCTS VERSION UPDATE
PROJECT NEEDED IMPROVEMENT

The Issue

NASA has made substantial investments of both time and money in developing and
implementing the Core Financial system—the backbone of the Integrated Enterprise
Management Program (IEMP). NASA estimates the cost of this effort, through fiscal
year 2008, to be about $998 million. IEMP initiated the Systems Applications and
Products’ (SAP) Version Update (SVU) Project in September 2005 to update from SAP
R/3 SAP 4.6c to mySAP Enterprise Resource Planning 2005/Enterprise Core Component
6.0 (mySAP ERP 2005/ECC 6.0).

System integration testing (SIT) is done to ensure that related systems (such as the Core
Financial system, Contract Management Module, Travel Manager, Business Warehouse,
and the Bank Card system) operate in accordance with the process design specifications
for all business functions and that the parts of the system work together as an integrated
whole. The SVU Project Office developed the 7.1 System Integration Test Plan (Test
Plan), May 18, 2006, to provide guidance on conducting SIT for the SVU and began SIT
in May 2006. For the SVU project, SIT tests business processes, system functions
(standard and custom), application security, and user procedures. Paragraph 5.1 of the
Test Plan sets up SIT for SVU in three “Passes” or iterations of testing to validate that the
software correctly implements the software requirements and design as each software
component is incrementally integrated with each other.

The overall objective of our audit, which began in May 2006, was to determine whether
testing for the SVU Project was adequately planned and performed. During the audit, we
attended key SVU Project meetings including all Test Readiness Reviews (TRR),
Operational Readiness Reviews (ORR), Authority to Proceed decision meetings, and

defect management meetings. Details of the audit’s scope and methodology are in
Appendix A.

Results

During the audit, we maintained continuous communication with the Integrated
Enterprise Management (IEM) Program Director, the SVU Project Manager, and the

Report No. IG-07-031


APPENDIX B

a

IssuES COMMUNICATED DURING
THE AUDIT

During the audit, we attended key SVU Project meetings including TRRs, the ORR,
Authority to Proceed decision meetings, and defect management meetings. For the TRRs
and the ORR, the SVU Project Office established testing entry and exit criteria in the Test
Plan, Test entry criteria are used to ensure that all necessary components, from detailed
planning aspects to environment and technical architecture, are available to support
productive, effective test execution. Test exit criteria are used to ensure successful
completion of each pass of testing. Throughout the course of the audit, we informed the
TEM Program Director, the SVU Project Manager, and the Competency Center Director
of 14 issues discussed in this report. Of the 14 issues identified, they agreed or partially
agreed with 12 issues and took action on 1 issue and partially took action on 2 issues.
The 14 issues and management’s actions are provided in the table.

Date of NASA
E-mail Issue Response Action Taken on SVU

1 June 5, 2006 Backout plan not Agreed Took Action - The SVU Project Office
prepared added backout plan to the existing
Competency Center Release 7.1
Cutover Plan.

2 | June 5, 2006 Test contingency plans | Partially No Action Needed - The SVU Project
not prepared agreed Office stated they had already defined
contingency test plans for areas of
highest risk from an integration
perspective, outlined a process in the
Test Plan that dealt with issues as they
occurred during testing, and defined
very specific exit criteria for each SIT
Pass.

3 | June 9, 2006 Test procedures were Agreed No Action Taken - Although

not adequate and did management agreed with our

not trace to project assessment, test procedures were never
requirements adequately documented in any of the
SIT Passes.

4 June 9, 2006 Test procedures not Agreed No Action Taken - For future projects,
being reviewed and they agreed to adjust their standard test
analyzed prior to TRR plan approach to ensure that test
procedures were developed and
reviewed before the TRR and planned
to consider implementing a process that
enabled baselining test procedures and
tracking changes that occur after the
baseline date.

Report No. IG-07-031


RESULTS

¢ the IA contractor’s test plan review was sufficient to ensure that they adequately
covered requirements,

e there was “little value for further review to determine objective testability of
requirements, since at the current time the project has now entered the final test
cycle,” and

© aclose, proactive working relationship with the IV&V representative provided the
greatest value to their efforts.

We agree that little value would have been gained by the IA contractor determining
whether requirements were objectively testable in September 2006. (That the IA
contractor did not determine objective testability of requirements before the project
entered the final test cycle is an example of an IA benefit that was not realized.)
However, the review of a judgmental sample of 10 out of a total of 4,127 test procedures
could not provide confidence that all appropriate requirements were adequately tested.
After reviewing management's response, we reiterated our findings in an e-mail on
October 6, 2006, and suggested that the IEM Program Director should verify how much
money was paid for work that the IA contractor did not perform and determine whether
any of those funds could be recouped. The IEM Program Director did not comment on
this suggestion.

Software Deficiencies Were Not Fully Identified

Although the SVU Project Office implemented the SVU on November 20, 2006, Project
staff failed to identify, prior to implementation, significant software deficiencies that had
to be corrected after the system was implemented. Implementing the system with known
software deficiencies increased the risk that as-yet unidentified software defects could

impact financial reporting and management’s ability to rely on this financial information.

Travel Manager Interfaces. The SVU Project Office did not identify software
deficiencies concerning Travel Manager and SAP interfaces during testing. Travel
Manager interfaces with SAP to record funds used for travel. The interface between
Travel Manager and SAP should have been completely tested but was not. The SVU
Project Office identified a high severity software defect concerning multiple vendor
records in Travel Manager, but that defect was not fixed and retested prior to go-live. By
not fixing and retesting that defect during the regression test, the SVU Project Office did
not identify additional software deficiencies that resulted in an unusually large number of
transaction errors in Travel Manager involving multiple vendor records and causing due
dates to occur before the posting date. As a workaround, staff at each Center manually

input travel data into SAP. However, payments of travel vouchers were significantly
delayed because

* approximately 3,200 travel vouchers failed processing from November through
December 2006,

18 Report No. IG-07-031


Final Cy released Yi,

LG Hla

Col R. Klemstine
Assistant Inspector General for Auditing

Acronyms

FY Fiscal Year

IA Independent Assessment

IEEE Institute of Electronic and Electrical Engineers
IEM Integrated Enterprise Management

IEMP Integrated Enterprise Management Program
IV&V Independent Verification and Validation
MOA Memorandum of Agreement

NPD NASA Procedural Directive

NPR NASA Procedural Requirements

OIG Office of Inspector General

ORR Operational Readiness Review

SAP Systems Applications and Products

SIT System Integration Testing

SVU SAP Version Update

TRR Test Readiness Review

Report No. IG-07-031


42

APPENDIX C

GLOSSARY

Data Conversion. The Competency Center Director defines data conversion as the initial
SAP/Core Financial (4.6C) project activity of converting data from Center legacy
financial systems to SAP. This process included the development of load programs to
export and transform legacy data to an intermediate format that could be imported into
SAP data structures. As part of the data conversion process, the Agency financial
community had to make decisions about how much history to convert, appropriate levels
of data summarization (as opposed to converting lowest levels of transaction details), etc.

Data Integrity. The Competency Center Director stated data integrity refers to the
quality of correctness, completeness, wholeness, soundness, and compliance with the
intention of the creators of the data. It is achieved by preventing accidental or deliberate
but unauthorized insertion, modification, or destruction of data in a database by an end
user or the transaction processing software logic. With regard to the SVU project, this
term is specifically used in relation to the migration of data from 4.6C to mySAP ERP
2005/ECC 6.0 at go-live as verified by experts within the Chief Financial Officer
community and then during stabilization as performance of the new software was

monitored to ensure that transactional processing produced the expected data integrity
results.

Data Migration. The Competency Center Director defines data migration as the process
required to move data from SAP 4.6C database table structures to mySAP ERP
2005/ECC 6.0 database table structures. This activity was necessary due to significant
structural changes SAP made to its initial Joint Financial Management Improvement
Program (now Financial Systems Integration Office) certified version (4.6C) in an effort
to improve/expand the capabilities in subsequent versions (4.7 and mySAP ERP
200S/ECC 6.0). In some cases, new data elements were added that did not exist in 4.6C:
in other cases the relational table structure was changed to support various improvements
in functionality and performance.

Data Transition. The Competency Center Director stated that the term data transition
was used specifically to describe the activity to close out open sales orders
(reimbursables) in 4.6C and re-open them in mySAP ERP 2005/ECC 6.0. It was not a
conversion or a migration.

Defect. Software defects take different names in different organizations, e.g., errors,
issues, bugs, defects, or incidents. Whatever they are called and whatever form they take,
defects can have a serious detrimental impact on the development phase and can continue
to affect the product through its maintenance phase. Discussions within the software
development community consistently recognize that most failures in software products
are due to errors in the specifications or requirements.

Report No. IG-07-031




APPENDIX B

Project Manager to conduct another Pass Two Delta TRR after meeting the following
conditions:

¢ solidify and conduct a documented review of test procedures to include tracing
requirements,

¢ resolve significant technical issues surrounding data migration, and

¢ provide TRR participants with briefing charts and access to test procedures,
allowing them adequate review time prior to conducting the Delta TRR.

We received management’s response on August 9, 2006. The Competency Center
Director agreed, stating that Competency Center staff relied on the Agency’s subject
matter experts to document appropriate test procedures, ensure traceability of test
procedures to SVU requirements, and provide expert consultation on an ongoing basis to
members of their governing bodies so that they may make informed decisions at key
decision points, primarily the TRRs. In addition, the SVU Project Manager stated that
Project Office staff would continue to work with SAP to resolve technical issues
associated with data migration as they surfaced during testing.

7. Migration of Sales Order Data. On August 7, 2006, we informed the IEM Program
Director that the SVU Project Manager, with input from other key decision makers,
decided to proceed with testing even though established entry and exit criteria concerning
data migration were not met. According to the Test Plan, the entry criteria concerning
data migration is “Data migration programs scheduled for execution have been unit tested
in a test environment,” and the exit criteria is “Data migration has successfully executed

and been validated according to previously documented validation and reconciliation
procedures.”

Prior to the Delta TRR on June 12, 2006 the SVU Project Office contracted with a SAP
programming expert to assess and correct issues with the special ledger and sales orders
accounts. During the SIT Pass One Delta TRR on June 12, 2006, the SVU Project Office
reported the special ledger and sales orders accounts did not migrate successfully during
unit testing. During the SIT Pass Two Delta TRR on July 18, 2006, the SVU Project
Office reported that significant issues surrounding the special ledger accounts were
corrected. However, issues with migrating sales orders were not addressed and a new
method for migrating the sales orders needed to be developed. The custom developed
program to migrate sales orders was expected to be ready on or about September 20,
2006. The program will not be ready for testing until the end of SIT Pass Three. We
recommended that the IEM Program Director direct the SVU Project Manager to resolve

significant technical issues surrounding migration of sales order data before continuing
with testing.

We received management’s response on August 22, 2006. The Competency Center
Director did not clearly specify whether he agreed with our recommendations. The
Competency Center Director stated that Competency Center staff did not fully understand

Report No. IG-07-031


RESULTS

the Competency Center, they found NASA had major issues with financial reporting and
some of the SAP data correction tools. According to the Competency Center Director,
the escalation team will continue to operate until NASA and SAP jointly agree that all
priority issues have been resolved and that the Core Financial system is stable. As of
September 5, 2007, NASA was still trying to stabilize the Core Financial system.

Management’s Comments on the Finding and Evaluation of
Management's Comments

Management’s Comments on the Finding. In response to a draft of this report, the IEM
Program Director provided comments on various issues discussed in the finding. The
TEM Program Director stated the following:

¢ The SVU implementation occurred on November 15, 2006, for existing funds and
November 20, 2006, for FY 2007 funds and that those dates were in accordance
with their established plan. The explanation for changes to the SVU
implementation date was discussed with the Financial Steering Group as part of
the Delta TRR on September 7-8, 2006.

¢ Itis IEMP’s standard practice to establish a baseline of test plans and sets as well
as steps associated with each test plan prior to SIT. The auditors misinterpreted
the SVU Project Manager’s statement about not establishing a baseline of test
procedures to be executed during each SIT. What she intended to communicate
was that testers are encouraged to execute tests beyond the baseline if time
permits.

¢ The project team provided all of the workarounds needed for go-live prior to
November 16, 2006, as they were developed in the various reviews and at the
Authority to Proceed checkpoints (October 25, 2006, and November 10, 2006).

¢ The OIG appears to interpret the MOA to require a detailed review of the contents
of all test procedures and all Level V requirements, to verify that every test
adequately addresses the requirements to which it is linked, and to verify that
every Level V requirement is objectively testable. However, the costs of such a
review may outweigh the benefits. The intent of the MOA was to leverage IA
resources to ensure that the SVU project adhered to IEMP quality assurance
processes. The IEM Program Director stated that he never expected the subject
matter content of all tests and all requirements to be cross-validated.

Evaluation of Management’s Comments. In response to the IEM Program Director’s
comments, we are providing clarification to address each of his concerns.

e Although, the IEM Program Director stated that the November 2006
implementation date was part the established plan, the implementation date in the
project timeline presented at the Preliminary Design Review on March 8, 2006,

ReEPorT No. IG-07-031

21


RESULTS

Project Manager, and the Competency Center Director with 14 issues that might
negatively impact SVU testing and implementation. Of the 14 issues identified, they
agreed or partially agreed with 12 issues.

Of the 12 issues management agreed with, they took action on 1 issue and took partial
action on 2 issues. Specifically, the SVU Project Office took action to incorporate a
backout plan into their contingency planning requirements. The SVU Project Office took
partial action by reporting deviations of testing to stakeholders and reporting test
procedures that were not completed. However, the SVU Project Office did not report that
they created, modified, and deleted test procedures during testing. In addition, the SVU
Project Office took partial action by improving controls for assigning and changing defect
severity levels; however, they did not make the process a formal policy and did not
implement a process to escalate high severity defects to very high in compliance with the
Test Plan. The SVU Project Office did not take action on the other 9 issues because there
was either insufficient time or they did not fully agree with our concerns.

The IEM Program Director, the SVU Project Manager, and the Competency Center
Director did not agree with 2 issues. Specifically, they did not agree with our
assessments concerning the adequacy of the IA work performed by the Independent
Verification and Validation Facility contractor or whether a successful regression test was
conducted. (Appendix B provides a more detailed explanation of the issues and
management’s responses.)

Testing Guidance

Institute of Electronic and Electrical Engineers Standards. The Institute of Electronic
and Electrical Engineers (IEEE), a nonprofit organization, publishes nearly a third of the
world’s technical literature on electrical engineering, computer science, and electronics.
NASA Procedural Directive (NPD) 2820.1C, “NASA Software Policy,” August 30, 2005,
recommends using IEEE Standards for software development and implementation. IEEE
Standard 1012, “IEEE Standard for Software Verification and Validation,” June 8, 2005,
provides guidance for a SIT. A SIT validates that the software correctly implements the
requirements and design as each software component is incrementally integrated into the
overall system. To realize the full benefits of the SIT, IEEE Standard 1012, table 1,
sections 5 and 6, recommend tracing requirements to test procedures and to test results
and documenting test execution and results.

NASA Software Testing Requirements. NPR 7150.2 establishes a minimum set of
requirements for NASA software acquisition, development, maintenance, operations, and
management. NPR 7150.2, paragraph 5.1.3, requires, for software created for or acquired
by NASA, a test plan to describe the software test environment to be used for testing;
identify the tests to be performed; and provide schedules for environment, development,
and test activities. The test plan should provide an overview of software testing, test
schedules, and test management procedures. NPR 7150.2, paragraph 3.4.5, requires the
project office to document defects identified during testing and track defects to closure.

Report No. IG-07-031


APPENDIX C

Defect Severity Level. The level of importance assigned to a defect during testing.

Low Severity Level Defect. A problem which may require that a change be
made. It is not critical to the operation of the application and a date for correction
will be negotiated.

Medium Severity Level Defect. A problem which requires that a change be
made by a requested date. If the change is not implemented by the requested date,
the problems caused would not be materially damaging but would cause a serious
impact.

High Severity Level Defect. A problem which requires that a change be made by
a specific date. If the change is not implemented by the specific date, the problem
would materially affect testing causing an immediate and substantial impact.
Failure to implement the change on the specific date would cause the defect to
automatically escalate to very high severity.

Very High Severity Level Defect. A problem that cannot be circumvented, i.e.,
there is no workaround available, and that impacts the operation of the affected

application. The problem requires that a change be made on an immediate-
response basis.

Entry Criteria. Criteria that ensure that all necessary components—from detailed
planning aspects to environment and technical architecture—are available to support
productive, effective test execution.

Exit Criteria. Criteria that ensure successful completion of each Pass of testing and
establish the foundation for the next Pass of testing.

Independent Assessment (IA). A type of IV&V service provided by the IV&V Facility
that includes an analysis of the software development life cycle and/or products,
performed to identify critical software risks that could jeopardize mission safety and/or
success. IA can range from an analysis of a specific subsystem performed at a specific
time during the life cycle to performing assessments of several segments of the system
over the entire life cycle of the development activity.

Independent Verification and Validation (IV&V). An engagement performed to
provide assurances on the verification and validation of a project that is financially,
managerially, and technically independent of the project.

Operational Readiness Review (ORR). A review intended to provide management
with the actual system characteristics and the procedures used in the system or product’s
operation and ensures that all system and support hardware, software, personnel,
procedures, and user documentation accurately reflect the deployed state of the system.

REPoRT No. IG-07-031


APPENDIX C

Unit Test. Procedure used to validate that individual modules or units of source code are
working properly. A unit test is a test for a specific unit. Unit testing will not catch
integration errors, performance problems, or any other systemwide issues. In addition, it
may not be easy to anticipate all special cases of input the program unit under study may
receive in reality. Unit testing is only effective if it is used in conjunction with other
software testing activities.

REPORT No. IG-07-031

45


RESULTS

requirement volatility, the corresponding test procedures should be retraced and also
changed to reflect the new requirement. Consequently, SIT Pass One and, to a lesser
extent, SIT Pass Two were not as effective as they could have been if requirements had
been stabilized and traced to test procedures and if corresponding test procedures had
been kept updated as requirements changed. Therefore, the SIT Passes were performed
with an increased risk that the tests developed did not address all of the SVU
requirements.

We notified the IEM Program Director about our concerns regarding stabilizing
requirements before beginning SIT testing in an e-mail on June 9, 2006. The IEM
Program Director agreed with our assessment and responded,

We are striving to have all low-level design requirements (Level Vs) solidified and
test .. . [procedures] created prior to testing, scheduled to start on June 26. We think we
can achieve this goal. However, there are still a few remaining items to be worked out in
the detailed design of some custom developed objects, as well as in some process areas
(e.g., reimbursable); therefore, it may not be realistic for us to expect zero volatility in our

detailed design requirements as we move into our first System Integration Test execution.

Although the IEM Program Director agreed with our assessment and we agree that some
volatility is expected, requirements for custom developed objects and process areas
should be stable before testing starts.

Documentation and Control of Test Procedures Was Inadequate

The SVU Project Office management did not prepare an adequate test plan. Specifically,
SVU Project management did not establish a baseline of test procedures to be tested
during each SIT, did not adequately report on deviations from the Test Plan, and did not
develop a backout plan as part of their contingency plans.

Test Plan Did Not Include a Baseline of Test Procedures for Each SIT. The SVU
Project Manager did not adequately document a baseline of test procedures in the Test
Plan. NPR 7123.1, Table G-6, requires that, prior to a TRR, “the objectives of the testing
have been clearly defined and documented and all of the test plans, procedures,
environment, and the configuration of the test item support those objectives.” For each
SIT Pass to be effective, the test procedures should be identified prior to the start of
testing so that the results can be objectively measured against specific criteria. Also, once
a baseline of test procedures is established, changes to the baseline should be documented
and reported to key decision makers, in accordance with NPR 7150.2, paragraph 5.3.2.1.
Instead of a baseline of test procedures to be performed during each SIT Pass, the SVU
Project Office established a baseline of test sets.” The test sets included only the name
of the general area that would be tested during each SIT and did not clearly define or
document the test procedures, as required by NPR 7123.1, Table G-6.

" A set of related business events. For SVU, a test set consists of the name of the general area that would
be tested during each SIT.

8 Report No. IG-07-031