U.S. ENVIRONMENTAL PROTECTION AGENCY 

OFFICE OF INSPECTOR GENERAL 

Catalyst for Improving the Environment 

Audit Report 

Improvements Needed in 

EPA’s Efforts to Replace Its 

Core Financial System 

 Report No. 11-P-0019 

 November 29, 2010 
 

 

Report Contributors: Rudolph M. Brevard

 Charles M. Dade

 Andrew Gibson 

Abbreviations 

EPA U.S. Environmental Protection Agency 

FFMSR Federal Financial Management Systems Requirements 

FMLoB Financial Management Line of Business 

FSIO Financial Systems Integration Office 

IFMS Integrated Financial Management System 

IV&V Independent Verification and Validation 

JFMIP Joint Financial Management Improvement Program 

OCFO Office of the Chief Financial Officer 

OIG Office of Inspector General 

OMB Office of Management and Budget 

PAT Product Acceptance Testing 

PMA President’s Management Agenda 

SLCM Systems Life Cycle Management 
 

11-P-0019 

November 29, 2010 U.S. Environmental Protection Agency 

Office of Inspector General 

At a Glance 

Catalyst for Improving the Environment 

Why We Did This Review 

We sought to evaluate whether 

the Office of the Chief 

Financial Officer (OCFO) 

effectively managed the system 

development project to replace 

the U.S. Environmental 

Protection Agency’s (EPA’s) 

core financial system, the 

Integrated Financial 

Management System (IFMS). 

We also sought to evaluate 

whether the system 

development project is 

achieving the desired results. 

Background 

In 1989, EPA implemented 

IFMS as its core financial 

management and budget 

execution system. In 2001, 

EPA began the process to 

replace IFMS. EPA selected a 

commercial-off-the-shelf core 

financial system certified by 

the General Services 

Administration Financial 

Systems Integration Office. 

For further information, contact 

our Office of Congressional, 

Public Affairs and Management 

at (202) 566-2391. 

To view the full report, 

click on the following link: 

www.epa.gov/oig/reports/2011/ 

20101129-11-P-0019.pdf 

Improvements Needed in EPA’s Efforts to 

Replace Its Core Financial System 

What We Found 

OCFO’s management control processes do not ensure compliance with EPA’s 

Systems Lifecycle Management policies and procedures. Such compliance is 

necessary to provide reasonable assurance that efforts to replace the Agency’s core 

financial system achieve the desired results. EPA’s system development policies and 

procedures identify specific activities and documents required during a system 

development project. However, OCFO’s internal control environment does not 

enforce these policies and procedures. OCFO proceeded with the design subphase of 

the system project without obtaining executive management approval of the updated 

system requirements or developing and obtaining the required approval of test plans 

to ensure the system will meet Agency needs. Furthermore, OCFO did not 

predetermine the acceptable product acceptance test script failure percentages to be 

used as the basis for management’s go/no-go decision to proceed with using the 

evaluated product. These conditions could result in a system that does not meet 

management’s expectations and EPA’s needs, and/or does not comply with all 

applicable federal and EPA requirements. 

What We Recommend 

We recommend that the Chief Financial Officer develop and implement formal 

procedures for future projects to ensure that the requirements document(s) and test 

plans are authorized by executive management prior to approving the system to move 

into the next phase of the lifecycle. We recommend that any subsequent changes to 

the requirements document(s) and/or test plans be authorized by executive 

management prior to making changes to the design of the system. 

We also recommend that the Chief Financial Officer develop and implement formal 

procedures to ensure that the test plan associated with product acceptance testing, or 

any other test on which management relies, includes criteria that define what 

constitutes pass or failure to ensure that management has a basis for making go/no-go 

decisions. 

The Agency agreed with the recommendations with agreed-upon corrective actions 

pending. 
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON, D.C. 20460 

THE INSPECTOR GENERAL 

November 29, 2010 

MEMORANDUM 

SUBJECT:	 Improvements Needed in EPA’s Efforts to 

Replace Its Core Financial System

 Report No. 11-P-0019 

FROM: Arthur A. Elkins, Jr. 

 Inspector General 

TO: Barbara Bennett 

 Chief Financial Officer 

This is our report on the subject audit conducted by the Office of Inspector General (OIG) of the 

U.S. Environmental Protection Agency (EPA). This report contains findings that describe the 

problems the OIG has identified and corrective actions the OIG recommends. This report 

represents the opinion of the OIG and does not necessarily represent the final EPA position. 

Final determinations on matters in this report will be made by EPA managers in accordance with 

established audit resolution procedures. 

The estimated cost of this report, calculated by multiplying the project’s staff days and expenses 

by the applicable daily full cost billing rates in effect at the time, is $136,849. 

Action Required 

In accordance with EPA Manual 2750, you are required to provide a written response to this 

report within 90 calendar days. You should include a corrective actions plan for agreed-upon 

actions, including milestone dates. Your response will be posted on the OIG’s public website, 

along with our memorandum commenting on your response. Your response should be provided 

as an Adobe PDF file that complies with the accessibility requirements of section 508 of the 

Rehabilitation Act of 1973, as amended. The final response should not contain data that you do 

not want to be released to the public; if your response contains such data, you should identify the 

data for redaction or removal. We have no objections to the further release of this report to the 

public. We will post this report to our website at http://www.epa.gov/oig. 

If you or your staff have any questions regarding this report, please contact Patricia H. Hill at 

202-566-0894 or hill.patricia@epa.gov, or Rudolph M. Brevard at 202-566-0893 or 

brevard.rudy@epa.gov. 
Improvements Needed in EPA’s Efforts to 11-P-0019 

Replace Its Core Financial System 

Table of Contents 

Purpose ............................................................................................................................. 1 

Background ....................................................................................................................... 1 

Scope and Methodology .................................................................................................. 2 

Findings ............................................................................................................................. 2
 

Key Requirements Phase Documentation Not Authorized 

Before Moving Into Design Subphase........................................................... 2 

Product Acceptance Criteria Not Defined Before Testing ....................................... 3
 

Recommendations............................................................................................................ 4 

Agency Comments and OIG Evaluation ......................................................................... 4 

Status of Recommendations and Potential Monetary Benefits.................................... 5 

Appendices 

A Summary of Issues That May Have Contributed to Delays............................... 6 

B Overview of EPA’s System Life Cycle Management Phases ............................ 9 

C Agency Response ................................................................................................. 10 

D Distribution ............................................................................................................ 13 
 

Purpose 

We sought to evaluate whether the Office of the Chief Financial Officer (OCFO) 

effectively managed the information technology project to replace the U.S. 

Environmental Protection Agency’s (EPA’s) core financial system, the Integrated 

Financial Management System (IFMS). We also sought to evaluate whether the 

system development project is achieving the desired results. 

Background 

Based on requirements and demands placed on EPA by internal and external 

stakeholders, EPA decided to explore new core financial system options. The 

Financial Replacement Systems investment is OCFO’s ongoing modular approach 

to replacing its legacy financial systems, which include IFMS. EPA implemented 

IFMS in 1989 as its management and budget execution system. 

In February 2001, EPA hired a contractor to conduct an analysis of the Agency’s 

current financial systems and document the results in a strategic assessment. This 

assessment initiated the exploration for a new application. In January 2006, EPA 

issued a request for proposal to obtain bids for the project. EPA selected a 

commercial-off-the-shelf core financial system that is certified by the General 

Services Administration Financial Systems Integration Office (FSIO). 

EPA issued its Systems Life Cycle Management (SLCM) policy to promote 

effective and efficient processes for designing and operating information systems. 

Consistent with the policy, EPA’s SLCM procedure requires periodic, 

documented, management-level review of projects by the sponsoring office. The 

procedure requires program managers to oversee activities and establishes key 

opportunities to review development as the project progresses. These EPA policy 

documents define major decision points as “control gates.” The SLCM procedure 

requires the system manager to provide the required SLCM documentation to 

senior management for approval at each control gate. The procedure then requires 

senior management to make a go/no-go decision based on a review to ensure the 

required work products for each control gate are completed, approved, and 

verified. The policy and procedure also dictate that a system cannot move to the 

next phase without a go decision for the specific control gate. 

Many internal and external issues occurred during the lifecycle of this project, 

resulting in baseline changes and schedule delays. Appendix A highlights some of 

the issues that appear to have significantly altered the course of this project. The 

OCFO Office of Technology Solutions is managing the implementation of EPA’s 

SLCM processes for this project. See appendix B for specific details of EPA’s 

SLCM phases, subphases, and control gates. 

11-P-0019 1 
 

Scope and Methodology 

We conducted this audit from January through September 2010, at EPA 

headquarters in Washington, DC. We conducted the audit in accordance with 

generally accepted government auditing standards. These standards require that 

we plan and perform the audit to obtain sufficient and appropriate evidence to 

provide a reasonable basis for our findings and conclusions based on our audit 

objectives. We believe the evidence obtained provides a reasonable basis for our 

findings and conclusions. 

We performed a limited review of the status of the SLCM processes the Agency 

had undergone up to this point to replace IFMS. According to the project 

manager, at the start of the audit, the Agency had just moved into the design 

subphase of the acquisition/development phase of the SLCM process. We 

considered relevant internal controls associated with the objectives of our review. 

We reviewed system lifecycle documentation and interviewed Agency personnel 

involved with the replacement system project. We did not look in detail at any 

specific subphase of the SLCM process, but rather evaluated the applicable 

documentation to assess compliance with EPA’s SLCM procedures. For example, 

we did not test the adequacy of system requirements but rather verified whether 

EPA defined and authorized them. 

Findings 

OCFO’s management control processes do not ensure compliance with EPA’s 

SLCM policies and procedures necessary to provide reasonable assurance that 

efforts to replace the Agency’s core financial system achieve the desired results. 

OCFO proceeded with the design subphase without obtaining executive 

management’s approval of the revised requirements or developing and obtaining 

required approval of the associated test plans as required by EPA’s SLCM 

procedure. Furthermore, OCFO did not predetermine the acceptable product 

acceptance test (PAT) script failure percentages to be used as the basis of 

management’s go/no-go decision on the project. These conditions could result in a 

system that does not meet management’s expectations and EPA’s needs, and/or 

does not comply with all applicable federal and EPA requirements. 

Key Requirements Phase Documentation Not Authorized Before 

Moving Into Design Subphase 

In October 2008, the acting director for the Office of Enterprise Technology 

Innovations issued, and the deputy chief financial officer approved, a decision 

memorandum directing the project team to proceed to the design subphase prior to 

completing key activities in the requirements subphase. OCFO proceeded to the 

acquisition/development phase’s design subphase of implementation without 

solidifying requirements or developing and approving test plans. Without 

11-P-0019 2 
 

solidified requirements, it is difficult for OCFO to create test plans that ensure 

that the system meets the needs of the Agency. 

According to Agency policies and procedures, the requirements document(s) and 

test plans must be completed and approved by executive management prior to 

moving from the requirements subphase into subsequent phases of the system’s 

lifecycle. Also, any subsequent changes to either the requirements or test plans 

must be authorized by executive management prior to making changes to the 

design of the system. OCFO did not receive approval to make changes to its 

requirements from the Change Control Board until May 2010, over a year and a 

half after OCFO had moved into the design subphase. 

The system manager must ensure that specific, complete, measurable, and testable 

requirements and associated test plans are developed and approved by executive 

management prior to moving from the requirements subphase to subsequent 

phases. Additionally, the system manager must ensure that any changes to the 

requirements and/or test plans in subsequent phases are approved by executive 

management prior to making changes to the design. If specific, complete, 

measurable, and testable requirements and test plans are not developed, EPA 

cannot be sure that all requirements are met. 

Product Acceptance Criteria Not Defined Before Testing 

OCFO did not establish thresholds for PAT success or failure on which 

management would make a go/no-go decision prior to executing the test. 

According to the contract: 

The awardee shall conduct a PAT using the configuration that 

demonstrates the solution meets the requirements indicated as met 

“out of the box” in the awardee’s response to the Requirements 

Matrix. The awardee shall complete the PAT within the timeframe 

and acceptance metrics proposed by the Awardee and accepted by 

the Government. 

Without predefined test criteria, OCFO did not have objective measures to ensure 

that the intent of the PAT was met and the out-of-the-box system met the 

requirements as indicated by the contractor. 

Additionally, OCFO could not provide a test plan approved by senior 

management that defined and documented the test (purpose, requirements, test 

scripts, the criteria associated with the results to be applied for making the go/nogo decision, etc.). Such a test plan should have been developed and approved by 

executive management prior to the acquisition/development phase (the phase in 

which the PAT occurred). The lack of predefined test result thresholds for 

pass/failure on the PAT scripts could result in a product that requires significant 

customization to meet Agency requirements. 

11-P-0019 3 
 

Recommendations 

We recommend that the Chief Financial Officer: 

1.	 Develop and implement formal procedures for future projects to ensure 

that: 

a.	 the requirements document(s) and test plans are authorized by 

executive management (as a part of the definition phase’s 

requirements subphase) prior to approving the system to move into the 

next phase of the lifecycle, and 

b.	 any subsequent changes to the requirements document(s) and/or test 

plans are authorized by executive management prior to making 

changes to the design of the system. 

2.	 Develop and implement formal procedures to ensure that the test plan 

associated with PAT or any other test includes criteria that define what 

constitutes pass or failure to ensure that management has a basis for 

making go/no-go decisions. 

Agency Comments and OIG Evaluation 

In its October 12, 2010, response to the draft audit report, OCFO agreed with our 

recommendations. OCFO is planning to take actions to improve its processes 

related to our findings and recommendations. OCFO indicated that it plans to 

develop a checklist to identify, track, and monitor the requirements approval 

process and provide training on the procedures for obtaining executive 

management’s approval for changes to requirements and test plans. OCFO also 

plans to develop processes, revise existing procedures, and provide training 

associated with go/no-go decisionmaking criteria and ensure that the criteria are 

defined and approved prior to the start of work. We consider all of the 

recommendations open with agreed-upon corrective actions pending. The 

Agency’s complete response is provided in appendix C. 

11-P-0019 4 
 

 

 

Status of Recommendations and 

Potential Monetary Benefits 

POTENTIAL MONETARY 

RECOMMENDATIONS BENEFITS (in $000s) 

Rec. 

No. 

Page 

No. Subject Status1 Action Official 

Planned 

Completion 

Date 

Claimed 

Amount 

Agreed-To 

Amount 

1 

2 

4 

4 

Develop and implement formal procedures for 

future projects to ensure that: 

a. the requirements document(s) and test plans 

are authorized by executive management (as 

a part of the definition phase’s requirements 

subphase) prior to approving the system to 

move into the next phase of the lifecycle, and 

b. any subsequent changes to the requirements 

document(s) and/or test plans are authorized 

by executive management prior to making 

changes to the design of the system. 

Develop and implement formal procedures to 

ensure that the test plan associated with PAT or 

any other test includes criteria that define what 

constitutes pass or failure to ensure that 

management has a basis for making go/no-go 

decisions. 

O 

O 

Chief Financial Officer 

Chief Financial Officer 

O = recommendation is open with agreed-to corrective actions pending 

C = recommendation is closed with all agreed-to actions completed 

U = recommendation is undecided with resolution efforts in progress 

11-P-0019 5 

1 
 

 

 

Appendix A 

Summary of Issues That May Have 

Contributed to Delays 

A number of events occurred during the system development project that slowed the project’s 

progress. Each is shown in the timeline below (figure A-1) and explained in the text that follows.

 Figure A-1: Financial Replacement System chronological project roadblocks 

Agency Policies and Procedures Evolved 

Project Management Turned Over 

Financial Management
 

Line of
 

Business Established
 

Bid Protest 

Lodged 

Federal 

Financial 

Management 

System 

Requirements 

Changed 

System 

Development 

Model Changed 

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 

Calendar Year 

Source: OIG analysis. 

Agency Policies and Procedures Evolved 

Throughout the lifecycle of this project, EPA governed its systems development activities under 

multiple, evolving systems lifecycle management procedures. When the project began in 2001, 

EPA policy for SLCM was documented in the Information Resources Management Manual 

Chapter 17, Systems Life Cycle Management, issued August 28, 1994. An Interim Agency 

SLCM Procedure, which was approved on April 29, 2005, and the SLCM Procedure, which was 

approved on June 28, 2007, superseded the Information Resources Management Manual. 

11-P-0019 6 
 

 

Differences in these documents may have contributed to delays throughout the lifecycle of this 

project. In the 2005 and 2007 SLCM procedures, the SLCM is broken into the following phases: 

 Definition phase 

 Development or acquisition phase (2005 procedure) 

Acquisition/development phase (2007 procedure) 

 Implementation phase 

 Operations and maintenance phase 

 Termination phase 

The 2007 SLCM procedure documents the same phases but broke down some of the phases into 

subphases and established control gates to which the Agency must adhere. The subphases for the 

definition phase are concept exploration, system planning, and requirements. The acquisition and 

development subphases are acquisition, design, development, and test. 

Project Management Turned Over 

Since this project began, persons holding project management responsibility on the Financial 

System Modernization Project have retired or left to pursue other opportunities. Loss of expertise 

and knowledge over the lifetime of the project has presented challenges in maintaining artifacts 

as well as understanding undocumented thought processes. 

Financial Management Line of Business Established 

A major hurdle that the Agency faced during the procurement was deciding which procurement 

vehicle to use as well as ensuring that the process was competitive and complied with federal 

procurement regulations and the federal government’s implementation of the Financial 

Management Line of Business (FMLoB) initiative established by the Office of Management and 

Budget (OMB). 

In 2001, President Bush created the President’s Management Agenda (PMA) to address the need 

for citizen-centered, results-oriented, and market-based federal government initiatives. The 

success of the PMA depended on federal agencies working as a team across traditional 

boundaries to better serve the American people by focusing on citizens rather than individual 

agency needs. Pursuant to the PMA, OMB created the lines-of-business initiatives, which 

address redundant information technology investments and business processes across the federal 

government. The lines-of-business initiatives afforded agencies an unprecedented opportunity to 

influence the direction of specific core business functions government-wide. The FMLoB was 

created as part of these initiatives. FSIO, within the General Services Administration Office of 

Technology Strategy, is the program manager for FMLoB. 

In June 2005, EPA issued the initial acquisition strategy for the replacement system project, 

stating that it planned to use a General Services Administration schedule to issue a blanket 

purchase agreement for services needed for the replacement system project. In fall 2005, OMB 

began to issue guidance directing agencies to use commercial vendors or the center of excellence 

11-P-0019 7 
 

 

vendors for implementation as well as hosting services In January 2006, EPA revised its 

acquisition strategy to address OMB’s guidance. 

Federal Financial Management System Requirements Changed 

EPA’s original requirements were documented under the Joint Financial Management 

Improvement Program (JFMIP) Federal Financial Management Systems Requirements 

(FFMSR), in place in October 2005 when the original solicitation package was put together. The 

JFMIP principals voted to modify the roles and responsibilities of the JFMIP and established 

FSIO within the General Services Administration, which was formerly known as the JFMIP staff 

office. FSIO issued new FFMSRs in 2006, which replaced the associated JFMIP FFMSRs. In 

August 2009, OCFO underwent a process to recertify the requirements to ensure compliance 

with the new FSIO FFMSRs, among other inputs. 

Bid Protest Lodged 

A bid protest over the contract award associated with this project also added to the delays. EPA 

awarded the contract on February 12, 2007. One of the unsuccessful vendors filed a bid protest 

on February 26, 2007. The U.S. Government Accountability Office sustained/approved the bid 

protest on June 4, 2007. The award protest was resolved on April 10, 2008, and the first task 

order was issued on May 6, 2008. 

System Development Model Changed 

The decision to begin the implementation with the Rapid Prototype Life Cycle system 

development model in 2005 and then to change to the spiral model in 2008 may have contributed 

to some of the Agency’s time lags and schedule delays. 

In a rapid prototype development effort, an initial set of system requirements is translated into a 

test environment, and end users provide feedback. The feedback may lead to changes in 

configuration, business process reengineering, or other modifications. While every attempt will 

be made to minimize customizations to the commercial software, some customizations may be 

needed. The process of building, using, evaluating, and refining goes through several iterations 

before the system is ready for deployment. As the project progresses, the rapid prototype model 

may be tailored. 

The Spiral Life Cycle Model encompasses the best features of both the Waterfall Life Cycle and 

prototyping models, while simultaneously adding a new element: risk analysis. The model 

divides the software engineering space into four quadrants: planning, risk analysis, engineering, 

and customer evaluation. 

11-P-0019 8 
 

Appendix B 

Overview of EPA’s 

System Life Cycle Management Phases 

EPA’s system lifecycle consists of five phases. The objective of each phase is to ensure sound 

project planning and management practices throughout the system lifecycle. Figure B-1 depicts 

these phases and provides a context in relation to EPA’s enterprise architecture, capital planning 

and investment control, and security processes. The diagram also identifies the principal 

executive-level reviews (control gate reviews) that apply during the system lifecycle. 

Figure B-1: EPA’s Lifecycle Management Framework 

Source: EPA’s System Life Cycle Management (SLCM) Procedure (2121-P-01.0), June 28, 2007, p. 8. 

11-P-0019 9 
 

 

 

 

 

Appendix C 

 Agency Response 

October 12, 2010 (date stamped) 

MEMORANDUM 

SUBJECT: Response to the Office of the Inspector General Draft Audit Report, 

“Improvements Needed in EPA’s Efforts to Replace Its Core Financial System,” 

Project Number OMS-FY10-0006, dated September 9, 2010 

FROM: Barbara J. Bennett /s/ 

 Chief Financial Officer 

TO: Rudolph M. Brevard 

Director, Information Resources Management Assessments 

Office of Missions Systems 

Office of Inspector General 

Thank you for the opportunity to review and comment on the draft audit report entitled, 

“Improvements Needed in EPA’s Efforts to Replace Its Core Financial System, Project Number 

OMS-FY10-0006.” I would like to express my appreciation to you and your staff for working 

collaboratively with OCFO to ensure the report’s factual accuracy and provide recommendations 

to improve OCFO processes. 

The Office of Inspector General issued the following two recommendations. OCFO’s 

responses follow each recommendation: 

Report Recommendation 1: 

	 Develop and implement formal procedures for future projects to ensure that: 

o	 the requirements document(s) and test plans are authorized by executive 

management (as a part of the definition phase’s requirements subphase) prior 

to approving the system to move into the next phase of the lifecycle, and 

o	 any subsequent changes to the requirements document(s) and/or test plans are 

authorized by executive management prior to making changes to the design of 

the system. 

11-P-0019 10 
 

OCFO Response: 

OCFO agrees that the management control processes could have been more structured as 

the project progressed and has taken steps to formalize its current requirements and management 

approval procedures. However, at no time did OCFO’s management approval process 

compromise the development of system requirements and test plans to provide reasonable 

assurance that EPA will achieve its desired results. 

The draft report accurately described that revisions/changes made in the System 

Lifecycle Management (SLCM) document and the related control gate impacted how OCFO 

progressed in developing its new core financial system. OCFO used its program management 

support contractor to review the changes that occurred from the initial SLCM document in place 

at the beginning of the contract to its current version. The contractor identified the specific 

actions OCFO had taken that were different than the required actions in the initial SLCM. The 

contractor further prepared a justification for an approval waiver, if warranted. Based on an 

Independent Verification and Validation (IV&V) conducted by the Office of Environmental 

Information (OEI), a waiver was deemed unnecessary because OCFO’s actions and approach did 

not materially compromise or change the eventual decision. The program management report for 

OCFO and the IV&V report from OEI are available for review. 

The draft report portrayed the OCFO system requirements approval process less 

favorably. OCFO took great strides to ensure that requirements were complete, measureable, 

and testable. OCFO documented and tracked requirements in a matrix that was reviewed and 

approved by senior management at key points. The draft report accurately identified OCFO’s 

progress in reviewing and revising its requirements as the project progressed. For example, 

OCFO worked with EPA’s program and regional office subject matter experts to identify and 

confirm that approximately 1,400 requirements satisfied EPA, Financial Management Line of 

Business (FMLoB), and Financial Systems Integration Office (FSIO) requirements. This 

baseline set of requirements was an attachment to the formal OCFO acquisition in the Request 

for Proposal. Due to subsequent changes to FSIO requirements and new Momentum 

capabilities, OCFO continued its review and approval of system requirements. Rather than 

submitting a revised list of all the requirements for management’s approval, EPA separated the 

requirements into two categories: (1) no additional changes and/or approvals needed, and (2) 

changes that required management approval through the OCFO Change Control Board. 

Although the draft report states that OCFO proceeded with the design subphase without 

obtaining executive management approval, OCFO based the design phase on requirements that 

did not require additional management approvals. 

To improve the requirements approval process and procedures for current and future 

OCFO projects, OCFO is developing a checklist to identify, track, and monitor those actions that 

require executive management approval. The OCFO project manager will see that the checklist 

is regularly utilized and maintained. OCFO will train its control account managers and other 

subject matter experts working on the project on the procedures for obtaining executive 

management approval over changes to requirements and test plans. 

11-P-0019 11 
 

Report Recommendation 2: 

	 Develop and implement formal procedures to ensure that the test plan associated with 

PAT or any other test includes criteria that define what constitutes pass or failure to 

ensure that management has a basis for making go/no-go decisions. 

OCFO Response: 

OCFO agrees that it should have followed a more formal process in identifying and 

documenting its PAT plan and the associated criteria for formulating a “go/no go” decision on 

the project. However, at no time did the PAT “go/no-go” criteria: (1) incorrectly influence 

management’s decision to accept or not accept the Momentum product, (2) result in a system that 

could not meet management’s expectations and EPA’s needs, and (3) result in the selection of a 

product that does not comply with applicable federal and EPA requirements. 

OCFO worked with EPA’s program and regional office subject matter experts to test and 

verify whether the Momentum product could meet EPA’s critical requirements. The Momentum 

product was a commercial-off-the-shelf product that is in place in other government agencies. 

Accordingly, OCFO was fully aware that the Momentum product was an established suite and 

relied on the success/failure criteria established by EPA’s subject matter experts during their 

assessment of performance against test scripts during the PAT testing. The Momentum 

contractor developed the test scripts with input from the EPA experts. In some cases, the test 

script failed due to a missing step and not because the Momentum software could not meet the 

EPA requirement. Accordingly, EPA allowed some test scripts to be revised and retested. The 

EPA control account manager made the decision to develop the “go/no-go” decision criteria 

based on input from the EPA subject matter experts. OCFO documented the PAT results and 

made a presentation to the Administrative System Architecture Steering Committee (ASA SC) 

that serves as the Agency-level governance body for the project. Based on the information 

provided, the ASA SC made the recommendation to the Deputy CFO, as the project sponsor, to 

accept the Momentum product and move forward with the project. 

To improve its process for defining “go/no-go” decision making criteria, OCFO will 

develop processes and revise existing procedures to apply more rigorous use of test script results 

in driving “go/no-go” decisions on current and future projects. We have done this with FSMP 

and will use the criteria in obtaining executive management go/no-go decisions. The OCFO 

project manager will see that decision making criteria are developed based on test script pass/fail 

metrics and approved prior to the start of the work. OCFO will train its control account 

managers and other subject matter experts working on the project on the appropriate evaluation 

of projects based on test script pass/fail measures. At present, EPA has applied the OIG 

recommendation and established success/failure metrics and acceptance criteria for the upcoming 

User Acceptance Testing phase of the Core Financial System implementation. 

11-P-0019 12 
 

 Appendix D 

Distribution 

Office of the Administrator 

Chief Financial Officer 

Agency Followup Coordinator 

General Counsel 

Associate Administrator for Congressional and Intergovernmental Relations 

Associate Administrator for External Affairs and Environmental Education 

Director, Office of Financial Management, Office of the Chief Financial Officer 

Audit Followup Coordinator, Office of the Chief Financial Officer 

11-P-0019 13