Joint Financial Management Improvement Program

Forum Highlights: System Implementation Success Factors using COTS Financial Systems

June 12, 2003

## Key Points

Public and private sector leaders in system implementations discussed a wide range of topics concerning system implementation success factors using commercial off-the-shelf (COTS) systems.  Some of the key points are the critical needs that the government should address.  They include:

· Leaders need greater awareness that implementing new systems is more about management than technology.  

· Government should further rationalize, standardize, and communicate its requirements for financial management systems, in order to move the market.  

· The competencies of Project Managers are critical for success and must be better defined and developed.

· Testing should be expanded and improved, and should address scalability. 

· Government should examine what is the right model for measuring success—by processes followed or by outcome measures.

## Opening Remarks

There is a debate in the federal government whether COTS financial management systems are meeting federal agencies’ needs and expectations. During the past decade, agencies have implemented major financial management systems using COTS software with mixed results. The Joint Financial Management Improvement Program (JFMIP) partnered with the Private Sector Council in organizing a forum on June 12, 2003, sponsored by the JFMIP Steering Committee and Chief Financial Officers’ Council. The forum’s overall objective was to engage senior federal financial managers, financial system program managers, and private sector thought leaders in a dialog regarding whether to use COTS software and, if the answer is yes, to discern the management imperatives leading to successful implementation of COTS software to meet agency financial management needs and expectations.

Joseph Kull, Deputy Controller, Office of Management and Budget (OMB), opened the session with introductions and the challenge regarding use of COTS software to meet federal financial management needs: *expectations are not being met, better data is needed, and money is tight*.  Mark Carney, Chair of the E-Government Financial Systems Committee and Deputy Chief Financial Officer (CFO) at Department of Education (ED), followed with an overview of ED’s experience with implementing COTS software which over the last decade traveled from painful abandonment of a software “stinker”, to effective implementation of a new system which met their needs, but required 30 workarounds. Getting to this better state relied on *testing* at several levels: by JFMIP to identify and improve market options and by Education to make sure the accounting and data fields were properly mapped before the system went live. Jerry Williams, Chief of the Financial Systems Branch, OMB, gave his perspective that 6 years after the passage of the Clinger Cohen Act and the Federal Financial Management Improvement Act, COTS financial management systems do not appear to “perform” out of the box. The question is why—lack of rationalization of agency specific requirements? Inadequate government management processes?  Inadequate market response to federal requirements?

## Private Sector Perspective

Following Mr. William’s remarks, a panel of representatives from the private sector provided introductory comments and thoughts on the private sector’s experience in implementing COTS software. 

A.W. “Pete” Smith, President of the Private Sector Council (PSC), moderated the panel discussion. Mr. Smith began the discussion by making two observations. First, some organizations do things better than others, both in the private sector and in government. Second, the private sector faces difficulties in implementing COTS software, just like the government.  Following Mr. Smith’s opening statements, five advisors to the PSC presented brief overviews of their perspectives on implementing COTS financial management systems.

David Carney, Senior Member of Technical Staff at the Software Engineering Institute, Carnegie Mellon, noted that COTS failures are prevalent in private sector companies, but are not well known due to nondisclosure agreements and fear of negative publicity. Not withstanding the difficulties, the option to custom develop applications is unrealistic, given the cost of technology development and the rate of change. The real question is whether the COTS software should be modified, and that is a management and engineering question that can only be answered in the context of the organization being served by the technology. Dr. Carney pointed out that an attitudinal change is needed regarding the use of technology. The software industry is relatively new and products are not highly stable. With technology shelf life of 2 years, the desired state of being “done” will never be reached.

Gopal Kapur, President of The Center for Project Management agreed with Dr. Carney that COTS software should be used, and organizations that use COTS software must be prepared to deal with software modification and workarounds to meet business needs.  The more important consideration is how well an organization will implement the system. Mr. Kapur described three organizational competencies needed for successful implementations: mapping the complexity of the system to the organization’s capabilities; commitments of sponsors and senior management to the project; and commitment that is sustained throughout the project’s life cycle, typically at least 2.5 years. He also shared results of a recent survey of government senior level managers and executives that indicate significant shortfalls in critical success factors including organizational sponsorship, project management, and project measurement.  

Morgan Kinghorn, Partner for IBM Business Consulting Services, provided his perspectives based on extensive government and consulting experience. Mr. Kinghorn believes that realistic expectations should be established at the outset that new systems, while delivering greater power and technology improvements, require business process changes and will not work in the same way as custom-developed applications. Mr. Kinghorn proposed several success measures for implementations, such as: on budget; better internal controls; improvement in response times; reduction in overtime; better and more timely information; reduced cost of ownership. Mr. Kinghorn also observed that COTS financial management systems are typically designed and implemented by accountants, for the CFO organizations. He recommends that program personnel be involved in defining what the agency wants out of the financial management system. 

Richard Rogers, Vice President of Strategic Positioning at Titan Systems Corporation, concurred that financial management systems should be COTS based.  He pointed out that the key to successful implementation of COTS based systems is how the process is managed.  Agencies tend to focus on a product’s technical base in making a selection; however, of equal importance is the agency’s underlying culture and business processes.  The agency will have to be willing to alter its processes and be able to spark, manage and control change in the organization.  Managing culture and process change in large, diverse, organizationally and geographically decentralized agencies is a much greater challenge.

Will Tracz, Principal Research Scientist, Lockheed Mission Systems, observed that the technology base is changing rapidly. Implementing COTS software is buying into a process where there will be continuous change over time. It is critical to anchor decisions about COTS software choices in an information architecture and an enterprise architecture to drive effective use of enterprise information systems that use COTS software that support business needs. Dr. Tracz believes that agencies need to look at the bigger picture, namely their enterprise IT infrastructure that is being put in place. 

Following the opening remarks provided by the private sector representatives, the attendees participated in an interactive discussion on how to address financial management system challenges. Highlights of the discussion are presented in the sections that follow.

## Setting Realistic Expectations for Technology Environment

###### Forum participants discussed the importance of setting realistic expectations about what technology can do for the organization:

· Technology and markets change rapidly. To respond quickly to these changes, the government must use COTS software, not individually developed products. Since COTS products are not static, expect continuous change. One challenge is to select among rapidly changing options in an inherently “unstable” product base, where “standards” are constantly adapting.  The good news is that products are constantly improving.  However, better understanding of the potential gaps between product capabilities and user needs, by both software suppliers and users, would inform better management and more realistic expectations about outcomes. 

· Management processes must be appropriate to the rate of technology change.  Mandating management practices that take longer to execute than the expected technology life cycle may misallocate resources.  Methodologies desirable for software development such as capability maturity model (CMM) may not be appropriate for implementation of developed software (i.e. COTS).  Understanding the business needs, the context, how the applications fit together, and how they interoperate is more critical.  An enterprise architecture and a data architecture provide that framework.  

· Leadership must be educated on the importance of change management in a COTS software implementation.  Project sponsors must be responsible for assessing organizational readiness, setting expectations, and ensuring accountability.  Business processes must change to support new COTS products.

## Underlying Conditions for Successful COTS System Implementations

Forum participants discussed issues and strategies related to the underlying conditions for successful COTS system implementations.  Project success relies on multiple factors.

### Senior Management Expectations and Leadership/Ownership

· Commitment of senior management sponsors is key to a successful implementation.  This commitment must be sustained through the duration of the project, the duration of which typically is at least 2 years.  Sponsorship must include political appointees, senior career staff, and “process” owners.

· Senior leaders must establish appropriate success measures based on performance outcomes.  Avoid arbitrary deadlines as the measure of success.  They lead to date driven failures.

· Modern system information tools are not static.  Managing the mosaic of tools will be continuous. 

### Project Management Competence

· Projects tend to be under planned.  Organizational competency to map the complexity of the business process being replaced, understand the politics of change, and manage the change over a sustained period is necessary.  

· “Industrial Strength” project managers are required in the federal environment to manage the technical program management, and to manage communications and change in large, complex organizations.  An added challenge is communicating with multiple oversight bodies. 

· Project managers roles include scope management, expectations management, “360°” ownership management, and culture management as well as management of vendor performance.  Project managers need to be aware of project “vital signs”.  These vary by organization and need to be considered in advance.

· Use integrated project teams of government subject matter experts and contractors; however, government has the ultimate responsibility to manage the project. 

· Involve agency program managers in defining requirements and in implementation decisions to gain acceptance and use. 

· Project Management training and preparation is a must.  

### Change Management

· Business goals must be clearly defined and communicated. 

· The organization must be willing and ready to accept the changes to its processes that will result from the new financial management system.

· Evaluate the ”health” of the organization before, during and after the implementation as a measure of project risk.

· Change management considerations drive cost and risk.  Without dedicated resources for training of senior management, middle management and users, the project outcome is at risk.  

### Acquisition Management

· Contracting Officers are invaluable allies to project managers if a strategy is worked out in advance to achieve specific performance capabilities, and penalties and incentives are built into the acquisition strategy.  

### Defining Requirements

· Defining requirements before conducting the “gap” analysis is critical.

· Requirements should be focused on what needs to be done, rather than how to do it.

· Funds control and appropriations accounting are “niche” federal requirements.

· For the COTS developers to respond to “special” needs, those needs have to be clearly defined and communicated.  Part of that requirements determination process is organizing them, eliminating unnecessary complexity, rationalizing and standardizing them, and vetting them so that vendors can respond and users can confirm that they did.  

· Financial management system requirements, as promulgated through JFMIP, address the minimum standard for government financial systems. Vendors should understand that these requirements are the “entry” criteria into the federal marketplace. Each agency may have specific requirements that need to be met as well. 

· Performance imperatives (e.g., scalability, no manual intervention, etc.) need to be better defined.  However, vendors will not commit to certain technical performance requirements without unambiguous visibility (and control) of technical infrastructure.   

· Information needs of program managers need more focus and visibility. Financial management system requirements now primarily reflect needs of the CFO organization.

· Government is moving in the right direction with their approach of defining government-wide requirements upfront and qualifying systems against those baseline requirements.  

· There is a need and advantages of having standard business processes across government.

· Once requirements are defined, the market can be organized to support (e.g., pay for the GOTS).

### Testing

· Testing and ensuring that the test results have market consequences give customers strong management leverage.  Requiring the vendor to provide test plan, test data, test results and using this information in a legal document (e.g., request for proposal) is a quick and effective reality check on vendors.

· Centralized testing can shape the market place and force the market to meet federal “niche” requirements over time.

· JFMIP qualification testing assures a certain level of compliance with government-wide requirements; but should be viewed as “entry criteria.” Agencies should conduct supplemental testing to ensure the financial management system meets their specific requirements, and to ensure adequate system performance.

· Current government-wide qualification testing does not extend to subsidiary systems (e.g. other components that support an enterprise architecture).

· Scalability testing is desirable, but more difficult given the diversity of technical environments.

## Measuring Financial Management System Success

Forum participants discussed the following outcome measures of financial management system success:

· on budget

· stronger internal controls

· improved response times

· reduced overtime

· better and more timely information

· reduced cost of ownership

· These system implementation outcome measures should be key criteria for evaluating success of systems as opposed to implementation process measures (e.g. earned value).

## Major Success Factors for Acquiring and Implementing COTS Financial Management Systems

Forum participants discussed the following best practices and success factors for acquiring and implementing COTS financial management systems:

· Determination of the agency’s information and enterprise architecture should precede the acquisition of a financial management system. The system should meet the agency’s business goals.

· Agencies should modify their business processes to take advantage of the best practices embedded in COTS financial management systems.

· Modifications to the software to meet agency needs may be unavoidable, but should be minimized. Modifications to software increase project risk exponentially. 

· The scalability of the COTS financial management system may be a critical determinant to a successful implementation.  Many of today’s systems are not designed to support the volumes of data processed in large, cabinet-level organizations.

· The project management team should manage the expectations of senior management, program managers, and oversight entities. There are tremendous political pressures to implement a system ”on time” even if the system is not ready; project managers must be empowered to delay implementation if the system is not ready.

· Projects should be broken down into phases with “go/no go” decisions after each phase.

· While much of the implementation effort may be outsourced, leadership responsibilities must remain within the government. 

## Critical Needs

Critical needs that the government should address:

· Leaders need greater awareness that implementing new systems is more about management than technology.  Management roles include establishing business goals, realistic expectations, accountability, examining the need for change in business processes and managing these changes, and leading cultural change necessary to accept capabilities of new system.

· Government should further rationalize, standardize, and communicate its requirements for financial management systems, in order to move the market.  Continuing efforts to “rationalize” federal requirements, include challenging unusual or overly complex requirements.   

· The competencies of Project Managers must be defined and developed.   Programs such as the Defense Acquisition University could be expanded as a government-wide resource to support development of Project Managers.

· Testing should be expanded and improved, and should address scalability. Government needs to move the market in order to limit the need to customize COTS software and to avoid disasters. The role and importance of agency testing needs to be better understood.

· Government should examine what is the right model for measuring success—by processes followed or by outcome measures?

Attachment 1. Forum Participants

Attachment 2. Questions for the Private Sector Council

1. What are the best practices and success factors for acquiring and implementing COTS financial management systems?

2. How should financial management system success be measured?

3. What are the underlying conditions for successful COTS implementations and are these trade-offs appropriate and sustainable?

4. What is the role for financial management system requirements and testing?

5. How can organizations successfully plan and implement interfaces between their core financial management systems with the business systems to meet the business informational needs of program (line) managers?

Attachment 3. References

1. Kapur, Gopal, “Avoiding Disaster”.

2. Software Engineering Institute, http://www.sei.cmu.edu/

3. Tracz, Will, “Architectural Issues, other Lessons Learned in Component-Based Software Development,” in CrossTalk, January 2000, pp 4-7.

COTS software should come with a warning, “batteries not included, extensive assembly required”.

COTS software is not a “silver bullet”. However, like a bullet, it can hit the mark if it is properly loaded, the target is identified, and the user is properly trained on how to use it.

“Meeting requirements is like walking on water—easier done when frozen.”

1

DRAFT— MACROBUTTON NoMacro [Click **here** and type report #)] —8/1/034Final June 12th Forum Highlights

7