# Presentation: 147177

## Evaluation of Enterprise CSTARS Pilot 

**Evaluation of Enterprise CSTARS Pilot**** **

**Briefing for the Commerce IT Review Board, **
**Chief Financial Officers Council and the CSTARS Enterprise Team**
**Final Report**
**July 26, 2001**

_**C**_**ommerce **_**ST**_**andard **_**A**_**cquisition and **_**R**_**eporting **_**S**_**ystem**

## Overview

- CSTARS Background
- Assessment Summary
- Key Issues (Security, Interfaces, Risk)
- Recommendations
- Risk Mitigation
- The Road Ahead

## What Is CSTARS?

**Enterprise-wide e-Initiative**
- supports “e-Commerce”, “e-Business”, and “e-Government” policies and pilot programs
**Automated Procurement System**
- provides a robust procurement/acquisition functionality, powered by a Competitively Awarded COTS Solution (Comprizon.BuyTM )
- eliminates duplication with CAMS/CFS
- Promotes Standard Business Practices for the Acquisition Workforce and Their Customers** **

## CITRB  - Overarching Goals

- Streamline operations
- Reduce cost by solving systems and business process problems at the enterprise level
- use COTS solutions wherever practicable
- Reduce the duplication of system functionality
- Support e-Government

**Notes:**
The Office of Federal Procurement Policy (OFPP) and the Federal Acquisition Regulation (FAR) support e-Government and initiatives to streamline acquisition operations.

## CITRB - Direction for the DOC Procurement Executive

- Acquire a Commercial-Off-The-Shelf (COTS) solution for DOC-wide procurement automation needs
- Charter an Automated Procurement System Enterprise Team (APSET) to establish the foundation and strategic direction for implementation
- Pilot the proposed solution at OS and NIST under the direction of the APSET

## CITRB  -  Conditions for COTS (CSTARS)

- No customization
- Implementation will not interfere with the CAMS core financial system (CFS) implementations at NOAA and Census
- All operating units will use CSTARS , regardless of financial system.  This will be coordinated with the Enterprise Team
- Present to the CITRB results from successful deployment of the NIST and OS pilot
- A single instance of CSTARS will be installed on centralized servers with multiple databases

## Enterprise Team Goals

- Provide management data for decision-making
- real-time current information gathering tools
- Solve problem in procurement operations
- immediate improvements from legacy systems
- Accommodate future technology trends
- flexibility, EC and web-based functionality
- Standard Enterprise Practices
- streamlined, use reforms, market-place driven
- Selection of COTS product
- Comprizon.Buy = CSTARS

**Notes:**
Management of data to understand workload, plan strategically, determine whether the DoC is meeting small business goals.

Future tech to competitively position to respond to market trends.

## Champions and Partners for Change

- Government
- CITRB (also CIO’s Office of Computer Services)
- CFO Community
- Procurement Community
- CAMS Executive Board
- CSTARS Enterprise Team (Bureau Heads of Contracting Offices and stakeholders)
- Federal Comprizon.Buy Users’ Group
- Commercial
- CACI, Federal Inc.
- Booz-Allen & Hamilton
- I. M. Systems Group

**Notes:**
Helen Hurcombe	NOAA
Howard Price		O/S
Norm Osinski		NIST
Mike Palenski		Census
Mike Anastastio	PTO
Diana Hynek		CIO
John Sansing		CFO
John Shackelford	CACI

Helps to create the collaboration needed to operate as a matrix organization.  Ultimately this team and ownership approach helps ensure improved BSC data and to understand our performance outcomes.

## Enterprise Model for CSTARS

- Re-order and reduce scope of implementation
- Limit functions to internal procurement needs
*OS+ implementation as of October 1, 2001*
*NIST implementation as of November 1, 2001** *
*interfaced with the NIST financial system*
- Centralized database architecture - CIO’s Office of Computer Services (OCS)
- Evaluate, Assess, Report Findings and Recommendations

**Notes:**
OS+
OAM/Acquisition Services
OAM/COMMITS Office
Office of Administrative Services
Office of Security
Office of the Inspector General

Resources 
WCF budget reality
Dying systems on OS, Census, NOAA
 DGS, SPS
limited rollout, faster and cheaper
PTO last

Given the model:  IMSG is here to help as a disinterested party.
Understand what we’re doing right, where improvements are needed in our implementation approach.  Identify the missing elements to move forward.

## Commerce Dept. demonstrates sound business practices for progress.

***Commerce Dept. demonstrates sound business practices for progress.***

- 1998 	1999 	2000 	2001 	2002

- OMB Circular A-11
-       ITRMA
-             FARA
-                   FASA

**Management Approach**

**6 ****Bureaus with  procurement authority**
**9 ****Different procurement systems**** **

- Enterprise Pilot

- Continuous Improvement

********* ******Standard Business Practices***
********* ******Lower Purchasing       ******	******Costs***
********* ******Leverage Technology ******	******Investments***

- Business
- Case

- Procurement
- Automation
- Team

- Program
- Management
- Plan

- Pilot
- Evaluation

- Migration to
- e-commerce &
- e-marketplace

***Towards an Enterprise-Wide Procurement System***

## Pilot Evaluation -    schedule and scope

**  ****March 01                     April 01                        May 01                         June 01**

**Project**
**Start**
**Feb 23**

- Weekly Status / Progress Briefings

**Detailed**
**Evaluation **
**Plan**

**Finalize**
** ****Report &**
**Briefings **

- Conduct
- Initial Assessment

- Conduct Evaluations

- Perform Analysis

- Prepare report

**Pilot Evaluation -    ****schedule and scope**

**1.  Conduct Initial Assessment****  **
** ****Reviewed Program Management Plan (PMP), dated Dec 10, 99.**
** ****Updated Evaluation Criteria (Appendix 6 to PMP).**
** ****Identified/collected existing documentation.**
**2.  Conduct Evaluations****  **
** ****Survey technical architecture.  Review documents, records, logs.**
** ****Interview OS+ and NIST users.**
** ****Collect performance metrics. **
**3.  Perform Analysis**
** ****Enterprise Standardization.**
** ****System Setup.**
** ****System Performance.**
**4.  Identify Impact**
** ****CSTARS Program Objectives.**
** ****Department Goals.**

## Evaluation Criteria  -  from Appendix 6 of Program Mgmt Plan

**Revised Performance Measures**
- Confirm existence of data dictionary. Number of data elements.*
- Confirm data dictionary contains data formats.*
- Confirm capability to handle non-standard formats.
- Confirm existence of process documentation.Identify number.*Confirm capability for Bureaus to review processes.
- Number of data elements identified
- Confirm existence of ICDs
- (no change)      NIST         OS+
- (no change)      NIST         OS+
- (no change)      NIST         OS+
- (no change)*     NIST         OS+
- Number of vendors loaded into database.*         NIST         OS+
- Number of contracts migrated.*	            NIST         OS+
- Number of routing paths created/loaded             NIST         OS+
- Load test standards exist?  Expected transaction response times.
- % of successful data transmissions
- (no change)
- (no change)
- (no change)* *(Define “transactions”)*
- (no change)*     NIST         OS+
- Capability exists to track processing time? Number of days.*
- Capability exists to track cost? Cost to accomplish specific actions.*
- *Provides a baseline measurement for further evaluations.

## Evaluation Criteria  -  Progress Summary

**DATA SUMMARY*****	**** NIST****	****OS+****	****NOAA**
**Processes documented****	**** **136	124**	****	****	**
**Data elements documented ****	**** **285	216
**Users with profiles****	**** **130	 42                    88
**Vendors in database****	**3400	175                21,000
**Contracts migrated****	**** **800**	**113
**Users trained****	**** **130	 22
**Completed transactions****	**3000**	**345

- * Data represents approximate values, and is intended to be used as baselines to indicate progress over time.
- ** “Completed Transactions” include all new contracts/purchase orders, delivery orders, and modifications
- using CSTARS.  This does not include in-progress requisitions and solicitations or cancellations.

- Mapping Sessions
- Completed

- Mapping Sessions
- Completed

- Begin
- August 01

- Begin
- October 01

- Begin
- October 01

## Assessment Summary

**Standardization**
- CSTARS Program is progressing towards commonality of processes, formats, and data.
**Set Up**
- Implementation (data initialization, process migration, interface control) is more difficult than originally planned.
**Production**
- Process is taking longer.  Can be attributed to learning curve issues, process trade-offs, quality gains, and change management challenges.

**Bottom Line:  We found **_***no technical “show stoppers”***_** **
**that indicate any reason to discontinue CSTARS **
**implementation throughout the Department.**

## Major Lessons Learned

**Implementation **
- More difficult, more resource intensive than originally planned.
- Better to have a bureau person become CSTARS expert than to have a CACI person become a bureau expert.
**Training**
- Suspected system shortcomings are usually traced to user lack of understanding.
- Bureau dedicate resources as a core group.
- Desk side training is critical; 6-month burn-in is to be expected.
**Reporting**
- Current Ad hoc reporting capabilities do not satisfy user requirements.
- DoC needs to define reporting standards.
**Interfaces**
- Currently - lots of re-keying going on.
- The tall pole  -  financial system interface. Process integration needed.
**Management commitment**
- Top-level commitment of CIO, CPO, & CFO is critical.
**Support to Bureaus**
- Written agreement from OAM  -  who pays for what  -  will facilitate bureau planning.
- Credible, reliable SLA with OCS is critical for bureau-level confidence.
- Planning factors are needed to develop cost-loaded implementation plans.

## Transaction cost estimates

- (Data from controlled CSTARS runs conducted through OCS, May 2001)

**CSTARS supported at OCS.   Based on Oracle CPU usage.  **

*Estimates can be used as planning factors*.

*Example:  For an organization during*
*       **early months of CSTARS use:*

- *Based on current cost per CPU.

## Other Observations

- Security
- CFS-CSTARS Interface
- Risks

## Security

## Security  -  assessment in progress

-   System Security Plan in development
-   Process-oriented approach, based on data sensitivity
-   CSTARS has capability for access segregation, based on roles

**User Groups/**
**Roles**

**Procurement Process**

## Security  -  observations

_**Observations:**_
- - No overall security policy between organizations.
- - Each box requires separate username and password for access.
- - Goal is to facilitate use of the system, and
- Prevent unauthorized access to C* systems/data
- 	Restrict authorized users to appropriate systems/data.

- References:
- NIST Special Publication 800-18  “Guide for Developing Security Plans for IT Systems”
- NIST Special Publication 800-XX  “Self Assessment Guide for IT Systems”
- NSA Infosec Assessment Methodology

## CFS & CSTARS  -  Planning Horizons 

**CFS & CSTARS  -  ****Planning Horizons **

**Comprizon.Web**
**Initial Release**

**CSTARS**
**Pilot**

**CSTARS -CFS**
**Interface Development**

***Bureaus finalize program***
***funding for FY 03.***

**CSTARS Implementation at**
**NOAA, NIST, OS+**

**FY 03 Commitments**

**FY 04 Commitments**

**CFS at NIST**
**Live**

**CSTARS-CFS**
**Interface Release**

-  FY 01                  |	      FY 02               |	     FY 03                 |	         FY 04           	  |
**Jan                               Oct                Jan                               Oct                   Jan                                 Oct                  Jan                                 Oct**

- DRAFT

**CFS/CAMS**
**Continued Development & Deployment**

**Transition to**
**CSTARS - WEB**

**CAMS in production at**
**Census & EDA as of Oct 1, 2000**

**CAMS in production at the NIST**
**Crossed Serviced Bureaus as of July 1, 2001**

**CAMS in production at**
**NOAA as of  Oct 1, 2002**

## Enterprise Application Integration 

**Enterprise Application Integration **

**Current DoC**
**Systems **

-  **Financial System**

**Perpetuation of**
**Stovepiped**
**Systems**

**IPT   CONOPS    BUSINESS    PMP    PILOT     WEB**
**                                ****CASE                                     DEV**

**FINANCIAL**
**REQUIREMENTS**

**PROCUREMENT**
**REQUIREMENTS**

-  **Procurement System**

***Enterprise***
***Application***
***Integration***

***DoC demonstrates sound business processes, but...***
**CAUTION:  BUILDING INERFACES IS NOT ENOUGH.**

***YOU NEED***

***Requirements***
***Integration***

***Process***
***Integration***

**IPT   CONOPS    BUSINESS    PMP    PILOT     WEB**
**                                ****CASE                                     DEV**

## Risks

- 1.  With any COTS solution -
- Who controls product’s evolution?
- Vendors respond to paying customer. All others must adjust.
- 2.  Migration to web-based application(s):
- Today’s solution  =  tomorrow’s legacy (how are your interfaces doing?)
- What are the ancillary costs? (training, data migration, protocols, security)
- 3.  Integration of Enterprise Systems (CFS and CSTARS)
- No process integration (leads to stove-piped requirements)
- Financial Standards not same as  Procurement Standards
- 4.  Cohesive enterprise OR waning credibility
- As landscape changes, how will Dept support Bureaus?
- Good strategy without resources = delays, burnout, death march .
- Long-term top-level commitment needed for managed expectations.
- 5.  Distributed implementation could drive divergence from         standardization over time.

## Putting the Pilot in Perspective

## Recommendations

- 1.  Continue CSTARS implementation.  Leverage NIST and OS+ experience to enable implementation at other Bureaus.
- 2.  Re-examine Centralized vs Distributed implementation of enterprise-widee-procurement system.
- 3.  Update the CSTARS Program Management Plan (Dec 10, 1999) to include current implementation guidance, schedule, and transition to web-CSTARS.
- 4.   Establish standards and for centralized reporting requirements.
- 5.  Restore contractor support to supplement OAM program management:
- 	-  Revise 10-Year Life Cycle Cost Estimate
- 	-  Leverage best practices, lessons learned
- 	-  Facilitate implementation, migration to web-CSTARS
- 6.  Provide for on-going strategy assessment, to include impact of technologies such as web requisitioning, e-catalog, e-marketplace.

## The Road Ahead ...

## Major CSTARS Events

**Notes:**
What to look forward to.  

HANDOUT

## CITRB Approval from Business Case

Actual (FY99-00) and Projected (FY01-05)

- Actual (FY99-00) and Projected (FY01-05)

- Cost - Business Case versus Actual/Projected

- * Data from Business Case, January 19, 1999

- ** Data based on OAM current WCF budget analysis.

*****

******

**Notes:**
Business Case:  Rough Order of Magnitude
FY 99 Reduction
Lost PM support
Budget freeze
Budget thaw - too late to implement large task orders
FY 00 Reduction
Reduced human resources to tackel the work
What do you get for your $$$?
Technical planning and coordination
Site license and maintenance to operate
software installation @ OCS/Springfield
Desktop audits; user profiles established
Conference Room Pilot
Mapping the data elements
Enterprise desk guide and Addenda for bureau-specific business practices
Benefit of lessons learned
Data initialization
Initial user training and desk-side production support

## Risk Mitigation

- Evaluate implementation process
- Task order to evaluate and assess security
- Contract for program management assistance
-  Insure enterprise integration with CAMS
-  Look for e-Opportunities for e-Commerce
- Initiate Data Warehouse implementation
- Revive the CSTARS Enterprise Team
- Participate in the Comprizon.Buy Federal Users Group

**Notes:**
Best laid plans always come w/unknown challenges.  
Evaluate implementation process
keep focused on the target
Contract for program management
working towards meeting the financial community’s goal for clean audit
Initiate Data Warehouse implementation
strengthen our ability to report accurate, timely data
Help ensure clean audit.

## Next Steps ...

******  *****What concerns do you have?***
******  *****What involvement do you want?***
******  *****When do you want the next update?***

**Notes:**
It’s all about you.