****

**Financial Services Assessment Guide (SAG)**

**Phase I**

**March 30, 2007**

**Version 1.0**

**Table of Contents**

1Overview

Implementation2

Phase I2

Phase II3

**APPENDICES**

Appendix A. FMLoB Financial Performance Measures4

Appendix B. FMLoB Financial Performance Measure Definitions5

Appendix C. FMLoB Performance Metric Reporting Procedures15

Appendix D. Current Performance Measures Working Group17

# Overview

The Financial Management Line of Business (FMLoB), in collaboration with the federal financial management community, is establishing a set of Financial Services Metrics that will facilitate an assessment of financial services government‐wide. These metrics are designed to help identify opportunities to improve the performance and affordability of the financial services provided by Shared Service Providers (SSPs) and Federal agencies.

The Financial Services Metrics will be used for the following purposes:

· Enable SSPs to make more informed judgments regarding the performance and affordability of the financial services they provide, and how they compare to their competitors.

· Enable agencies to make more informed judgments regarding the performance and affordability of the financial services they provide in-house, how they compare to other agencies, and which SSP might best serve their needs as they look at potential migration of services under the FMLOB framework.

· Enable key stakeholders, including the Chief Financial Officers’ Council (CFOC), the Office of Management and Budget (OMB), and FMLOB, to make more informed judgments regarding the financial performance of SSPs and agencies, and work with SSPs and agencies on strategies to continually improve both performance and affordability.

Performance reports will be provided to various audiences as appropriate, including the CFOC, SSPs, agencies and OMB, and may be made available to the public. Performance will be evaluated against established baselines and applicable performance benchmarks. As this effort progresses, reports will provide performance trends, distinguish top performers from poor performers, identify improvement opportunities, and identify the need for corrective actions as warranted.

By making SSP and Agency performance more transparent, establishing accountability for improved results, and increasing competition among SSPs, both SSPs and Federal agencies will be encouraged to continually improve both the performance and affordability of the financial services they provide.

Performance metrics will be implemented using a phased approach. Phase I will focus on defining and collecting an initial set of high value, low burden metrics. Phase II will focus on refining these metrics, collecting an expanded set of metrics, adding cost metrics, improving reporting capabilities and accountability mechanisms, and streamlining the collection effort.

All agencies will be required to report performance data through a single system managed by the Financial Systems Integration Office (FSIO). All data submitted to FSIO will be considered public information unless specifically identified as confidential and approved as such by FSIO. 

# Implementation

The overall performance metrics will be implemented using a phased approach. Each phase will build upon the previous phase’s accomplishments and will emphasize communication and coordination with the financial management community.

## Phase I

Phase I will be critical to building a baseline of performance and learning lessons for subsequent phases. The Phase I measures have been selected from the two mandatory service categories: IT Infrastructure Hosting and Administration and Application Management. 

The initial set of seven metrics was developed based on SSP and agency responses to the Value and Burden Survey conducted by the FMLoB during November and December 2006, and refined by an interagency working group. Specific attention was paid to ensuring the metrics selected offered both a high value for potential users and a low burden of collection. The survey results and working group discussions indicated that the collection of cost measures would pose a significant burden, and therefore no cost metrics were selected for Phase I. 

The Phase I metrics to be reported can be found in Appendix A. Detailed descriptions of each metric are included in Appendix B. 

Performance measures will be collected and reported at the system level or by Service Level Agreement (SLA). This will avoid differences in aggregation methodology and improve data accuracy. For reporting done at the system level, performance measures must be entered for each system listed as a Core Financial Management System in the Financial Management System Inventory (See Appendix C for more detail). For SSPs that host multiple agencies on a single system with different SLAs, or agencies that offer different service levels on a single system, performance measures should be reported for each Service Level offered. 

**By June 15, 2007, SSPs and CFO Act Agencies will report data for April 2007 for the metrics included in Appendix A.**Data for subsequent months must be reported by the last business day of the month following the reporting period (according to the schedule included in Appendix C).****

Non-CFO Act agencies are not required to report but are encouraged to participate to determine their relative level of performance. 

If the data for a specific measure is not available, an explanation and estimated date for initial entry must be entered into the system. All data that is available must be reported. FSIO and OMB will work with agencies to develop collection plans for all data currently not available. 

Data will be submitted to FSIO according to the preliminary instructions for data entry included in Appendix C. More detailed instructions and a User’s Guide will be provided prior to data entry. 

Initial reports will be generated and circulated by FSIO, though targets and goals for the measures will not be set until a baseline can be established.

## Phase II

Phase II will focus on expanding and refining the measures for IT Infrastructure Hosting & Administration and Application Management Services, and add a limited set of measures for the Systems Implementation and Business Process categories. Additional metrics beyond these categories may also be considered. A key goal of Phase II will be to add cost measures. Emphasis will also be placed on lowering the collection burden. 

During Phase II, lessons learned from Phase I will be incorporated and result in an update to this guide. 

Planning for Phase II will begin immediately after the Phase I Service Assessment Guide (SAG) is released. Details for the Phase II Kickoff will be sent to the Financial Management Line of Business Transformation Team and the current working group. 

**Appendix A. FMLoB Financial Performance Measures**

**Appendix B. FMLoB Financial Performance Measure Definitions**

**Service Category: IT Infrastructure Hosting and Administration**

**FMLoB Metric 1:IT Hosting Infrastructure Availability**

**What does it measure?**The metric measures the****hours the IT hosting infrastructure is available, expressed as a proportion of the agreed upon hours of availability.

**Why is it important?**IT hosting infrastructure availability is essential to accomplish the customer mission/functions and meet internal and external financial management and reporting requirements.

**Definition:**IT hosting infrastructure is available for use during scheduled/agreed upon hours and days. Scheduled time excludes pre-approved downtime and scheduled downtime.

**Notes:**Measurement is specifically for the IT hosting infrastructure as it relates to core financial management systems.

IT hosting infrastructure includes mainframes, servers, network routers and switches, operating system software and associated communications equipment and facilities. 

Computation of this metric includes network outages that occur beyond your control. When these occur include an explanation in the text field when reporting.

No differentiation between batch and online processing.

In the absence of set service levels (which are often included in an SLA), select an industry standard that meets your business needs. Industry standards are 95%, 98%, or 99.6%, each providing greater uptime reliability and higher costs. 

**Metric Methodology:**Available hours / agreed upon hours (results expressed as a percentage).

**Reporting Frequency:**Monthly

**Calculation Steps:**

1. Determine the actual number of hours the IT hosting infrastructure was agreed upon to be available for the month (per service level agreement, contract, memorandum of agreement, baseline performance measure etc.). Enter agreed upon hours to 2 decimal places.

2. Determine the number of hours the IT hosting infrastructure was actually available for the month. Input actual hours available to 2 decimal places.

3. The reporting system will divide the actual hours available during the month by the obligated hours for the month and express the result as a percentage of obligated hours.

**FMLoB Metric 2:Help Desk Response Time**

**What does it measure?**The metric measures responsiveness of the help desk to the resolution of trouble ticket.

**Why is it important?**The metric provides a means for determining the level of customer service provided. 

**Definition:**Average time to close all help desk trouble tickets closed in the calendar month.

Time is based on your help desk agreed upon hours of operation. Time recorded to resolve a trouble ticket will only include hours of operation. A trouble ticket opened one day and closed the next would only show time when the help desk was available for operation.

**Metric Methodology:**Calculates the average elapsed time to address and close all help desk trouble tickets.

**Reporting Frequency:**Monthly

**Calculation Steps:**

1. Determine the total number of trouble tickets closed for the calendar month period. Enter the total number.

2. Based on your help desk agreed upon hours of operation, compute the total amount of time elapsed in hours for all closed tickets between opening and closure of the ticket. Enter agreed upon hours to 2 decimal places.

3. The reporting system will divide the total elapsed time to close all closed tickets by the total number of closed tickets for the month. 

**FMLoB Metric 3a:System Security Scans**

**What does it measure?**The metric measures the frequency of security scans on core financial systems. 

**Why is it important?**Regular scans to detect security vulnerabilities and ensure the availability, integrity, and confidentiality of federal information are an important component of effective management and oversight of information security programs. 

**Definition:**The number of security scans of servers on which the core financial management applications are hosted.  Scans refer to use of tools and processes for detecting information system vulnerabilities and incidents; such as, network and host-based intrusion detection systems, antivirus software, and file integrity checking software.

The following publications are applicable to this metric:

FIPS 200, March 2006, Minimum Security Requirements for Federal Information and Information Systems

FIPS 199, February 2004, Standards for Security Classification of Federal Information and Information Systems

Special Publication 800-53, December 2006, Recommended Security Controls for Federal Information Systems

Special Publication 800-61, January 2004, Computer Security Incident Handling Guide

United States Computer Emergency Readiness Team Federal Agency Incident Reporting Guidelines and Categories 

**Metric Methodology:**The number of scans conducted during the reporting month.

**Reporting Frequency:**Monthly

**Calculation Steps:**

1. Determine the total number of scans conducted during the calendar month. Enter the number of scans.

**FMLoB Metric 3b: System Security Remedial Actions**

**What does it measure?**The metric measures the time to successfully resolve moderate and high risk security incidents and identified vulnerabilities from security scans.

**Why is it important?**Expedient analysis, resolution, and recovery from security incidents and vulnerabilities impact an organization’s ability to accomplish its assigned mission, protect its assets, fulfill its legal responsibilities, maintain its day-to-day functions, and protect individuals.   

**Definition:**The average time to successfully resolve moderate and high risk information security incidents and vulnerabilities detected from security scans. 

Levels of potential impact (low, moderate, high) for assessing risk associated with vulnerabilities and threats are outlined in FIPS 199, February 2004, Standards for Security Classification of Federal Information and Information Systems

Additional reference material for this metric are:

FIPS 200, March 2006, Minimum Security Requirements for Federal Information and Information Systems

Special Publication 800-53, December 2006, Recommended Security Controls for Federal Information Systems

Special Publication 800-61, January 2004, Computer Security Incident Handling Guide

United States Computer Emergency Readiness Team Federal Agency Incident Reporting Guidelines and Categories 

**Metric Methodology:**Total time to successfully resolve moderate and high security incidents and identified moderate and high vulnerabilities / total number of moderate and high security incidents and identified moderate and high vulnerabilities. 

**Reporting Frequency:**Monthly

**Calculation Steps:**

1. Determine and enter the total number of moderate and high security incidents that were resolved during the calendar month.

2. Determine and enter the total number of moderate and high security vulnerabilities that were resolved during the calendar month.

3. Determine and enter the total elapsed time in hours for resolution of all moderate and high incidents that were resolved during the month. Elapsed time starts with the identification of a security incident and ends with successful resolution of the incident.

4. Determine and enter the total elapsed time in hours for resolution of all identified moderate and high vulnerabilities from security scans successfully resolved during the month. Elapsed time starts with the identification of a security vulnerability from a security scan and ends with successful resolution of the vulnerability.

5. The reporting system will compute the average time to remediate detected moderate and high security incidents and identified vulnerabilities from security scans.

**Service Category: Application Management**

**FMLoB Metric 4:Core Financial System Application Availability**

**What does it measure?**The metric measures the****amount of time the****core financial system application(s) is available on-line, expressed as a proportion of the agreed upon hours of availability.

**Why is it important?**Core financial system application(s) on-line availability is essential to accomplish the customer mission/functions and meet internal and external financial management and reporting requirements.

**Definition:**The amount of time the core financial system application(s) is available for use during scheduled/agreed upon hours. Scheduled time excludes pre-approved downtime and scheduled downtime and downtime requested by the customer.

**Notes:**Measurement is specifically for core financial management systems only.

**Metric Methodology:**Available hours / agreed upon hours (results expressed as a percentage)

**Reporting Frequency:**Monthly

**Calculation Steps:**

1. Determine the number of hours the core financial system application(s) is agreed upon to be available during the month (per service level agreement, contract, memorandum of agreement, baseline performance measure etc.). Enter the number of hours to 2 decimal places.

2. Determine the number of hours the core financial system application(s) was actually available for the month. Enter the number of hours to 2 decimal places.

3. The reporting system will divide the actual hours available during the month by the agreed upon hours for the month and express ratio as a percentage of agreed upon hours.

**FMLoB Metric 5: Report Production Time**

**What does it measure?**The metric measures the total elapsed time to produce an FMS 224 report and a related selected report from ‘click to run’ to ‘ready to print’. 

**Why is it important?**To determine software and configuration efficiency as it applies to report generation time.

**Definition:**The amount of time it takes to produce two selected reports. This measure also captures the number of general ledger transactions since the beginning of the fiscal year, as contextual information. 

****For the purposes of this report, general ledger transactions will be defined as the total number of debits and credits down to the detail level used to support the trial balance. 

This metric is intended to capture application time only. Elapsed time should be reported as calculated by the system – from ‘click to run’ to ‘ready to print’.

The amount of time it takes to produce an FMS 224  – Statement of Transactions (or Partial FMS 224) and one of the following reports as applicable to your agency:

· SF 1218 – Statement of Accountability (Foreign Service Account)

· FMS 1219 – Statement of Accountability

· FMS 1220 - Statement of Transactions According to Appropriation, Funds, and Receipt Accounts

· SF 1221 – Statement of Transactions According to Appropriation, Funds, and Receipt Accounts (Foreign Service)

If your agency does not produce any of these reports, please report the nearest substitute. For guidance, please contact FSIO.

The calculation should not include “no transaction” reports, i.e., reports for accounting periods in which no transactions occurred.

This reporting requirement may change as standard FMS reporting requirements change.

**Metric Methodology:**This metric collects two data points:

1)Total elapsed time to produce two standard reports from ‘click to run’ to ‘ready for print’, and 

2)The total number of general ledger transactions since the beginning of the fiscal year.

**Reporting Frequency:**Monthly

**Calculation Steps**

1. Determine the total number of cumulative general ledger transactions for the fiscal year, as of the end of the reporting month. Enter the total number of general ledger transactions. 

2. Identify FMS 224 report in drop down menu box and select.

3. Determine and enter the elapsed time in hours to tow decimal place that it takes to run the FMS 224 (if applicable).

4. Choose a second report in the drop down menu box and select, or select not applicable (NA) if you only run one report.

5. Determine and enter the elapsed time in hours and minutes that it takes the system to run the selected report from ‘click to run’ to ‘ready to print’.

**NOTE:** The FMLOB Performance Measurement Team understands that the data for this metric will be difficult to compare across agencies, and may not be reliable. This data is being captured in order to learn lessons for Phase II. Agencies expressed a strong desire to see a measure of reporting timeliness, and the working group determined that it was necessary to begin collecting a baseline of data to build an understanding of the issues involved. Number of transactions is captured as a proxy for agency size and financial complexity, and in the future may be used for peer groupings and as a factor that can be controlled for. 

Please report all available data, and participate in future working group sessions to contribute toward further refinements of this metric. 

**FMLoB Metric 6:Average Response Time for User Access Requests**

**What does it measure?**The metric measures the****efficiency of the process for granting system access to new users.

**Why is it important?**(1) Evaluates provider response time;(2) Measures potential loss of user productivity while awaiting access.

**Definition:**Time required for administrator to grant appropriate access, once all necessary approvals are received (final approval), to time of notification to end users of the core financial application system.

Measures from the time/date stamp of final approval to the time the user is notified that access was granted.

Includes new users to the system during ongoing operations but not batch processing of users during transition to a new system.

**Metric Methodology:**Total elapsed time between approved new user requests granting appropriate user access, divided by the number of user access requests granted during the period.

**Reporting Frequency:**Monthly

**Calculation Steps:**

1. Measure elapsed time of each new user request from the time/date stamp of final approval to time the user is notified that access was granted. 

2. Sum the elapsed time for each user notified that access was granted, and enter the total elapsed time in minutes.

3. Enter the total number of user requests granted during the period.

4. The reporting system will divide total elapsed time by the number of user requests during the period.

**Appendix C. FMLoB Performance Metric Reporting Procedures**

What follows is preliminary guidance on the reporting procedures, to assist agencies and SSPs when preparing to collect and report these measures. Detailed procedures and a User’s Guide will be issued prior to data entry, after the system is fully developed.

**Preliminary Guidance**

The FMLOB Performance Metrics will be collected in a web based environment at the Federal Interagency Databases Online (www.FIDO.gov). The metrics will be reported as part of the Chief Financial Officers’ Metric Tracking System (MTS). 

Instructions for gaining access to this system will be distributed along with the User’s Guide. Within the MTS system you will have two options: 

· Financial Management Health Metrics (FMHM; the current MTS metrics)

· Financial Management Services Metrics (FMSM) 

The FMLOB performance metrics will be reported under the Financial Management Services Metrics (FMSM) option. 

Within FMSM you will be able to select your agency. Within your agency you will see a list of the core financial management systems entered into the Financial Management Systems Inventory (FMSI).

Metrics will be reported for each core financial management system. Core financial systems are fully described in the Core Financial System Requirements Document, which can be accessed at www.fsio.gov. If your agency or SSP has multiple core systems, measures must be reported for each one. You are only required to report measures for your core financial system and not all of its components or feeder systems. We will be making changes to FMSI to distinguish between the core system and its components. More details on these changes will be included in the User’s Guide. In addition, you will be able to add reporting lines for Service Level Agreement (SLA) reporting or to report for a system not entered into FMSI. 

It will be possible for multiple users to enter data, and the agency administrator will be able to determine access rights to individual systems. Information will be able to be saved and reviewed prior to final submission. 

If the data for a specific measure is not available, an explanation and estimated date for initial entry must be entered into the system, into the appropriate field. All data that is available must be reported, for each metric field. FSIO and OMB will work with agencies to develop collection plans for all data currently not available. 

**Reporting deadlines**

***** these dates were adjusted for the end of the fiscal year from September 28, 2007 to October 5, 2007, and the end of the calendar year from December 31, 2007 to January 4, 2008

**Appendix D. Current Performance Measures Working Group**

(all CFO Act agencies and SSPs are invited to participate)