**[Performance-Based]{.smallcaps}**

**[Management]{.smallcaps}**

Eight Steps

To Develop and Use

Information Technology

Performance Measures Effectively

**For further information contact:**

Patrick T. Plunkett

General Services Administration

Office of Governmentwide Policy

1800 F Street, NW.

Washington, DC 20405

Telephone: (202) 501-1123

**[Table of Contents]{.smallcaps}**

Foreword

Intended Audience v

Executive Summary

Introduction

Step 1: Link Information Technology Projects to Agency Goals and
Objectives

Step 2: Develop Performance Measures

Step 3: Establish a Baseline to Compare Future Performance

Step 4: Select Information Technology Projects with the Greatest Value

Step 5: Collect Data

Step 6: Analyze the Results

Step 7: Integrate into Management Processes

Step 8: Communicate the Results

Things to Consider

Supplement 1: Developing Performance Measures

Supplement 2: Selecting IT Projects with the Greatest Value

Appendix A---Key Success Factors for an Information Systems Performance

Measurement Program

Appendix B---Agency Measures

Appendix C---Performance Measurement Legislation

Appendix D---Omb and GAO Investment Factors

Appendix E---Recommended Reading List

Foreword

The General Services Administration's (GSA) Office of Governmentwide
Policy developed this guide to help those who want to gain a further
understanding of performance measurement and for those who develop and
use performance measures for information technology (IT) projects.

Recent documents related to IT performance measurement were developed by
the Office of Information and Regulatory Affairs (OIRA) in the Office of
Management and Budget (OMB) and the General Accounting Office (GAO).
This paper complements the OIRA guide, "Evaluating Information
Technology Investments" and the framework provided in the soon-to-be
released GAO Exposure Draft, "Information Technology --- Measuring for
Performance Results."

The OIRA guide sets out an analytical framework linking IT investment
decisions to strategic objectives and business plans in Federal
organizations, and supplements existing OMB policies and procedures. The
approach relies on the consistent use of performance measures to
indicate potential problems. It emphasizes the need for an effective
process when applying information technology in this period of reduced
resources and greater demand for government services.

The GAO guide assists in creating and evaluating IT performance
management systems. It provides examples of current performance and
measurement practices based upon case studies. GAO recognizes the need
for more research and analysis, but asserts that these practices serve
as a starting point to establish effective strategic direction and
performance measurement requirements.

This document presents an approach to help agencies develop and
implement effective IT performance measures. Patrick Plunkett, a senior
analyst with GSA\'s Office of Governmentwide Policy, developed the
approach based on many valuable inputs from colleagues at numerous
federal agencies and on research of performance measurement activities
in state governments and in private industry.

GSA is grateful to the following individuals for providing their time
and sharing their performance measurement experiences:

**Defense Commissary Agency**

-   John Goodman

```{=html}
<!-- -->
```
-   Tom Hardcastel and Mark Schanuel, Logistics Management Institute

**Defense Finance and Accounting Service**

-   Audrey Davis

**Department of Defense Office of the Director of Research and
Engineering**

-   Larry Davis, User Technology Associates

**Federal Aviation Administration**

-   Louis Pelish

-   Roni Raffensperger, CSSI

**Immigration and Naturalization Service**

-   Janet Keys

-   J.T. Lazo, Electronic Data Systems

-   Linda Goudreau, Dave Howerton and Dave Ziskie, Electronic Data
    Systems

**Social Security Administration**

-   Jim Keiner and Vince Pianalto

The following individuals at GSA enhanced the readability of this guide:

-   Sandra Hense, Don Page, Virginia Schaeffer, Joanne Shore, and Judy
    Steele

Intended Audience

This document is for anyone who develops and implements performance
measures for information technology (IT). It is also intended for those
who want to understand the principles of performance measurement. This
guide describes the major tasks to follow to measure the contribution of
IT projects to an organization's goals and objectives. These same
principles and tasks also apply when measuring mission performance.

Organizations succeed when their business units and support functions
work together to achieve a common goal. This holds true for performance
measurement, which entails more than just developing performance
measures. It also includes establishing business strategies, defining
projects that contribute to business strategies, and evaluating, using
and communicating the results to improve performance.

The following are descriptions of the principal roles associated with
each step. The roles vary by organization:

-   Step 1 - Senior management translates vision and business strategies
    into actions at the operational level by creating a Balanced
    Scorecard for the organization. Business units and IT professionals
    contribute to the Balanced Scorecard by defining the information and
    IT capabilities that the organization needs to succeed. The IT
    professionals include managers, analysts and specialists who plan or
    analyze requirements.

-   Steps 2 through 8 (except 4) - IT professionals solicit feedback
    from business units to refine the information and capabilities
    defined in Step 1; create a Balanced Scorecard for the IT function
    and develop performance measures; and communicate results. Together,
    IT professionals and business units establish baselines, and
    interpret and use results to improve performance. The IT
    professionals include managers, analysts and specialists who plan,
    analyze or deliver IT assets and services.

-   Step 4 - IT professionals estimate the cost, value and risk of IT
    projects to perform Information Economics calculations. Senior
    management and business unit managers define the evaluation factors
    and their associated weights to evaluate IT projects. Then they
    determine the value of each IT project and select the projects that
    provide the greatest value. The IT professionals include managers,
    analysts and specialists who analyze the cost or benefits of IT
    solutions.

Executive Summary

The General Services Administration (GSA) prepared this guide to help
agencies develop and implement effective information technology (IT)
performance measures. Effective performance measures are customer
driven; give an accurate and comprehensive assessment of acquisitions,
programs, or activities; minimize the burden of data collection; and are
accepted and used to improve performance.

Performance-based management links investment planning with the
systematic use of select feedback to manage projects and processes.
Projects cannot be managed unless they are measured. The "eight steps"
constitute a measurement process that includes translating business
strategies into actions at the operational level; selecting projects
that have the greatest value; developing measurement mechanisms;
measuring, analyzing and communicating the results; and finding ways to
improve performance. The eight steps provide a logical sequence of tasks
that can be integrated with existing management practices.

Successful performance-based management depends upon the effective use
of performance measures. The steps to develop and use IT performance
measures effectively are:

> Step 1: Link IT Projects to Agency Goals and Objectives
>
> The effective measurement of an IT investment's contribution to agency
> accomplishments begins during the planning stage. Done properly, IT
> investment planning is based upon the agency mission and strategic
> business plans. IT organizations build partnerships with program
> offices and functional areas to define projects that contribute to the
> agency's goals and objectives. Linking IT projects to goals and
> objectives can be done using a framework known as the "Balanced
> Scorecard." The Balanced Scorecard consists of four perspectives that
> provide a comprehensive view of a business unit. The perspectives
> include Financial, Customer, Internal Business, and Innovation and
> Learning. The Balanced Scorecard in Step 2 also serves as a framework
> to assess performance.
>
> Step 2: Develop Performance Measures
>
> To assess the efficiency and effectiveness of projects, select a
> limited number of meaningful performance measures with a mix of short-
> and long-term goals. For large IT projects, the project manager or
> another key individual leads a team to develop the measures. Measure
> the outcomes of the IT investment, not just its cost, timeliness and
> quality. An outcome is the resulting effect of the IT investment on an
> organization. Examples include measurable improvements in the quality
> and delivery of the organization's services and products.
>
> To develop performance measures, determine the objectives of the
> project; decide how requirements will be met; know the purpose of the
> results; and understand why the results matter. Measure that which is
> most important. Agencies will improve the quality of their measures
> and ensure acceptance if their IT organizations develop and nurture
> partnerships with customers and stakeholders. Effective performance
> measures reflect a strong customer focus.
>
> Step 3: Establish Baseline to Compare Future Performance
>
> Baselines enable agencies to determine whether performance improves or
> declines as a result of an IT investment. Valid baselines are
> documented, recognized and accepted by customers and stakeholders.
> Standard agency reports can serve as the baseline if, and only if, the
> reports apply to the indicators chosen. If no baseline exists, then
> the performance measures establish the baseline.
>
> Step 4: Select IT Projects with the Greatest Value
>
> In today's tight budget environment, agencies can only fund a limited
> number of IT projects. Consequently, agencies need to select projects
> that provide the greatest value. Value is based on the estimated
> economic return of an IT investment plus its estimated contribution to
> an organization's business priorities. (This guide uses the terms "IT
> projects" and "IT investments" interchangeably.) To select the IT
> investments with the greatest value, establish Investment Review
> Boards (IRBs) to estimate the value and risks of each investment. The
> IRB should comprise the major stakeholders from the agency's core
> functional areas and program offices.
>
> Step 5: Collect Data
>
> The optimal time to focus on the data needed for the chosen indicators
> is during Steps 2 and 3. Agencies need to ask: "What data are needed
> to determine the output of the project? What data are needed to
> determine the effectiveness of the project?" The data used will depend
> upon availability, cost of collection and timeliness. Accuracy of the
> data is more important than precision.
>
> Step 6: Analyze Results
>
> After obtaining results, conduct measurement reviews to determine if
> the project met the objectives and whether the indicators adequately
> measured results. A key question is: "Do the results differ from what
> we expected?" During reviews, seek ways to improve performance, refine
> indicators and identify lessons learned for future projects. The most
> useful performance reports track results over time and permit
> identification of trends.
>
> Step 7: Integrate with Management Processes
>
> To assure that results improve performance, integrate them with
> existing management processes. If the results are not used, no one
> will take the measurement process seriously. Laws require agencies to
> submit performance reports with their budget submissions. Because it
> may take years to realize a project's results, agencies face the
> challenge of identifying results in their annual budget submissions.
>
> Step 8: Communicate Results
>
> Take the initiative to communicate results internally to improve
> coordination and increase the focus of workers and managers. Leverage
> results by sharing them with OMB and Congress to obtain support and
> continued funding. Communicate results with customers and the public
> to foster and sustain partnerships.

Implementing Performance-Based Management

Performance measurement requires an investment in resources. Some
Federal implementors believe that organizations should dedicate
resources up-front to properly set up their measurement structure.
Reports from industry and state governments confirm that organizations
use more resources initially to develop a knowledge and skills base and
to instill performance-based management methods in their organizations.
As organizations learn how to develop and use performance measures, less
resources are necessary.

Initially, measuring performance and linking IT projects to organization
outcomes are hard to conceptualize and recognize due to the inherent
ambiguity of outcomes. Practitioners require time and experience before
they can develop and use performance measures effectively. Agencies can
reduce their learning curve by creating performance measurement guides
tailored to their mission.

The amount of resources and time necessary to develop measures depends
on the scope of the project; the extent of the partnership between the
business and technical groups; quantity and quality of available data;
the knowledge and skill of the developers; and the level of proactive
involvement by management. The resources needed to develop and use
performance measures will vary from project to project.

A change in mindset and culture is required to develop and use
performance measures to improve performance. Agencies can lay the
foundation for these changes by encouraging and fostering the use of
performance measures. This will happen only if senior managers support
and participate in the process itself.

It will take time for agencies to institutionalize performance
measurement. Agencies can accelerate implementation by consistently
using a framework and methodology such as the Balanced Scorecard during
project planning and measurement.

Introduction

The Federal government spends over \$25 billion annually on IT systems
and services. Do these systems and services improve service to the
public? Do these systems and services improve productivity or reduce
costs of Federal agencies? Without measuring and communicating the
results, how will anyone know?

For the remainder of this decade and into the next century, the Federal
government will decrease in size as government balances the Federal
budget. IT will play a significant role in making the Federal government
more efficient and effective as it downsizes. The Clinger-Cohen Act
requires each Executive Agency to establish a process to select, manage,
and evaluate the results of their IT investments; report annually to
Congress on progress made toward agency goals; and link IT performance
measures to agency programs.

The Clinger-Cohen Act evolved from a report by Senator Cohen of Maine,
entitled "Computer Chaos." In the report, Senator Cohen identified major
projects that wasted billions of dollars because of poor management. To
improve the success of IT projects in the Federal sector, Senator Cohen
stated the government needs to do better up-front planning of IT
projects particularly when they define objectives, analyze alternatives
and establish performance measures that link to agency accomplishments.

This publication provides an approach to develop and implement IT
performance measures in concert with guidance provided by OMB and GAO.
It cites and explains an eight step process to link IT investments to
agency accomplishments that meets the requirements of the Clinger-Cohen
Act and the Government Performance and Results Act (GPRA).

Congress and OMB emphasize performance measures as a requirement to
receive funding. Soon, agency funding levels will be determined to a
large degree on the projected results of IT investments and the measures
selected to verify the results. This guide presents a systematic
approach for developing and using IT performance measures to improve
results.

The eight step approach focuses on up-front planning using the Balanced
Scorecard. IT performance measures will be effective if agencies
adequately plan and link their IT initiatives to their strategies. The
Balanced Scorecard translates strategy into action. The eight step
approach is a logical sequence of tasks. In practice, some steps can be
combined. Because performance measurement is an iterative process,
agencies should expect to apply the eight steps repeatedly to obtain
effective performance measures and improve performance.

Step 1: Link Information Technology

> Projects to Agency Goals
>
> and Objectives

The process to effectively measure the contribution of IT projects to
mission results begins with a clear understanding of an agency's goals
and objectives. Linking IT projects to agency goals and objectives
increases the likelihood that results will contribute to agency
accomplishments. Accordingly, this linkage improves an agency's ability
to measure the contribution of IT projects to mission accomplishments.

Accomplishments are positive results that achieve an organization's
goals and objectives. Because information system (IS) organizations and
IT projects support the mission and programs, the organization's vision
and business strategies need to be established before IT projects can be
linked to goals and objectives. To establish clear linkage, strategic
plans need to define specific business goals and objectives and
incorporate IT as a strategic resource.

The GPRA requires executive agencies to develop strategic plans and
performance measures for major programs. (See Appendix D for a summary
of the GPRA.)

Each strategic business unit (SBU) should have a strategic plan. An SBU
is an internal organization that has a mission and customers distinct
from other segments of the enterprise. Processing disability claim
requests, launching satellites, or maintaining military aircraft are
examples of SBUs.

As important as strategic plans can be, they often are forgotten soon
after prepared because they don't translate well into action. In most
cases, business strategies reflect lofty objectives ("Be our customers'
number one supplier.") which are nearly impossible to translate into
day-to-day activities. Also, strategic plans typically focus three to
five years into the future in contrast with performance measures which
focus on on-going operations. This difference in focus causes confusion,
and sometimes conflict, for line managers and program managers.

The Balanced Scorecard (BSC) is a framework that helps organizations
translate business strategies into action. Originally developed for
private industry, the BSC balances short- and long-term objectives.
Private industry routinely uses financial measures to assess performance
although financial measures focus only on the short-term, particularly
the results of the last year or quarter. The BSC supplements financial
measures with measures from three perspectives: Customer, Internal
Business and Innovation and Learning.

The Customer Perspective examines how customers see the organization.
The Internal Business Perspective examines the activities, processes and
programs at which the organization must excel. The Innovation and
Learning Perspective, also referenced as the Growth Perspective,
examines ways the organization can continue to improve and create value
by looking at processes, procedures and access to information to achieve
the business strategies.

Used effectively, these three perspectives drive performance. For
example, hypothetical Company XYZ developed a BSC that measures customer
satisfaction. Their current assessment indicates a serious level of
customer dissatisfaction. If not improved, lower sales will result. At
the same time, however, the company's financial measures for the last
two quarters indicate that sales are healthy. With only financial
measures, management would conclude erroneously that the business is
functioning well and they need not make changes. With the additional
feedback from the customer measures, however, management knows that
until recently they performed well, but that something is causing
customer dissatisfaction. The company can investigate the cause of the
results by interviewing customers and examining internal business
measures. If the company is unable to improve customer satisfaction,
eventually the result (lower sales) will appear in the financial
measures.

The BSC provides organizations with a comprehensive view of the business
and focuses management on the handful of measures that are the most
critical. The BSC is more, however, than a collection of measures. If
prepared properly, the BSC contains a unity of purpose that assures
measures are directed to achieving a unified strategy. "Every measure
selected for a BSC should be an element in a chain of cause-and-effect
relationships, linkages, that communicates the meaning of the business
unit's strategy to the organization."[^1] For example, do process
improvements increase internal business efficiency and effectiveness? Do
internal business improvements translate into improved customer service?

A good BSC incorporates a mix of outcome and output measures. Output
measures communicate how the outcomes are to be achieved. They also
provide an early indication about whether or not a strategy is being
implemented successfully. Periodic reviews and performance monitoring
tests the cause-and-effect relationships between measures and the
appropriateness of the strategy.

Figure 1 illustrates the use of the BSC to link the vision and
strategies of an SBU to critical performance measures via critical
success factors. The BSC allows managers to examine the SBU from four
important perspectives and to focus the strategic vision. The business
unit puts the BSC to work by articulating goals for time, quality, and
performance and service and then translates these goals into specific
measures.

![](media/image1.wmf)

*Figure 1*[^2] - *The Balanced Scorecard at Work*

For each perspective, the SBU translates the vision and mission into the
factors that will mean success. For the success factors to be critical,
they must be necessary and sufficient for the SBU to succeed. Each
critical success factor or objective needs to focus on a single topic
and follow a verb-noun structure. For example, "Improve claims
processing time (Internal Business Perspective) by 'X' percent by 'Y'
date." The more specific the objective, the easier it will be to develop
performance measures. The less specific the objective, the more
difficult it will be to develop performance measures.

Figure 2 shows how Rockwater, a worldwide leader in underwater
engineering and construction, applied the BSC. A senior management team,
that included the Chief Executive Officer, developed the vision and the
four sets of performance measures to translate the strategy and critical
success factors into tangible goals and actions.

Figure 2[^3] - Rockwater's Balanced
Scorecard![](media/image2.wmf){width="6.529861111111111in"
height="5.0625in"}

Rockwater has two types of customers. Tier 1 customers are oil companies
that want a high value-added relationship. Tier 2 customers are more
interested in price. Before using the Balanced Scorecard, Rockwater's
metrics focused on price comparisons with its competitors. Rockwater's
strategy, however, emphasized value-added business. The Balanced
Scorecard enabled Rockwater to implement its strategy and make
distinctions between its customers.

Organizations are unique and will follow different paths to build the
Balanced Scorecard. At Apple Computer and Advance Micro Devices, for
example, a senior finance or business development executive, intimately
familiar with the strategic thinking of the top management group,
constructed the initial scorecard without extensive deliberations.
Kaplan and Norton provide a profile to construct a scorecard.[^4]

The BSC provides Federal agencies with a framework that serves as a
performance measurement system and a strategic management system. This
framework allows agencies to:[^5]

-   Clarify and translate vision and strategy

-   Communicate and link strategic objectives and measures

-   Plan, set targets, and align strategic initiatives

-   Enhance strategic feedback and learning

Because Federal agencies do not have the profit motive of private
industry, the orientation of the BSC is different. For private industry,
the Financial Perspective represents and assesses a company's
profitability. The other perspectives represent and assess a company's
future profitability. For government, the Financial Perspective
represents the goals to control costs and to manage the budget. The
Customer Perspective represents and assesses programs to serve taxpayers
or society, other government agencies or other governments. The Internal
Business and the Innovation and Learning perspectives represent and
assess the Government's ability to continually complete its mission.

The BSC addresses the contribution of IT to the business strategy in the
Learning and Innovation Perspective. The contribution includes improved
access to information that may improve business processes, customer
service and reduce operating costs. After the desired business outcomes
and outputs are determined, the IT needs can be identified. A separate
BSC is recommended for the IT support function to integrate and assess
the IT services provided to the organization. Step 2 addresses the use
of the BSC for the IT function and specific projects.

Clear strategic objectives, definitive critical success factors, and
mission-level performance measures provide the best means to link IT
projects to agency goals and objectives and ultimately agency
accomplishments. Some believe that IT performance measures cannot be
established until this has been done. Others believe that IT
organizations must take the lead within their parent organizations to
establish performance measures. Agencies may risk funding for their IT
projects if they wait until critical success factors and mission-level
measures are in place before developing IT performance measures. Whether
the cart is before the horse or not, the experience gained from
developing and using IT performance measures helps agencies develop more
effective performance measures.

Agencies can identify information needs while developing strategic plans
by having a member of the IT project management team (an individual who
has extensive knowledge of the agency's programs and operations)
involved in development of the strategic plans. At the least, grant a
member access to the latest version of the plan. To identify information
needs, agencies should define the following:

-   Critical success factor(s) to be implemented

-   Purpose and intended outcome

-   Outputs needed to produce intended outcomes

-   Users of the resulting product or service

-   What the resulting product or service will accomplish

-   Organizational units involved and their needs

IT professionals identify IT solutions that contribute to their agency's
strategies and programs. They do this by exploring ways to apply
technology to achieve one or more critical success factors. This
requires an understanding of the organization, its structure and its
operating environment. Successful IT project managers understand their
agency's programs and processes and can describe how technology fosters
improvement in agency business performance.

Linking IT projects to agency objectives requires involvement by senior
management and consensus among stakeholders. Senior managers possess the
broad perspective necessary for strategic planning. Stakeholders (e.g.,
managers, workers, support organizations, OMB and Congress) have a
vested interest in the project. They judge if linkage exists and to what
degree it exists. The IT project manager identifies the stakeholders and
works to obtain their agreement and support. The project manager faces
the challenge of balancing the interests of internal and external
stakeholders which often differ.

Example of An IT Project Linked To Agency Goals And Objectives

Figure 3 shows how the Immigration and Naturalization Service (INS)
linked its Integrated Computer Assisted Detection (ICAD) system
performance measures to the agency's objectives. ICAD is the second
generation of automated assisted detection systems used by the United
States Border Patrol (USBP). With the installation of remote field
sensors connected to Border Patrol communication facilities, ICAD
displays remote sensor activity, processes incident (ticket)
information, maintains the status of Border Patrol Agents in the field,
provides access to state and national law enforcement services, and
generates a variety of managerial reports. USBP management utilizes
information that ICAD produces to make tactical decisions on the
deployment of Border Patrol resources and strategic decisions on future
Border Patrol operations.

The INS developed performance measures to show the effect of ICAD at the
strategic, programmatic and tactical levels of the organization. At the
tactical level, the ICAD performance measures indicate the number of
unlawful bordercrossers detected in two categories: migrant and
smuggler. By increasing the effectiveness of the border patrol
(programmatic level), ICAD contributes to achievement of the strategic
goal to promote public safety by deterring criminal aliens. Figure 3
also shows the information INS uses to assess this goal.

Although the INS did not employ the BSC framework, they did use the
following principles of the BSC: link IT to organization strategy; use a
mix of short- and long-term measures; and select measures that have
cause-and-effect relationships.

![](media/image3.wmf){width="6.5in" height="5.781944444444444in"}

Figure 3 --- The Immigration and Naturalization Service's Objectives and
Measures for the

> Integrated Computer Assisted Detection System

Step 2 describes ways to determine what to measure and how to measure IT
projects. Step 2 also provides an example of an agency IT measure and
describes how to develop IT performance measures using the Balanced
Scorecard.

Step 2: Develop Performance Measures

No one set of performance measures will be effective for all agencies or
for all projects. Organizations differ and their priorities change over
time. To be effective, measures must be tailored to the organization's
mission and management style. Given that, certain universal concepts and
principles apply to agency programs and IT investments.

The concept of performance measurement is straightforward: You get what
you measure; and you can't manage a project unless you can measure it.
Measurement focuses attention on what is to be accomplished and compels
organizations to concentrate time, resources and energy on achievement
of objectives. Measurement provides feedback on progress toward
objectives. If results differ from objectives, organizations can analyze
the gaps in performance and make adjustments.

Applying the measurement concept to the complex business of government,
however, is not as straightforward as it is in the manufacturing sector
where a clear "bottom line" exists. For support functions such as
information technology, the connection to a bottom line or to the
mission of the organization is not always obvious. By integrating the
principles of performance measurement into management practices, the
connection becomes clearer.

Historically, organizations measured the cost of operating data centers,
user reports, lines of print, communications and other elements. Seldom
did they measure the contribution of IT to overall organizational
performance. As mentioned earlier, the Clinger-Cohen Act mandates that
federal agencies measure the contribution of IT investments to mission
results.

The principles of performance measurement apply to mission-level
programs, procurements and IT investments. The principles include the
relationship of inputs, outputs, outcomes and impacts. Figure 4
represents this relationship through the ideal flow of results.

Each project employs people, purchased inputs and some forms of
technology. These constitute the inputs. A project transforms the inputs
into products or services (outputs) for use by customers. Customers can
be taxpayers, other government agencies or internal agency personnel who
receive or use the products and services. The outcomes are the effects
of the output on the customers. Impacts are the long-term effect of the
outcomes. The cloud around the impacts indicates that the impacts are
difficult to discern. Semantically, it is difficult to distinguish
between long-term outcomes and impacts.

![](media/image4.wmf){width="6.580555555555556in"
height="3.229861111111111in"}*Figure 4---Ideal Flow of Results*

The arrows represent cause-and-effect relationships and should be read
as "lead to." The thickness indicates the strength of the
cause-and-effect relationships. There is a direct relationship between
the level of input and the level of outputs. Outputs lead to outcomes
but the relationship is less direct than inputs to outputs. Outcomes
lead to impacts but the relationship is often negligible, if existent,
and difficult to determine. An ideal flow occurs when a relationship
exists between inputs and impacts.

The time line provides a context as to when the results occur and will
vary by types of projects and between projects. Near-term could be
twelve months. Long-term could represent one to three years or even
longer. For example, the benefits to the organization (outcomes) as a
result of an investment in IT infrastructure may take up to three years
to be realized. An investment in a system to improve claims processing
could accrue benefits within one to three months.

To illustrate a flow of results for an IT project, consider a project to
automate the identification of fingerprints to facilitate law
enforcement. The inputs to the project include government personnel
(technical, managerial, and contractual), contractor personnel and the
IT systems used to develop the system since it is not commercially
available. The systems to be developed are the outputs of the project.
The desired outcomes are reductions in the time to identify fingerprints
and the costs of identification. The desired impacts of the system on
law enforcement groups (local, state and Federal) may be to shorten the
time of investigations and increase conviction rates.

Initially, it is easy to get confused with the terminology and lose
focus on the measurement principles and the cause-and-effect
relationships between activity and results. Another way to look at the
type of results is to think in terms of efficiency and effectiveness.
Efficiency is about doing things right (output) and effectiveness is
doing the right things (outcomes). Doing the right things that
contribute to overall success is more important than just doing things
right on a project.

Determining What to Measure

Effective performance measures concentrate on a few vital, meaningful
indicators that are economical, quantitative and usable for the desired
results. If there are too many measures, organizations may become too
intent on measurement and lose focus on improving results. A guiding
principle is to measure that which matters most.

To assess the business performance of IT, agencies may want to consider
the following categories: [^6]

+------------------+---------------------------------------------------+
| **[Categor       | **[Definition]{.smallcaps}**                      |
| y]{.smallcaps}** |                                                   |
+------------------+---------------------------------------------------+
| -   Productivity | Efficiency of expenditure of IT resources         |
+------------------+---------------------------------------------------+
| -   User Utility | Customer satisfaction and perceived value of IT   |
|                  | services                                          |
+------------------+---------------------------------------------------+
| -   Value Chain  | Impact of IT on functional goals                  |
+------------------+---------------------------------------------------+
| -   Competitive  | Comparison against competition with respect to    |
|     Performance  | business measures or infrastructure components    |
+------------------+---------------------------------------------------+
| -   Business     | Criticality of the organization's operating       |
|     Alignment    | systems and portfolio of applications to business |
|                  | strategy                                          |
+------------------+---------------------------------------------------+
| -   Investment   | Impact of IT investment on business cost          |
|     Targeting    | structure, revenue structure or investment base   |
+------------------+---------------------------------------------------+
| -   Management   | Senior management's understanding of the          |
|     Vision       | strategic value of IT and ability to provide      |
|                  | direction for future action                       |
+------------------+---------------------------------------------------+

In Step 1, the BSC provided a framework that translated business
strategies into four perspectives: Financial, Internal Business,
Customer, and Learning and Growth. These important perspectives give a
comprehensive view to quickly assess organizational performance. The
Balanced Scorecard focuses on strategy and vision, not on control.[^7]

Information needs, or desired outcomes from IT systems, that are linked
to business goals and objects are identified after developing the
critical success factors for each perspective. These outcomes and
information strategies are contained in the Learning and Growth
Perspective. In a good BSC, they drive performance and link to
objectives in one or more of the other perspectives.

Figure 5 shows how the BSC can be used as a strategic management system.
Organizations translate their business strategies into objectives for
each perspective. Then they derive measures for the objectives and
establish targets of performance. Finally, projects are selected to
achieve the objectives. Step 4 describes a method to select projects
that provide the greatest value. The arrows indicate the linkage between
perspectives and an organization's vision and strategy.

![](media/image5.wmf){width="6.5in" height="4.875in"}

Figure 5---The BSC as a Strategic Management System[^8]

A separate BSC for the IT function helps align IT projects and
supporting activities with the business strategies. An IT function's BSC
links to the Learning and Growth Perspective in the parent
organization's BSC. A BSC for the IT function also translates an
organization's IT strategic plans into action via the four perspectives.

In an IT BSC, the Customer Perspective represents primarily the
organization's business domain. This perspective may include the
organization's customers. The Internal Business Perspective represents
the activities that produce the information required by the business
domain. The Financial Perspective represents the cost aspects of
providing information and IT solutions to the organization. The Learning
and Growth Perspective represents the activities to improve the IT
function and drive performance in the other perspectives.

To determine what to measure, IT organizations with customers and
stakeholders first need to determine the desired outcomes. For IT
projects, determine how the system will contribute to mission results
(benefits). Then, determine the outputs needed to produce the desired
outcomes. Next, analyze alternatives and select the project(s) that will
produce the needed outputs. Finally, determine the inputs needed to
produce the outputs. Agencies can develop meaningful measures using the
formulation questions of Figure 6 with the Balanced Scorecard in Figure
5. (See Supplement 2 for a list of sample measures for each
perspective.)

Figure 6 - Questions to Formulate Performance Measures

The concept of translating an organization's business strategies using
the Balanced Scorecard framework is the same for an SBU and the IT
function. If the business unit has not defined, or is in the process of
defining, its BSC, an organization can build a BSC for its IT functions.
The BSC can also be used to assess IT projects and manage the processes
that support their completion. This is done by examining the IT projects
via the four perspectives using the concepts presented. The challenge
becomes aligning IT projects and associated activities with business
strategies that may not be specific. Eventually, the IT and business
unit BSCs need to be synchronized in the future.

For some IT projects, it may not be important to have a measure for each
perspective. Yet, agencies need to attempt to develop goals and measures
for each perspective before making that determination. The objective is
to identify a few meaningful measures that provide a comprehensive
assessment of an IT project. The advantage of the Balanced Scorecard is
that it facilitates alignment of activities to achieve goals.

For example, an organization wants to improve its effectiveness by
making better decisions based on the cost of performance. The
organization's strategy is to implement activity based management (ABM).
ABM is a technique for managing organizational activity based on the
actual cost of the activity. The current accounting procedures allocate
costs, overhead, for example, on an organizational unit basis. It does
not provide the level of data needed for ABM. Correspondingly, the
current accounting system does not provide the needed cost information.

To implement the organization's strategy, activity-based costing data is
needed. Applying the BSC framework to the IT function, the first step is
to establish objectives for each perspective. Customers are concerned
with time, quality, performance and service, and costs. The organization
establishes goals for these concerns and then translates them into
measures for the Customer Perspective. The organization establishes
goals for its Internal Business Perspective for the processes that
contribute to customer satisfication. The organization can either modify
the existing system or an off-the-shelf product with their developers or
outsource to a software company.

The Financial Perspective focuses on cost efficiency and effectiveness
consistent with the IT strategy to reduce the amount of money spent on
legacy systems. It also focuses on providing the information within
budgeted costs. The organization examines the Growth Perspective and
establishes goals for skill levels for software development or
acquisition management. The organization also establishes goals for
improving the procedures to provide or acquire the necessary software
services.

When constructing a Balanced Scorecard, practitioners choose measures
that assess progress toward objectives. Creating the Balanced Scorecard
requires involvement by senior managers because they have the most
comprehensive view of the organization and their support is necessary
for acceptance within the organization.

Deciding How to Measure

Measurement is an iterative process. It focuses an organization on what
matters most, that in turn, results in higher performance. Developing
performance measures communicates an organization's objectives and
aligns activities to achieve them. This is accomplished over time by
communicating assumptions about the objectives and the organization and
building consensus with associates. Measurement requires the involvement
of a range of employees. Implementors often refine their measures to
assess the results that are most useful. Measuring performance should
not be costly and time consuming. Initially, additional time will be
necessary for training and experimentation. The time and resources
needed will diminish as performance measurement is integrated into
management processes.

To implement their performance measures for the Integrated
Workstation/Local Area Network (IWS/LAN) project, the Social Security
Administration (SSA) defined the information shown in Figure 7. The
information describes the necessary items to implement the measure:
what, how, who and when. Category 1 is one of six categories of measures
developed for the IWS/LAN project. See Appendix B for the complete set
of measures and measures for other projects.

Figure 7--- One of the Social Security Administration's Performance
Measures

A combination of output and outcome measures provides an effective
assessment. Output measures record whether or not what was done was done
correctly and if the products or services were provided as intended.
Outcome measures assess whether the completed work contributed to the
organization's accomplishments. For example, output measures for an
acquisition of a high performance computer assess if the computer
complied with the specifications and was delivered on-time and within
budget. Outcome measures assess how much, if any, the high performance
computer improved the quality and timeliness of the design of weapon
systems or weather prediction, for example.

Outcome measures have more value than do output measures. Outcomes can
only be measured, however, upon completion of a project. Measuring
intermediate outcomes, if possible, provides an assessment before
completion of a project. For example, by implementing a nation-wide
system in stages, an agency could assess the performance in one region
of the country before implementing the system in other areas.

If an agency cannot develop an outcome measure for an IT project,
agencies will have to use business logic to ascertain if the outputs are
meaningful and contribute to agency accomplishments. Business logic is
based upon common sense and an understanding of the organization,
mission and technology gained through knowledge and experience.

Setting Goals

When establishing goals or targets of performance, it is important to
have a mix of near-term and long-term goals. This is necessary because
it may take years to realize the benefits of the IT investment. This
holds particularly true for IT infrastructure investments where benefits
may not occur for three to five years. Near-term (less than one year)
targets may require organizations to segment projects into modules.

To identify goals, agencies can benchmark other internal projects, other
Federal or state agencies and corporations. Benchmarking is a systematic
examination to locate and investigate other organizations' practices,
processes and results in order to make a true comparison. Benchmarking
provides an effective technique whereby agencies compare themselves to
world-class organizations. Using this technique, agencies can learn what
customers expect of quality, what competitive goals are and how to
achieve them. For benchmarking to be effective, organizations with
similar processes must be found and workers must have knowledge of
benchmarking methods and data collection.

For its Infrastructure Project, the Defense Finance and Accounting
Service (DFAS) benchmarked companies to learn the ratio of network nodes
per administrator that is efficient and effective for industry. DFAS
learned that the ratio 150:1 was common. DFAS also learned valuable ways
to manage and operate infrastructures (local area networks, servers,
etc.).

The Social Security Administration met with representatives of the
Wal-Mart Corporation to discuss IT issues common to their
similarly-sized organizations. SSA is an active member of the Gartner
Group Networking Best Practices Group. Through this membership, SSA
meets with its peers in other corporations to discuss telecommunications
connectivity and implementation issues. The exchange of experiences and
knowledge from group participants, such as Shell, Southwestern Bell and
Allstate, enables SSA to apply best practices in managing its IWS/LAN
and wide area network infrastructure.

Step 3: Establish a Baseline to

Compare Future Performance

The baseline is an essential element of performance measurement and
project planning. Without a baseline, goals are mere guesses.
Establishing baselines primarily involves data collection and consensus
building. The importance of this step requires its separation as a
single task. For agencies to assess future performance, they must have a
clear record of their current level of performance.

If no baseline exists for the measures chosen, agencies can establish a
baseline when they collect the results data. To be effective, the
baseline must support the measures used. Establishing a baseline
requires collecting data about current processes, work output and
organizational outcomes. Some agencies, such as the Social Security
Administration and Defense Commissary Agency (DeCA), have done this for
years.

The SSA has been collecting a wide range of productivity data on its
operations for over twenty years. The SSA measures productivity by
office---the number of cases processed by month and the amount of time
to process a claimant's request.

Although DeCA has only existed as an organization since 1991, defense
commissaries have existed for decades. Each commissary maintains, on a
monthly basis, records of operating expenses and sales. DeCA established
its baseline by selecting a typical site and using existing cost and
revenue reports. Both SSA and DeCA will use existing reports to
determine whether productivity increases and expenses decrease.

The SSA and DeCA will establish baselines just prior to system
implementation because the level of performance may have changed over
the two years since project initiation. These agencies believe that
establishing baselines before implementation will allow them to
accurately measure the contribution of their IT projects to agency
programs and operations.

Agencies can save time and resources by using existing reports, as SSA
and DeCA did. To be worthwhile, formally document baselines and assure
their acceptance by customers and stakeholders. The baseline could be
based on the output of current operations, costs, productivity,
capacity, level of customer satisfaction or a combination of all of
these.

If no baseline data exists, agencies need to select indicators that will
establish the basis for comparing future performance. For example, the
Federal Aviation Administration (FAA) did not measure the productivity
of its oceanic air traffic controllers, or the impact of its current
operations on the airline industry as a course of normal business. The
FAA recognized the limitations of the current system and its impact on
the airline industry, but did not have "hard" numbers for a baseline. As
a result, the FAA chose to establish a baseline by conducting a job task
analysis and workload analysis. Through these analyses, the FAA defined
a typical scenario and measured the average capability of its air
traffic controllers and the number of planes they control in oceanic air
space. The FAA will use this baseline to determine the cost
effectiveness of its Advanced Oceanic Automation System after
installation and operation.

The DFAS will establish an IT infrastructure to support its new
organizational configuration. To compare the cost effectiveness of its
approach, DFAS benchmarked private industry firms to determine the cost
of administering similarly sized network configurations. As noted
earlier, DFAS found that the typical number of nodes per administrator
in industry is 150 to 1. This ratio serves as the DFAS baseline.

Establishing baselines using performance measures linked to goals and
objectives also helps agencies better define their information needs.
This will improve the formulation and selection of IT projects that will
be linked to agency goals and objectives.

Step 4: Select Information Technology Projects with the Greatest Value

For IT performance measures to assess effectively the contribution of IT
investments to mission objectives, the IT investments need to be linked
closely to business priorities. In a shrinking budget environment, it is
essential the IT projects that are selected produce the greatest value
with the resources available. Value consists of the contribution of IT
to business performance in addition to discrete benefits. Discrete
benefits include cost reduction, cost avoidance, productivity
improvement, and increased capacity that results when a particular
program area employs technology. This step will describe how to use
value as the basis to select IT investments.

Traditionally, Federal and private organizations customarily conduct a
cost-benefit analysis to evaluate and select large IT projects. In this
analysis, organizations typically identify the non-recurring and
recurring costs to acquire, develop, and maintain the technology over
its life; and the benefits likely to occur from use of the technology.
The types of benefits include: tangible (direct cost savings or capacity
increases); quasi-tangible, which focus on improving the efficiency of
an organization; and intangible benefits that focus on improving the
effectiveness of the organization.

In the Federal government, the net present value (NPV) method is used
commonly to evaluate projects. The NPV method accounts for the time
value of money to determine the present value of the costs and benefits
through the use of a discount rate. The benefits to cost ratio is an NPV
technique for comparing the present value of benefits to the present
value of costs. Agencies select the projects with the highest
benefit/cost ratio with some consideration of the intangibles benefits
and risks associated with each project. Industry typically compares
projects on their return on investment (ROI). ROI is the ratio of annual
net income provided by the project to the internal investment costs of
the project.

The NPV and ROI methods have limitations. They do not adequately factor
the intangible benefits of business value. Examples of business value
include the contributions of IT projects to long-term strategy or
information to better manage core business processes. These methods also
assume the availability of funds for all cost-justified projects. Yet,
funds are always limited. Furthermore, IT organizations typically
conduct the cost-benefit analyses and select the IT projects with
limited input from the user community.

Determining the Value of IT Projects

To determine the value of IT investments according to business
priorities, agencies can use the techniques of Information Economics[^9]
to go beyond traditional NPV and ROI analysis methods. Information
Economics is based upon the concepts of value and two-domain analysis.
Value is the contribution of IT to enable the success of the business
unit. Two-domain analysis segments organizations into business and
technology domains to assess the impact of IT investments on each
domain.[^10] The business domain uses IT. The technology domain provides
IT services to the business domain.

Decisions to invest in technology solely for technology reasons rarely
support improved business performance. For example, standardizing
workstation configurations reduces the cost of maintenance and training.
Although the investment appears necessary and prudent, the action has
little direct bearing on the business.[^11] Therefore, to maximize the
performance of IT investments, favor those projects that provide the
greatest impact on the performance of the business domain.

Information Economics provides the means to analyze and select IT
investments that contribute to organizational performance based upon
business value and risk to the organization. This is done using the
following business and technology domain factors. Agencies select the
domain factors that reflect their business priorities. (See Supplement 1
for more information on Information Economics and a description of the
domain factors.)

The business domain factors include the following:

-   Return on Investment (ROI)

-   Strategic Match (SM)

-   Competitive Advantage (CA)

-   Management Information Support (MI)

-   Legislative Implementation (LI)

-   Organizational Risk (OR)

The technology domain factors include:

-   Strategic IT Architecture Alignment (SA)

-   Definitional Uncertainty Risk (DU)

-   Technical Uncertainty Risk (TU)

-   Information System Infrastructure Risk (IR)

Organizations customarily evaluate the cost of IT to the technology
domain and the benefits of IT to the business domain. Information
Economics examines the cost and value that IT contributes to the
business and technology domains separately. This provides a more
accurate assessment of the impact of an IT investment to the
organization.

Using Information Economics tools means the business domain determines
the relative importance of the domain factors. Agencies can obtain
consensus within the business domain by establishing an Investment
Review Board with the major stakeholders identified in Step 1 as the
members. The IRB determines the importance of the factors and assigns
weights between one and ten to each factor.

The process of assigning weights to the factors helps agencies establish
and communicate business priorities. Just establishing the weights
provides enormous value. This is especially true when a turnover in
senior management or organizational restructuring occurs. Assigning
weights communicates the shared beliefs held by senior management.
Because agencies implement IT according to their priorities, the weights
will vary by agency.

To evaluate each project, the IRB assigns a score of one to five for
each domain factor according to specific criteria.[^12] The sum of the
value factor scores multiplied by the factor weights constitutes the
project value. The sum of the risk factor scores multiplied by the
factor weights constitutes the project risks. The factor weights and
scores can be displayed in an Information Economics Scorecard. An
example is shown in Figure 9.

The Information Economics Scorecard allows agencies to assess risks by
making them visible through the organizational risk, definitional
uncertainty, technical uncertainty and IS infrastructure risk factors.
The total risk score for a project may be acceptable to one organization
but not to another. Agencies determine whether they can manage or lower
the risks. They can lower their IT project risks by improving
organizational readiness, reducing the project scope, or segmenting the
project into more definitive modules.

Figure 9 shows the Information Economics Scorecard for a proposed
payroll system. In this hypothetical example, the organization placed
the highest weight, 10, on ROI; and 5, or half the importance of ROI, on
strategic match. The organization rated the proposed payroll system high
(4) on the ROI factor because of high labor savings.

<table style="width:100%;">
<colgroup>
<col style="width: 11%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 6%" />
<col style="width: 0%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 6%" />
<col style="width: 0%" />
<col style="width: 8%" />
<col style="width: 9%" />
</colgroup>
<tbody>
<tr class="odd">
<td></td>
<td colspan="6"><strong><span class="smallcaps">Business
Domain</span></strong></td>
<td colspan="5"><strong><span class="smallcaps">Technology
Domain</span></strong></td>
<td colspan="3"><p><strong><span
class="smallcaps">Project</span></strong></p>
<p><strong><span class="smallcaps">Score</span></strong></p></td>
</tr>
<tr class="even">
<td><strong>Factor</strong></td>
<td>ROI</td>
<td>SM</td>
<td>CA</td>
<td>MI</td>
<td>LI</td>
<td colspan="2">OR</td>
<td>SA</td>
<td>DU</td>
<td>TU</td>
<td colspan="2">IR</td>
<td>Value</td>
<td>Risk</td>
</tr>
<tr class="odd">
<td>Score</td>
<td>4</td>
<td>2</td>
<td>0</td>
<td>4</td>
<td>0</td>
<td colspan="2">3</td>
<td>4</td>
<td>2</td>
<td>1</td>
<td colspan="2">3</td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>Weight</td>
<td>10</td>
<td>5</td>
<td>5</td>
<td>2</td>
<td>1</td>
<td colspan="2">5</td>
<td>2</td>
<td>2</td>
<td>2</td>
<td colspan="2">2</td>
<td>66</td>
<td>27</td>
</tr>
</tbody>
</table>

Figure 9 --- Information Economics Scorecard for a New Payroll
System[^13]

The payroll system received a low score (2) on strategic match because,
although the system allowed the organization to manage its resources
more efficiently, it did not contribute significantly to the
organizational goals. The payroll system received a 3 on the
organizational risk factor because the personnel department did not make
adequate plans to integrate the new payroll system into its operations.

For multiple projects, the project scores by factor can be displayed in
a table as shown in Figure 10. In this example, the maximum possible
value score is 100. The maximum possible risk score is 35. For a project
to be selected, its value score must exceed a minimum acceptable score,
for example 70, and its risks be acceptable and manageable.
Organizations establish their minimum acceptable scores through past
experience with IT projects and repeated use of the Information
Economics Scorecard. Likewise, organizations determine the acceptable
level of risks through analysis of past experience and expert
assessment.

After the scores for each project are known, the IRB can rank the
projects according to their total value scores and select the projects
that provide the most value at an acceptable and manageable level of
risk. The IRB then selects the projects with the topmost value given the
funds available. Selecting projects using investment selection criteria
such as the Information Economics to allocate limited resources is known
as capital planning.

  --------------------------- --------- ----- -------- ---- -------- ---- -------- ---- -------- ---- -------- ---- -------- ----- -------- -------- ----- -------- --- ------------- ------- ---- ------------
  **[Project]{.smallcaps}**   **ROI**         **SM**        **CA**        **MI**        **LI**        **OR**        **SA**         **DU**   **TU**         **IR**       ***Value***                ***Risk***

  Automated Billing Sys.      25              8             6             8             5             0             3              6        4              2            55                         12

  Driver Pay System           15              8             0             4             0             0             3              2        2              0            30                         4

  Driver Scheduling Phase 2   45              10            4             6             4             0             12             6        2              0            81                         12

  Bar Code Project            50              10            4             10            4             2             15             0        8              6            83                         16

  Capacity Project                      50             10            6             8             4             1             12    0                 2              8                 90      11   
  --------------------------- --------- ----- -------- ---- -------- ---- -------- ---- -------- ---- -------- ---- -------- ----- -------- -------- ----- -------- --- ------------- ------- ---- ------------

Figure 10 ---Project Scores for a Shipping Company Using Information
Economics[^14]

Step 5: Collect Data

If organizations address properly the data needed for measurement during
the development and selection of their performance measures, the actual
collection of data becomes routine. Therefore, the time to address data
collection is during development of the performance measures.

The data that agencies need to collect depends on the indicators that
are chosen. To some degree, the choice of indicators depends on the data
available and the baseline. For each indicator, the IT organization
needs to decide up-front the "what, when, how and by whom" of collecting
the data. When determining which data are appropriate to assess results,
it is important to obtain consensus among customers and stakeholders.

When selecting data and establishing baselines, use the following
criteria:[^15]

-   **Availability:** Are the data currently available? If not, can data
    be collected? Are there better indicators for which data are
    currently unavailable?

-   **Accuracy:** Are the data sufficiently reliable? Are there biases,
    exaggerations? Are the data verifiable and auditable? A major
    challenge in outcome and impact measurement is determining the cause
    and effect between the investment and the results. The connection
    may be unclear. When deciding upon the data to collect, accuracy of
    the data is sufficient and consequently more valuable and cost
    effective than its preciseness. In an example of the number of
    people who attended a training class, 25 students attended reflects
    accuracy, 12 men and 13 women denotes precision.

-   **Timeliness:** Are the data timely enough to evaluate performance?
    How frequently do the data need to be collected and reported? (e.g.,
    monthly, quarterly, semi-annually, etc.). How current are the data?

-   **Cost of Data Collection:** What is the cost of collecting the
    data? Are there sufficient resources, for example, personnel and
    funding, available for data collection? Is data collection cost
    effective, that is, do the benefits exceed the costs anticipated?

Step 6: Analyze the Results

Results, particularly outcomes, rarely provide meaningful information by
themselves. Results must be examined in context of the objectives,
environment and external factors. Therefore after collecting the
results, organizations conduct measurement reviews to determine how well
the indicators worked and how the results contribute to objectives. The
purpose of this step is to improve the measures for the next measurement
cycle; to look for ways to improve the performance and effectiveness of
IT within agencies; and to make meaningful conclusions from the results.

The measurement reviews examine the effectiveness of the chosen
indicators, baseline and data chosen. The team or organization
responsible for the results conducts the reviews and includes key
stakeholders and customers as appropriate, and the team that created the
indicators, if different. The reviews examine the results by answering
the following questions. The question used depends on the stage of the
project.

Project outputs are easier to evaluate than outcomes and impacts. Output
measures indicate progress and reflect the level of efficiency. They
must be tracked until project completion. Outcomes occur after
completion of the project, or its segments or phases. Agencies use
intermediate outcomes, if they can be identified, to assess progress
towards outcomes of their IT projects before completion. Agencies also
use some output measures to assess progress toward project outcomes.

Customers and stakeholders are the ultimate judges of the outcomes and
impact of an IT investment. Agencies can use business logic and common
sense to assess whether the output of the IT project contributes to the
effectiveness of their programs and mission accomplishments.

After completing measurement reviews, the team or responsible component
prepares reports and briefings that summarize and track the results over
time. Simple, eye-catching graphics summarize performance data better
than narrative alone. Examples of graphics include process charts,
thermometer diagrams or a dashboard with dials to represent the Balanced
Scorecard measures. To facilitate comprehension of the results, use the
same report format across projects. Preparing reports according to the
frequency established during Step 3 enhances their value.

Not all reports will be understood automatically. Oregon's Office of the
Chief Information Officer (CIO) surveyed the recipients of its initial
performance reports and found that less than 40 percent either
understood or liked some of the reports. Agencies can prepare more
effective reports by talking with their customers and stakeholders to
learn what is useful to them, and how to better present the information.

If the IT project goals were not met, agencies must identify and explain
the factors that inhibited performance. The inhibitors possible are
numerous. They could include the design of agency processes,
interruptions of funding, site preparation delays, acts of God, change
of strategy or requirements, or loss of key personnel.

It becomes easier with practice and experience to develop and use
performance measures. Agencies refine their output, outcome and impact
measures to adequately track the results intended. Measures change over
time as the priorities of the organization change. At the Information
Resources Management Conference in September 1996, OMB stated it
recognized the difficulty of obtaining outcome measures. Initially, OMB
said it would accept output measures.

Figure 11 shows a sample performance report. There is no prescribed
format for reporting results. In the future, OMB is likely to specify a
format in terms of exhibits to agencies' budget submissions. Reports
should be consistent in format to ease comparison of projects.

  ------------------------------ --------------- --------------------------- ------------ -------------- ------------ ------------ --------------- ------------
  **[Performance                                                                                                                                   
  Report]{.smallcaps}**                                                                                                                            

  **[Objective:]{.smallcaps}**                   **[Performance                                                                                    
                                                 Indicators]{.smallcaps}**                                                                         

  **Type of Measure**            **Performance   **Near-Term**                            **Mid-Term**                             **Long-Term**   
                                 Measures**                                                                                                        

                                                 **Target**                  **Actual**                  **Target**   **Actual**   **Target**      **Actual**

  **Input**                                                                                                                                        

  **Output**                                                                                                                                       

  **Outcome**                                                                                                                                      

  **Impact**                                                                                                                                       

  **Mitigating Factors**                                                                                                                           
  ------------------------------ --------------- --------------------------- ------------ -------------- ------------ ------------ --------------- ------------

Figure 11 - Sample Performance Report[^16]

Step 7: Integrate into Management Processes

After agencies collect and analyze the measurement data, the payback
from performance measurement comes from using the data to improve
performance. Many organizations report that if the results are not used,
employees will not take performance measurement seriously nor will they
make the effort required to apply measurement effectively.

This step describes ways agencies can integrate the results into their
management processes and begin to create a performance culture. An
organization no longer can take its survival for granted. To remain
viable, agencies (as well as organizations and functions within an
agency ) must demonstrate their value through results. A performance
culture encourages and emphasizes activities that contribute to
organizational goals and objectives, and continually assesses its
activities to improve performance.

Performance measurement data can be integrated into a number of
management processes within the business and technology domains to
improve decision making. The types of management processes and the
management level involved depend on the scope of the project and the
measures employed. Good performance measures indicate whether the
activity was done right (efficiency) and whether the activity was the
right thing to do (effectiveness). Output measures assess efficiency.
Outcome measures assess effectiveness.

A good Balanced Scorecard provides managers a comprehensive view of the
business unit or organization. Using a mix of measures (output,
intermediate outcome, and outcome) with a combination of near-,
intermediate and long-term goals for a project, agencies can integrate
the results into the planning, budgeting, and operation processes in the
business domain. In the technology domain, agencies integrate the
results into their planning, budgeting, design, development,
implementation and operation processes.

For example, an IT modernization project integrates an organization's
nation-wide departmental systems and provides more capability to
end-users. The purpose of the new system is to increase user
productivity by increasing the number of cases processed by five percent
annually over a three year period. The developers from the IT shop and
representatives from user organizations participated in the design and
development of the system. The system will be implemented incrementally.

For this IT modernization project, the following questions enumerate
some of the possible ways to integrate performance data or results into
management processes:

Frequently, imprecise or ambiguous strategic goals and objectives lead
to ineffective performance measures. Senior managers remedy this by
setting specific goals and objectives that articulate and emphasize
mission themes. Senior management can encourage higher performance by
setting "stretch goals." Stretch goals allow for variance of
performance. For example, rewarding project managers for obtaining 80
percent of the goals.

Performance measurement allows managers and workers to focus on what's
important---performance. This enhanced focus provides managers and
workers with better understanding of the organization's operations.
Through the use of performance measures, managers from both public and
private organizations find that inefficient processes are the primary
inhibitors to their organization's performance. Consequently,
performance measurement often leads to business redesign to achieve
higher levels of performance.

The Information Technology Management Reform Act requires agencies to
rethink and restructure the way they do business before making costly
investments in information technology. Yet a majority of reengineering
efforts fail. Failures occur when business units do not take ownership
of reengineering efforts, senior management is not committed nor
involved, or organizations do not adequately prepare for reengineering.
To help agencies succeed at reengineering, GSA developed a Business
Process Reengineering (BPR) Readiness Assessment.[^17]

The Assessment contains a series of 73 questions designed to help
agencies determine their degree of readiness to accomplish worthwhile
BPR efforts. Agencies' responses to these questions help them identify
the factors necessary for conducting successful BPR programs early in
their development. Using this assessment tool to determine readiness
helps agencies evaluate their potential for success before committing to
major investments in time, money, personnel and equipment.

The relationship between IT investments and agency accomplishments may
not be immediately clear or self evident. If linkage of IT projects to
agency accomplishments is not possible, managers use business logic and
common sense to assess the intangible value of the IT project to their
business priorities and objectives. This holds especially true for
investments in communication and software infrastructures that support
multiple agency programs where benefits can take years to be realized.

All results will not have the same importance or weight. Agencies
benefit from studying the results to uncover cause-and-effect
relationships and the nature of the results. What do the results
actually mean? Are we being effective? Cost may be the main determinant
if resources are extremely scarce. However, focusing strictly on cost is
risky. Some organizations lose customer loyalty by sacrificing quality
for cost containment. Customer satisfaction may hold more importance
than cost.

Step 8: Communicate the Results

It is not enough to measure, analyze and incorporate the results into
management processes. It is vital also to communicate the results inside
and outside the agency. Communicating the results internally improves
coordination and better utilization of limited resources. Internal
communication also builds confidence and support of IT projects if
results are favorable. Communicating externally strengthens partnerships
with customers; and increases the likelihood for continued support and
funding from OMB and Congress. Communicating with customers and
stakeholders is important throughout the measurement process.

The type of information to communicate will depend on the intended
audience. Table 2 in Supplement 1 identifies the interests by management
level. In general, those directly involved in a project are interested
in outputs and outcomes. Those farther removed from the project are
interested in the outcomes. Did the IT project improve mission
performance? How effective was the IT investment? Because of the high
visibility of large IT projects, management and oversight bodies want
not only to know the effect of the project, they also want to know
whether projects are on schedule and within budget.

As with performance reports, results convey more meaning when displayed
graphically, using bar charts, histograms, thermometer or dashboard
diagrams, for example. The graphics will probably need to be explained.
Explanations need to be concise and consistent with the graphics chosen
and appropriate to the level and interests of the intended audience.
Feedback from those who know the intended audience but are not directly
involved with the project is useful to assure the results tell a story.

For its customers, Oregon's CIO office tracked completed IT projects.
The CIO office produced tabular reports that compared projects by:

-   Percentage of billable versus total hours

-   Percentage of approved projects delivered within budget

-   Percentage of projects that: delivered successfully, implemented
    best solution, and improved work-life quality

In their annual budget submissions to OMB and Congress, agencies have to
report progress toward agency or project goals. OMB and Congress focus
on the cost and the outcome of projects (investments). Results,
particularly outcomes and impacts, may take years to be realized.
Because outcomes do not occur until the end of the project, many
agencies work with OMB and Congress to plan and identify intermediate
outcomes to demonstrate progress. Agencies explain how project outputs
and intermediate outcomes lead to project outcomes.

The explanations include a summary of the resources employed, the
project goals and the results obtained. The explanations include the
reasons why there are, if any, significant variances between the goals
and the results. Identify and explain the impact of external influences
(e.g., funding or contract interruptions, acts of Nature, changing
priorities, technology changes) that inhibited performance. The impact
must be relevant. The explanation includes the actions that were taken
or will be taken to improve performance or to put the project back on
track.

The main focus on IT projects is whether they created efficiencies and
effectiveness for the organization. Proving this by explaining the
results may not be sufficient. Results may have more meaning if compared
to leading organizations. The comparisons could be with Federal, state
or private organizations. Educating stakeholders and customers is
another benefit of benchmarking. However, the comparisons must be valid
and relevant to the project.

For coordination and alignment of activities, some agencies publish the
results on their Intranets. For both internal and external customers and
stakeholders, it is best to eliminate surprises. Communication of
results is done through budget submittals. Agencies benefit if they meet
with budget examiners and congressional staffers to explain their
projects and results. Be aware and sensitive to their peak work periods.
Meet during "off-crunch" periods to explain projects and results in
progress. Being in the dark or an unknown breeds suspicion that works
against an agency's efforts.

Be aware of customers' budget cycles. The results may help them in their
planning and budget processes. Expect to be questioned on the results,
particularly outcomes. On highly visible or mission-critical projects,
it may be wise and worth the expense to have the results verified from
recognized outside experts.

Things to Consider

For performance measurement to be useful to their organizations,
agencies need to address the issues of organizational maturity,
accountability, and resources.

Organizational Maturity

Developing and using performance measures to improve performance
requires a change in mindset and culture. Agencies lay the foundation
for these changes by encouraging and fostering the use of performance
measures. This only happens when senior managers are supportive and
involved in the process itself.

Initially, performance measures and the linkage of IT projects to
organizational outcomes may be hard to conceptualize and recognize.
Practitioners require time and experience before they are able to
develop and use performance measures effectively. Agencies reduce their
learning curve by creating performance measurement guides tailored to
their mission.

It takes time for agencies to institutionalize performance measurement.
Agencies can institutionalize performance measurement sooner if they use
a framework and methodology such as the Balanced Scorecard. The
originators of the Balanced Scorecard report that an organization's
first Balanced Scorecard can be created over a 16-week period.

Accountability

The objective of performance measurement is to improve the performance
of the organization, not conduct individual performance appraisals. To
facilitate the acceptance of performance measurement, agencies encourage
and reward managers and workers who use measurement to improve
verifiable performance.

Many practitioners believe managers should be made accountable for
results as part of their personnel appraisal. Others believe the goal of
performance measurement is to improve performance, not evaluate
individuals, and that as soon as the latter occurs, cooperation is lost.
Sucessful implementation requires agencies to strike a balance between
these two perspectives.

Managers and employees alike have a legitimate concern when held
accountable for results over which they do not have complete control. In
reality, no one individual has total control of the results, but people
can influence the results. Consequently, evaluations must include a
review of the factors that influence results, not just the level of
results. Also, agencies should validate the performance data before
evaluating individuals.

Resources

Performance measurement requires an investment in resources. The INS
believes that agencies should dedicate resources up-front to properly
set up an agency performance-based management structure. Reports from
industry and state governments confirm that more resources are used
initially to develop a knowledge base and instill performance-based
management methods in their organizations. As organizations learn how to
develop and use performance measures, less resources are necessary.

The amount of resources and time to develop measures will depend on the
scope of the project, partnership of the business and technical groups,
data available, knowledge and skill of the developers, and how proactive
management becomes. The resources needed to develop and use performance
measures will vary from project to project as they did for the projects
surveyed for this report.

Conclusion

The eight steps discussed in this paper provide a systematic process for
developing IT performance measures linked to agency goals and
objectives. Agencies establish this linkage during the planning stage of
the IT project. The Balanced Scorecard provides a framework for agencies
to translate business strategies into action. A good Balanced Scorecard
provides a comprehensive view of an organization. It also helps IT
organizations align their projects and activities with business and IT
strategies.

A mix of short-, intermediate and long-term measures are needed to gauge
the efficiency and effectiveness of IT projects to agency goals and
objectives. Results need to be analyzed to learn how to improve
performance and the measures themselves. Measurement is an iterative
process. Measures improve with time and experience. For measurement to
be taken seriously, the results must be used by management. Throughout
the measurement process, communication with stakeholders and customers
is vital for performance measures to be effective. Communicating the
results internally improves coordination and support. Communicating the
results with OMB and Congress garners support and continued funding of
initiatives.

Agencies developing performance measures may benefit from the lessons
learned on the High Performance Computer Modernization Program. This
program will provide state-of-the-art high performance computing and
high speed networks for scientists and technicians in the Department of
Defense (DoD). The objective of this program is to improve the quality
of research through more realistic models and quicker solutions.

DoD was surprised at how difficult it is to quantify the measures. As
the process evolved, they realized they had underestimated the time
required and the resources needed. Later, DoD was pleasantly surprised
at the value of doing performance measures. While requiring substantial
effort, DoD considered the time and resources spent a worthwhile
investment.

**[Performance-Based]{.smallcaps}**

**[Management]{.smallcaps}**

**[Supplements]{.smallcaps}**

Supplement 1 : Developing

> Performance Measures

This supplement provides additional information to Step 2 for developing
performance measures for IT projects.

Laws That Require Performance Measures

Currently, Federal laws require Federal agencies to measure the
performance of fixed asset acquisitions over \$20 million (includes IT
services), IT investments, and agency programs. The scope of the
performance measures depends on the project's complexity and size, the
life-cycle phase and purpose of the IT investment. Legislative mandates
require Federal agencies to measure the performance of their IT projects
from acquisition to implementation. For example:

-   For all acquisitions over \$20 million, OMB requires Executive
    Agencies to develop and submit performance measures (OMB Bulletin
    95-03). Measures must include cost, schedule and performance. The
    Federal Acquisition Streamlining Act (FASA) requires agencies to
    meet, on average, 90 percent of their acquisition cost and schedule
    goals without a reduction in performance.

-   The Cinger-Cohen Act (a.k.a. Information Technology Management
    Reform Act) applies to acquisition and implementation. The
    Clinger-Cohen Act requires Executive Agencies to measure the
    performance of their IT investments and link their IT investments to
    agency accomplishments. The degree to which this is doable depends
    on the scope of the investment. An investment closely tied to an
    agency line of business or a program is easier to link to agency
    accomplishments. An investment in infrastructure is more difficult
    to link to accomplishments.

-   The Clinger-Cohen Act also requires Executive Agencies to explore
    business process reengineering before making a significant
    investment in IT. The degree of business process transformation has
    a direct influence on the range of potential benefits and agency
    accomplishments. For example, the benefits of a successful business
    process reengineering effort will be dramatically greater than an
    investment in infrastructure or end-user computing without
    reengineering.

-   The Government Performance and Results Act (GPRA) requires agencies
    to develop and submit strategic plans and performance plans for
    their major programs. The GPRA applies to an IT investment if the
    investment is a major agency program or provides a significant
    contribution to the agency's programs. If an IT investment is
    closely linked to an agency program, it may not be possible to
    segregate the contributions of the IT investment.

Table 2 illustrates the interests and types of measures by management
level and the corresponding legislative mandate. The intent of all three
laws is the same: improve the performance of agencies by requiring them
to define the goals of their acquisition and programs, linking their
investments to results, and communicating the results to OMB and
Congress. See Appendix D for additional information on FASA, GPRA and
ITMRA.

+--------------+-------------------------+--------------+--------------+
| **[M         | **[I                    | **[Type of   | **[Le        |
| anagement]{. | nterests]{.smallcaps}** | Measures]{.  | gislative]{. |
| smallcaps}** |                         | smallcaps}** | smallcaps}** |
|              |                         |              |              |
| **[Level]{.  |                         |              | *            |
| smallcaps}** |                         |              | *[Mandate]{. |
|              |                         |              | smallcaps}** |
+--------------+-------------------------+--------------+--------------+
| -   Agency   | Strategic,              | Impacts      | GPRA & ITMRA |
|              | Organizational Impact,  |              |              |
|              | Resources Utilization,  |              |              |
|              | Service to the Public   |              |              |
+--------------+-------------------------+--------------+--------------+
| -   Programs | Effectiveness, Quality, | Outcomes     | GPRA & ITMRA |
|              | Delivery, Cycle Time,   |              |              |
|              | Responsiveness,         |              |              |
|              | Customer Satisfaction   |              |              |
+--------------+-------------------------+--------------+--------------+
| -            | System availability,    | In           | ITMRA        |
|   Operations | systems capability      | puts/Outputs |              |
|              | and/or capacity         |              |              |
|              | Technical               |              |              |
+--------------+-------------------------+--------------+--------------+
| -            | Cost, Schedule,         | In           | FASA & ITMRA |
| Acquisitions |                         | puts/Outputs |              |
|              | Performance             |              |              |
|              | Requirements            |              |              |
+--------------+-------------------------+--------------+--------------+

Table 2[^18]---Management Level Measures/Legislative Mandate

Figure 3 in Step 1 shows the measures constructed by the Immigration and
Naturalization Service for the Integrated Computer Aided Detection
(ICAD) system. INS developed three levels of measures for ICAD:
strategic, programmatic and tactical. The INS measures correspond to
agency, programs, operations and acquisitions management level shown in
Table 2. INS also developed measures for the Information Technology
Partnership (ITP) contract to track the contractor's performance by task
order (See Appendix C).

Getting Started

Organizations typically establish a team to develop the performance
measures for large IT projects. The most effective teams include the IT
project manager or a member(s) of the IT management staff. Successful
teams have a strong customer focus and consistently solicit input from
customers and stakeholders who judge the performance of IT projects.

The FAA used an Integrated Project Team (IPT) comprised of a program
manager, project leader and engineers to develop performance measures
for the Oceanic System Development and Support Contract. Although users
and customers were not involved in the actual development of the
measures, the IPT used their input and feedback.

On task orders for the Information Technology Partner contract, the INS
uses service-level agreements between the customer (program offices),
the Information Resources Management task manager and the contractor.
The service level agreements document project goals and objectives,
establish task costs and schedules, and develop performance measures for
contract tasks and program-level measures for high-impact,
mission-critical tasks.

When formulating strategies and developing measures, it is important to
consider the following six benchmarks. The author of these benchmarks
believes organizations will be judged by them in the next decade:[^19]

-   Quality

-   Productivity

-   Variety

-   Customization

-   Convenience

-   Timeliness

**Characteristics of the Balanced Scorecard**

The Balanced Scorecard has the following characteristics:

-   Translates business objectives into performance measures

-   Serves as a portfolio of measures that are interrelated

-   Provides a comprehensive view of the entire IT function

-   A project may have measures in more than one perspective

-   Allows operational measures to be used also

-   Assesses multiple projects (this is important when projects consist
    of modules)

-   Facilitates integration and alignment of projects to common
    objectives

Sample Measures for the Balanced Scorecard

As agencies formulate performance measures, they may want to consider
the following measures for their Balanced Scorecards:

**Financial Perspective---**Focuses on the costs of IT

-   Planned versus actual contribution of IT project to the business
    domain

-   Return on Investment or Internal Rate of Return of each project

-   Percentage of projects completed on-time and within budget

-   Cost reduction or cost avoidance within program or line of business

-   Ratio of expenses of legacy systems to total IT expenses

> **Internal Business Perspective**---Focuses on internal processes that
> deliver products and services

-   Staffing --- Planned versus actual level

-   Alignment of the IT project with the strategic IT plan and
    architecture

-   Cost of services or operations --- Planned versus actual

-   Service --- Ratio of the number of requests closed to number of
    requests received

-   Acquisitions:

```{=html}
<!-- -->
```
-   Schedule --- Planned versus actual contract award date or delivery
    dates

-   Percentage of task orders on-time, within budget

-   Cost ---Variance between budgeted cost of work scheduled, budgeted
    cost of work performed and actual cost of work performed

> **Customer Perspective---**Focuses on how the customer experiences the
> project or system

-   Level of understanding of requirements

-   Level of satisfaction with the computer systems, networks, training
    and support

-   Productivity enhancement

-   Convenience---Access to the right information and the right time

-   Responsiveness ---Ratio of the number of service or assistance
    requests completed within acceptable time to the total number of
    requests

> **Learning and Innovation Perspective---**Measures the degree the
> project includes innovative technology and contributions to worker
> development

-   Degree to which new technologies are introduced to maintain
    competitiveness or currency

-   Percentage of change in costs of new procedures to old procedures

-   Percentage of change in cycle times of new procedures to old
    procedures

-   Percentage of users and IT personnel trained in use of technology

-   Percentage of employee satisfaction

Figure 12 lists some sample quality measures:

Supplement 2: Selecting IT Projects with

> the Greatest Value

This supplement provides additional information supporting Step
4---identifying the cost, benefits and value of IT projects. After
agencies define the project mission and critical success factors in Step
1, develop performance measures in Step 2, and a baseline in Step 3,
they identify and estimate the IT project's value and risks.

Table 1 lists the typical costs and benefits and possible opportunities
associated with IT investments. In this table, the benefits and
opportunities may apply to any of the costs. Agencies can use these to
identify the costs and benefits of their IT projects.

+-----------------------+----------------------------------------------+
| **                    | **[Benefits and Opportunities]{.smallcaps}** |
| [Costs]{.smallcaps}** |                                              |
+-----------------------+----------------------------------------------+
| Non-recurring:        | -   Higher productivity (cost per hour),     |
| (one-time)            |     increased capacity                       |
|                       |                                              |
| -   Hardware          | -   Reduced cost of rework, scrap, failure   |
|                       |                                              |
| -   Software          | -   Reduced cost of IT operations and        |
|                       |     support costs                            |
| -   Network hardware  |                                              |
|     and               | -   Reduced cost of business operations      |
|                       |     (primary or support functions)           |
|     software          |                                              |
|                       | -   Reduced errors                           |
| -   Software & data   |                                              |
|     conversion        | -   Improved image                           |
|                       |                                              |
| -   Site-preparation  | -   Reduced inventory costs                  |
|                       |                                              |
| -   Installation      | -   Reduced material handling costs          |
|                       |                                              |
| -   Initial loss of   | -   Reduced energy costs                     |
|     productivity      |                                              |
|                       | -   Better resource utilization              |
| Recurring: (on-going) |                                              |
|                       | -   Better public service                    |
| -   Hardware          |                                              |
|     Maintenance       | -   More timely information                  |
|                       |                                              |
| -   Software          | -   Improved organizational planning         |
|     Maintenance       |                                              |
|                       | -   Increased organizational flexibility     |
| -   Systems           |                                              |
|     Management        | -   Availability of new, better or more      |
|                       |     information                              |
| -   Data              |                                              |
|     Administration    | -   Ability to investigate an increased      |
|                       |     number of alternatives                   |
| -   Software          |                                              |
|     Development       | -   Faster decision-making                   |
|                       |                                              |
| -   Communications    | -   Promotion of organizational learning and |
|                       |     understanding                            |
| -   Facilities (rent) |                                              |
|                       | -   Better network and system                |
| -   Power and cooling |     interoperability                         |
|                       |                                              |
| -   Training          | -   Better information connectivity          |
|                       |                                              |
|                       | -   Reduced training costs due to personnel  |
|                       |     reassignments                            |
|                       |                                              |
|                       | -   Improved IT response time to user        |
|                       |     requests                                 |
|                       |                                              |
|                       | -   Expandability of standards-based systems |
|                       |                                              |
|                       | -   Greater access to corporate information  |
|                       |                                              |
|                       | -   Legislative and regulatory compliance    |
+-----------------------+----------------------------------------------+

Table 1 - Examples of IT Costs, Benefits and Opportunities[^20]

Agencies can estimate the acquisition and support costs for the useful
life of IT projects through market research and past experiences. As
shown in Table 1, the life-cycle costs consist of non-recurring, or
one-time costs, and recurring, or on-going costs. In addition to these
direct costs, there are the costs associated with the impact of IT
investments on an organization, e.g., initial loss of productivity. The
impact costs can be non-recurring or recurring, for example, the cost of
training the systems development staff on a new client-server system
(non-recurring), plus the costs of integration and management of both
the new system and the existing system (recurring). Although not part of
the hardware and software acquisition costs, these realities may affect
the performance of the organization for some time.

Using Table 1, agencies can identify the benefits generated from their
IT investments. By transforming their business operations and selecting
IT projects that will complement and facilitate transformations,
agencies can increase the benefits of their IT investments. Figure
13[^21] illustrates how potential benefits vary with the degree of
business transformation.

![](media/image6.emf){width="6.5in" height="4.875in"}*Figure 13 ---
Benefits and Degree of Business Transformation*

Localized exploitation and internal integration represent the
traditional approach of automating existing tasks and departmental IT
resources. At most, they include only incremental changes to the
business. Business process redesign uses IT to change dramatically how
an organization performs work. The Clinger-Cohen Act requires agencies
to reengineer their processes before making a significant investment in
IT. Business network redesign refers to linking an organization's
networks externally to customers' and suppliers' information systems.
For government agencies, this would equate to interconnecting their
information systems to other agencies, contractors and the public.

Business scope redefinition refers to expanding the view of agencies to
reorganize the Federal government around specific policies or functions.
The Federal government's approach to job training provides an example of
this. Currently, approximately 15 programs provide job training benefits
for eligible citizens. Conceivably, by coordinating and integrating
those programs, the government could provide more effective training and
reduce the costs of delivery. The Federal government often implements
policy piece-meal. Only by expanding traditional thinking to focus
outside their organizational boundaries can agencies discover and
estimate the potential benefits of business transformations.

Information Economics and Scorecard Factors

Agencies can determine the value of their IT projects by applying the
techniques of Information Economics.[^22] The techniques enhance simple
return on investment (ROI) analysis by expanding the concepts of
quantifiable costs and benefits to include value. Simple ROI and net
present value analysis compare only the discrete benefits (cost
avoidance and cost displacement directly related to an investment) to
the non-recurring and recurring costs for the life of the project.
Information Economics provides ten factors to assess the value and risks
of IT projects to the business and technology domains. Agencies may use
these factors or create their own.

The Information Economics Scorecard factors are**:** [^23]^\
^

-   **Enhanced Return on Investment (ROI)**--- Assessess the
    cost-benefit analysis plus the benefits created by the IT investment
    on other parts of the organization. Parker and Benson provide
    techniques for assessing costs and benefits of the impact of an IT
    investment on other departments of the organization. They also
    describe techniques for quantifying the benefits associated with
    increasing the value of a function. For example, electronic form
    processing provides a data entry clerk with the capability to
    process claims, a higher value function.

-   **Strategic Match (SM)** --- Assesses the degree to which the
    proposed project corresponds to established agency strategic goals.
    This factor emphasizes the close relationship between IT planning
    and corporate planning and measures the degree to which a potential
    project contributes to the strategy. Projects that are an integral
    and essential part of the corporate strategy receive a higher score
    than those that are not. Strategic Match assesses the extent to
    which an IT investment enables movement towards long-term direction.

-   **Competitive Advantage (CA)** --- Assesses the degree to which
    projects create new business opportunities, facilitate business
    transformation (e.g., interorganization collaboration through
    electronic commerce), increase the agency's competitiveness or
    improve the agency's reputation or image. Competitive Advantage
    requires placing a value on a project's contribution toward
    achieving one or more of these objectives.

-   **Management Information (MI)** --- Assesses a project's
    contribution to management's need for information about core
    activities that involve the direct realization of the mission,
    versus support activities. Measuring a project's contribution to the
    core activities of the business implies that the agency has
    identified its critical success factors. This measurement is
    obviously subjective because improved management information is
    intangible, but the benefit measurement can be improved if the
    agency first defines those core activities critical to its success,
    then selects a general strategy to address these issues.

-   **Legislative Implementation (LI)** --- Assesses the degree to which
    the project implements legislation, Executive Orders and regulatory
    requirements. For example, Federal law requires INS to process
    passengers arriving at airports from international flights within 45
    minutes. A project receives a high score if it directly implements
    legislation; a moderate score if it indirectly implements
    legislation; and no score if the project does neither.

-   **Organizational Risk (OR)** --- Assesses the degree to which an
    information systems project depends on new or untested corporate
    skill, management capabilities and experience. Although a project
    may look attractive on other dimensions and the technical skills may
    be available, unacceptable risks can exist if other required skills
    are missing. This does not include the technical organization, which
    will be measured on another dimension. Organizational risk also
    focuses on the extent to which the organization is capable of
    carrying out the changes required by the project, that is, the user
    and business requirements. For example, a high score (5) reflects
    that the business domain organization has no plan for implementing
    the proposed system; management is uncertain about responsibility;
    and processes and procedures have not been documented.

-   **Strategic IS Architecture (SA)** ---Assesses the degree to which
    the proposed project fits into the overall information systems
    direction and conforms to open-system standards. It assumes the
    existence of a long-term information systems plan --- an
    architecture or blueprint that provides the top-down structure into
    which future data and systems must fit.

-   **Definitional Uncertainty (DU)** --- A negatively-weighted factor
    that assesses the degree of specificity of the user's objectives as
    communicated to the information systems project personnel. Large and
    complex projects that entail extensive software development or
    require many years to deliver have higher risks compared to those
    projects segmented into modules with near-term objectives.

-   **Technical Uncertainty (TU)** --- Assesses a project's dependence
    on new or untried technologies. It may involve one or a combination
    of several new technical skill sets, hardware or software tools. The
    introduction of an untried technology makes a project inherently
    risky.

-   **IS Infrastructure Risk (IR)** ---Assesses the degree to which the
    entire IS organization is both required to support the project, and
    prepared to do so. It assesses the environment, involving such
    factors as data administration, communications and distributed
    systems. A project that requires the support of many functional
    areas is inherently more complex and difficult to supervise; success
    may depend on factors outside the direct control of the project
    manager.

Additional Investment Decision Factors

The Office of Management and Budget in conjunction with the General
Accounting Office published an approach that ranks IT projects using
Overall Risk and Return factors.[^24] See Appendix E for descriptions of
these factors.

OMB and GAO's approach is very similar to the Information Economics
Scorecard. Both approaches evaluate IT projects based upon their
business impact, not just their return on investment. Both approaches
evaluate risks. The advantages of the Information Economics Scorecard
are its quick comparison of investments, and its detailed methodology
for scoring IT investments.

**[Performance-Based]{.smallcaps}**

**[Management]{.smallcaps}**

**[Appendices]{.smallcaps}**

Appendix A---Key Success Factors for an

> Information Systems Performance
>
> Measurement Program[^25]

The following items are critical to the success of performance based
management system:

-   Management support and involvement from all levels in the
    organization to counteract resistance to change associated with
    introduction of new policies.

-   Appropriate measures that include defining the goals and objectives,
    identifying the measures to be used, evaluating the costs and
    benefits, and then implementing the cost-effective measures.

-   All the IS activities must be included, even those that are outside
    the IS department in the program areas.

-   Start small and measure the processes, not the people. Fewer
    measures mean less initial cost; measures can be added. Measure how
    things are done, and the result, not the people.

-   Provide feedback by using the results and reporting where
    improvements are being made.

-   Periodically review the measures to determine their usefulness and
    continued applicability.

-   Be patient. Performance measurement is a long-term process and may
    have no immediate pay-off because of the learning process involved.

Appendix B--- Agency Measures

This appendix includes the performance measurement plans from the
following six Federal agencies:

-   Defense Commissary Agency

-   Defense Finance and Accounting Service

-   Director, Defense Research and Engineering

-   Federal Aviation Administration

-   Immigration and Naturalization Service

-   Social Security Administration

These measurement plans describe how six Federal agencies will assess
the results of their major IT projects and acquisitions. These plans
present different measurement approaches and can serve as a source of
ideas for other agencies as they construct their own measurement plans.

The SSA's performance measures focus primarily on business process
improvements. There are six categories of measures: state disability
determination productivity, service delivery process improvement,
IWS/LAN local processing infrastructure capability, equipment
replacement, and pricing analysis and technology refreshment over the
life of the contract. Although the SSA did not use the Balanced
Scorecard framework to develop its measures, their measures provide a
comprehensive view of the IWS/LAN project.

The Defense Commissary Agency (DeCA) uses a combination of acquisition
and project measures to assess the Point of Sale (POS) acquisition. DeCA
intends to measure acquisition and operating costs, technical contract
requirements and customers' expectations for technical performance and
quality, and the acquisition schedule (contract award) and
implementation schedule. The Defense Accounting Finance Service
Infrastructure program, Defense High Performance Computing Program, FAA
Oceanic System Development and Support acquisition, and the INS ITP
contract also use a combination of acquisition and program measures.

**Defense Commissary Agency**

**Point of Sale Modernization Program**

**Performance Measurement Plan**

**Introduction:** Point-of-Sale Acquisition will provide the Defense
Commissary Agency (DeCA) with new point-of-sale equipment at 300
commissaries worldwide. The systems will have the capability of
equipment found in any modern grocery store. DeCA\'s current equipment
is obsolete. The new systems will provide innovative capabilities such
as acceptance of credit and debit cards, increased security, and better
scanning of Universal Product Codes. The new systems will be based upon
an open systems architecture, commercial operating systems and easily
upgraded hardware and software. The following is DeCA's performance
measurement plan:

**1**. **PROGRAM DESCRIPTION**

The Defense Commissary Agency (DeCA) -the product of a 1991
consolidation of Military Service commissary functions -operates
approximately 300 commissaries worldwide. Every commissary uses a point
of sale (POS) system to process customer purchases, capture sales and
financial data from those purchases, produce management reports, and
provide information to other DeCA business systems. DeCA\'s current POS
configuration consists of several unique systems that use proprietary
hardware platforms. Most of those systems were installed in 1985 and
1986, placing them at the end of their serviceable life. In addition,
components are malfunctioning at a higher than normal rate, which slows
the checkout process and adversely affects customer service. The
existing POS system inhibits DeCA from satisfying its primary mission of
providing "\...an efficient and effective worldwide system of
commissaries."

A new, commercially proven POS system will enable DeCA to improve its
efficiency and reduce its operating costs. The new system will also
enable DeCA to offer its customers more services including coupon
scanning, check authorization using a central data base, and electronic
payments. It should further enhance DeCA\'s ability to interface with
both internal business systems and external systems by using industry
standard communication techniques. The new POS system will use personal
computer-based hardware, open operating systems such as DOS and UNIX,
and applications developed using standard programming languages.
Furthermore, it will possess the same architecture, features, and
functions found in commercial grocery POS systems, which will enable
DeCA to offer similar services to its customers.

**2. PROGRAM OBJECTIVES**

DeCA has established objectives for its new POS system in three areas:
cost, technical/quality, and schedule. It has further broken out those
objectives into specific goals, with overall lower operating costs and
improved business practices underlying each goal. The goal categories
are listed below:

-   Cost

> · Operating costs
>
> · Acquisition costs

-   Technical/quality

· Technical contract requirements

· Customer expectations for technical performance and quality

-   Schedule

> · Contract award
>
> · Implementation schedule

**3.** **COST OBJECTIVES**

**3.1** **Operating Costs**

The primary goal of DEC's POS modernization program is to reduce POS
operating costs. Specifically, DeCA wants to reduce its POS hardware
maintenance costs. DeCA will measure all costs associated with the POS
modernization program using a Cost/schedule Status Report (C/SSR) and
**other** contract documentation.

**3-1.1** **OPERATING COSTS GOAL**

DeCA has established a goal of reducing its POS operating costs by 50
percent when compared to the projected life-cycle operating costs of the
current POS systems.

**3.1.2** **OPERATING COSTS BASELINE**

DeCA estimates that it will incur total operating costs of slightly more
than \$124 million for the period FY96 through FY03 if it retains the
current mix of POS systems. DeCA currently pays approximately \$12
million each year to maintain its POS systems and expects this amount to
increase substantially at the end of the current maintenance contract.
The projected maintenance cost for the new POS system is estimated at
approximately \$7.8 million per year, or \$62 million over the 8-year
contract. DeCA will realize reduced operating costs at every commissary
that receives the new POS system. Table I shows the **total** operating
cost savings that DeCA expects from implementing the new POS system.

**Table 1. *Operating Cost Savings***

**Status quo operating and support costs \$124,423,297**

**Estimated operating and support costs \$62,621,969**

**Net savings \$61,801,328**

**Percentage savings 49.67%**

**3.2** **Acquisition Costs**

DeCA will measure the acquisition cost of the new system by reviewing
and summarizing all contract actions, including contract modifications
against the contract baseline data.

**3.2.1** **ACQUISITION COSTS GOAL**

DeCA plans to acquire the new POS system with less than 15 percent
variance in total cost.

**3.2.2** **ACQUISITION COSTS BASELINE**

The baseline costs from the winning contractor\'s bid are expected to be
available in October 1995. The contractor will provide a life-cycle cost
estimate and monthly C/SSRs. Table 2 shows DeCA\'s estimates of the cost
to acquire and maintain the new POS system.

**Table 2. *Total POS Modernization Costs***

**[Cost category]{.underline} [Cost (\$)]{.underline}**

**Equipment 87,776,952**

**Software 2,942,541**

**Maintenance 38,095,719**

**Support services 13,949,027**

**Supplies 23,453,185**

**Total 166,217,424**

**DITCO contracting fee (2%) 3,324,348**

**Total baseline requirement (FY94 dollars) 169,541,772**

**Total escalated baseline requirement (then-year dollars) 195,090,889**

**4.** **TECHNICAL/ QUALITY OBJECTIVES**

The technical and quality goals for DeCA\'s new POS system focus on two
areas: how well the contractor meets the requirements identified in the
contract**;** and how well the performance of the contract meets DeCA\'s
technical and quality expectations for a modem POS system.

**4.1** **Technical Contract Requirements**

**4.1.1** **CONTRACT REQUIREMENTS GOAL**

DeCA\'s goal for the technical performance of its new POS system is to
keep the amount of significant technical variances during the testing,
implementation, and operation of the new POS system below one per
reporting period.

**4.1.2** **CONTRACT REQUIREMENTS BASELINE**

The technical baseline for DeCA\'s new POS system is established in the
POS modernization solicitation, DCA200-95-R-008, Section C, January
1995.

**4.1.3** **CONTRACT REQUIREMENTS SOURCE**

The source of technical performance data for DeCA\'s new POS system will
be the C/SSR technical variance reports and quarterly summaries.

**4.2 Customer Expectations for Technical Performance and Quality**

DeCA\'s POS program will provide new hardware and software to two groups
of customers: retail customers; and DeCA operators in the commissaries,
regional offices, and DeCA Headquarters. Retail customers will interface
with the new POS system during the check-out process. Commissary
personnel win operate the cash registers and in-store processors, while
regional and headquarters personnel will interface with the system
through terminals that accommodate remote maintenance, generation of
reports, and monitoring system performance. All of these interfaces will
be evaluated for \"customer satisfaction\" throughout the life of the
new POS system. The customer satisfaction data will be obtained through
DeCA surveys. Survey participants will include the POS program manager,
several commissary officers, and a sample of commissary customers.

**4.2.1** **TECHNICAL PERFORMANCE AND QUALITY GOAL**

The goal of DeCA\'s new POS system is to provide commissary customers
and system operators with a state-of-the-art POS system. DeCA seeks to
increase customer satisfaction by 20 percent over the current level of
satisfaction experienced by commissary customers and operators in the
categories listed in Table 3.

**4.2.2** **TECHNICAL PERFORMANCE AND QUALITY BASELINE**

The technical performance and quality baseline will be based on the
results of the customer satisfaction surveys, which will be available in
October 1995.

**4.2.3** **TECHNICAL PERFORMANCE AND QUALITY SOURCE**

DeCA will update its customer satisfaction surveys every two years**.**
Consistent with the results from the October 1995 survey, all future
surveys will focus on the performance categories shown in Table 3.

**Table 3. *Customer Expectations for Technical Performance and
Quality***

**[POS Operators]{.underline}**

> **Performance category Retail Store Region Headquarters**
>
> **Customers**

**Customer interface and ergonomics X X**

**Data integrity and accuracy X X X**

**Availability X X X**

**Maintainability X X X**

**Reliability X X X**

**Keeping pace with industry X X X**

**5. SCHEDULE OBJECTIVES**

**5.1 Program Milestones**

DeCA\'s goal is to update the POS system in all commissaries by the end
of FY98. DeCA\'s POS modernization program office has identified several
high-­level acquisition, test, and implementation milestones associated
with satisfying that goal. Further, DeCA has prepared a detailed
implementation schedule that will result in approximately 300
commissaries receiving the new POS system over a three-year period. DeCA
will monitor the contractor\'s progress against its major program
milestones and detailed installation schedule using the C/SSR and other
program documentation.

**5.1.1 ACQUISITION SCHEDULE GOAL**

DeCA\'s acquisition schedule goal is to meet all major acquisition,
test, and implementation milestones within three months of their
original dates, and all milestones with less than six months total
schedule variance.

**5.1.2 ACQUISITION SCHEDULE BASELINE**

Table 4 shows DeCA\'s major acquisition, test, and implementation
milestones for the new POS system.

**Table 4. *Major Acquisition, Test, and Implementation Milestones***

**[Milestone]{.underline} [Date]{.underline}**

**RFP release January 1995**

**Contract award October 1995**

**System testing complete 2nd Quarter 1996**

**Full operational capability (FOC) 4th Quarter 1998**

***Note:* RFP = request for proposal.**

**5.2 Implementation Schedule**

DeCA will implement POS systems on an as needed basis. DeCA operates
several stores that are in critical need of POS modernization.
Geographic installation efficiencies will be realized where possible,
however, the deployment of new POS systems will be made on a most needed
basis.

**5.2.1 IMPLEMENTATION SCHEDULE GOAL**

DeCA\'s goal is to implement the POS program with less than 15 percent
schedule variance over the three year deployment cycle.

**5.2.2 IMPLEMENTATION SCHEDULE BASELINE**

DeCA plans to install 50 new POS systems within one year of contract
award, 125 within the second year following contract award, and another
125 during the third year. The deployment schedule is not location
specific and allows the necessary flexibility to meet DeCA\'s
operational needs.

**DEFENSE FINANCE AND ACCOUNTING SERVICE**

**FEDERAL INFORMATION PROCESSING**

**INFRASTRUCTURE**

**PROPOSED**

**PERFORMANCE MEASURES**

**Introduction:** The Defense Finance and Accounting Service
Infrastructure Project will establish the infrastructure and
capabilities to support the new Defense Finance and Accounting Service
(DFAS) organizational structure. In 1991, to streamline operations and
increase effectiveness, the Department of Defense began consolidating
functions and personnel from over 300 Defense Accounting Offices into 21
operating locations aligned under five DFAS processing centers. DFAS
needs IT resources to standardize and consolidate finance and accounting
operations. The following is DFAS's performance measurement plan:

**1. Customer Satisfaction**

**Goal:** Meet the long term FIP Infrastructure needs of DFAS and
improve access to information between DFAS Centers and Operating
Locations.

**Measure:** FIP Infrastructure contractor\'s performance in achieving
customer objectives after award and degree of satisfaction.

**Strategy:** The FIP Infrastructure Program Manager (PM) will conduct a
survey of end users, project officers, and technical leads (collectively
referred to as users) every six months after implementation of proposed
technical architecture. Users will complete survey instrument that will
be designed to rate the contractor\'s integration attributes. Attributes
rated will include:

> **a. Knowing the Customer**
>
> \- Understands how proposed technical architecture will integrate with
> existing architecture.
>
> \- Sensitive to customer re-training/tooling needs for proposed
> technical architecture.
>
> \- Constantly looks for ways to help DFAS implement operating
> locations on schedule.

**b. Competence**

\- Projects implemented effectively

\- Technical competence

\- Re-training/tooling capability

**c. Reliability**

\- Provides deliverables that are on time, within budget, and complete

> \- Regularly follows-up before and after implementation of accepted
> technical architecture proposal.

**d. Strategic Partnership**

\- Invests the time needed to learn about existing DFAS technical
environment

\- Makes recommendations to help DFAS achieve strategic goals

\- Proposes a technical architecture that has a measurable bottom line
impact.

**e. Customer Oriented Culture**

\- Makes recommendations based on customer need, not contractor\'s
priorities

\- Facilitates transfer of skills to DFAS employees

\- Develops partnership to achieve mutual goals.

**f. Assurance**

\- Uses proven analysis methodology

\- Shows accountability throughout project

\- Consistent staffing

**g. Responsiveness/Problem Solving**

\- Responds to issues of dissatisfaction

\- Uses effective change request procedures

\- Takes ownership for problems

\- Flexible in solving problems

**h. Communication**

\- Good listening skills

\- Effective presentations

\- Awareness of their full range of services

\- Regular updates on project status

**i. Courtesy**

\- Shows mutual interest

> \- Effectively manages relationship (Coordinates with various
> stakeholders across organization)

The survey will identify the project requirements which are successful
or unsuccessful at meeting DFAS\' technical infrastructure needs. The PM
will analyze problem areas, report results, and develop recommendation
for promoting successful practices and correcting unsuccessful ones.

**2. Implement DFAS Operating Locations**

**Goal:** Reduce DFAS operating costs by providing a standardized
technical architecture which facilitates the consolidation of finance
and accounting operations.

**Measure:** Reduce number of DFAS operational sites from over 300 to 21
locations aligned under the 5 DFAS processing centers. This measure will
involve the implementation of the required technical architecture and
network to support the scheduled consolidation and opening of designated
operating locations.

**Strategy:** The PM will ensure implementation of cost effective
technical solutions that will meet DFAS consolidation requirements.
Monitor the implementation schedule for the designated operating
locations and ensure technical architecture is in place prior to
scheduled opening of operating locations. An evaluation and cost
assessment will be conducted for each operating location six months
after opening to determine if technical requirements and goals were met.
Results of the multiple operating location evaluations will be
consolidated in a single annual report. The evaluation will include an
assessment of the degree of the consistency, compatibility and
interoperability of all system components and work flow applications.

**3. Implement an Enterprise Local Area Network for DFAS**

**Goal A:** Reduce the operating and overhead cost of managing a LAN.

**Measure:** Establish a Central Enterprise LAN (ELAN) Management team
with less than 150 personnel to support the ELAN for all of DFAS.
Implementing separate or standalone LANs by site, organization or office
would require 440 personnel to provide LAN management support because
each individual LAN would require a LAN support person.

**Strategy:** It is our strategy to implement an Enterprise LAN for the
total DFAS workforce instead of several smaller LANs by site,
organization, or office. Since the total size of the DFAS workforce is
approximately 22,000 personnel, we will achieve significant savings by
implementing an ELAN with central LAN management. A central LAN
management team will be established at a ratio of one LAN support person
per 150 to 250 users in lieu of the average projected support of one LAN
support person per 50 users for local implementation of a LAN.

**Goal B:** Reduce the need to build redundancy into the LAN.

**Measure:** Standardize the LAN solution implemented at DFAS sites by
reducing the number of different LAN solutions from three to one.
Establishing master licensing agreements for standard office automation
(OA) software instead of buying approximately 22,000 separate copies of
OA software to support each DFAS employee.

**Strategy:** The strategy is to implement a standard ELAN solution
which will eliminate the different LAN solutions currently implemented
within DFAS. Through this contract the winning vendor will recommend a
technical infrastructure solution which will be approved by the Program
Manager. Once approved the technical infrastructure solution will be
implemented as the DFAS ELAN. A part of the technical infrastructure
solution will address the office automation software requirements within
DFAS. Our strategy is to establish master licensing agreements for OA
software to support the DFAS OA requirement instead of obtaining
individual copies of this same software for each employee workstation. A
master licensing agreement will provide DFAS the greatest flexibility to
provide the required OA software for every employee that needs it as
well as provide for future upgrades and software support of that same
software.

**DEPARTMENT OF DEFENSE**

**HIGH PERFORMANCE COMPUTING MODERNIZATION PROGRAM**

**MAJOR SHARED RESOURCE CENTERS**

**Performance Metrics**

**Introduction:** The Defense High Performance Computing (HPC)
Modernization Program will provide state-of-the-art high performance
computing and high speed networks for DoD scientists and technicians.
The HPC systems will be comparable to those in academia and industry and
will enhance the development of weapon and warfighting systems. The
objective is to improve the quality of research through more realistic
models and quicker solutions. The following is the draft HPC performance
measurement plan for the high performance computers only. Performance
measures for the high speed networks were not available.

**Mission Description.**

The Director, Defense Research and Engineering (DDR&E) is responsible
for ensuring the modernization of the Department of Defense (DoD)
laboratories with high performance computing (HPC) capabilities to
support the science and technology (S&T) program and the developmental
test and evaluation (DT&E) program. Computational science and high
performance computing are playing a pivotal role in changing the way
research and development (R&D) are conducted. HPC supports the
increasingly complex and computationally intensive R&D required by the
S&T and DT&E communities. This role is illustrated in two key documents
recently prepared by the DoD High Performance Computing Modernization
Program (HPCMP) within DDR&E. The *Mission Need Statement* (MNS), dated
June 1995, addresses the need to provide and sustain significantly
higher performance computing capability. The underlying requirements are
captured in the HPCMP *Functional Description*, dated June 1995. The
HPCMP will undertake activities designed to keep the major shared
resource centers (MSRCs), the distributed centers (DCs), and supported
research community abreast of technology advancements and trends in the
use and exploitation of the best state-of-the-practice in all areas of
HPC including computers, algorithms, and software needs.

The scope of the DoD HPC modernization effort includes fulfillment of
the HPC requirements for research, development, test, and evaluation
(RDT&E) efforts under the cognizance of the DDR&E and the Director of
Test, Systems Engineering and Evaluation (DTSE&E). The program must
encompass and address the need for a complete systems environment
enabling the researcher to do work seamlessly from the workstation to
the computing platform. The systems are incomplete without adequate data
storage, scientific visualization, software tools and libraries, and
advanced algorithm capability. The program must address the need for
robust wide area networking to link the MSRCs to DCs and to the
scientists and engineers at remote facilities. This will be accomplished
through the use of the Defense Research and Engineering Network (DREN).
Finally, it is necessary to engage *academe* and industry in cooperative
initiatives to improve applications and parallel algorithm development,
software engineering methodologies, and software tools and libraries to
use the HPC systems effectively. This effort is known as the Common HPC
Software Support Initiative (CHSSI). The DoD HPC environment will be
incomplete and will operate at a greatly reduced effectiveness if any of
these components is slighted.

**System Description.**

The HPCMP has been successful in forging a true DoD corporate program to
address the HPC requirements of the S&T and DT&E programs. This
corporate program supports a vision of maintaining and extending the
United States' technological advantage in warfighting systems by
providing to its scientists and engineers high performance computing
capabilities and programming environments that are second-to-none in
industry or academia. Four components have been created: (1) a set of
four Major Shared Resource Centers (MSRCs) that are patterned after the
National Science Foundation's supercomputing centers; (2) a group of
smaller Distributed Centers that focus on either exploring the
state-of-the-art in computational capabilities where local expertise
exists within the DoD laboratories and test centers or providing HPC
capabilities to address HPC requirements that cannot be met by the
MSRCs; (3) the Defense Research and Engineering Network (DREN) that
interconnects DoD's user community and its computational centers and
allows remote users routine access to these centers; and (4) the Common
HPC Software Support Initiative (CHSSI), which provides support for the
efficient, effective transition of DoD application software to the newer
scalable architectures with their promise of substantial increases in
computational performance.

## Major Shared Resource Centers

This document focuses on performance metrics for the MSRCs. Since
contract awards have not yet been made for the MSRCs, there are a number
of detailed, contract-level metrics that cannot be determined until
awards are made. Metrics contained in this document include HPCMP
program-level metrics for all components and MSRC-level metrics not
detailed in the contract awards. A detailed description of the MSRC
component of the program follows. The cornerstone of the HPCMP is the
set of four MSRCs that will provide complete HPC environments, including
a substantial majority of computing capability provided by the program.
The nature of the majority of user requirements, large batch-oriented
scientific and engineering computations, allow this consolidation of
computational resources at a few major sites with large systems that can
be time-shared effectively from remote sites.

This consolidation allows the very largest of systems to be provided,
resulting in an ability to solve the largest possible computational
science problems. These MSRCs are located at the Army Corps of Engineers
Waterways Experiment Station, Vicksburg, MS; the Naval Oceanographic
Office, Stennis Space Center, MS; the Army Research Laboratory, Aberdeen
Proving Ground, MD; and the Aeronautical Systems Center,
Wright-Patterson Air Force Base, OH. Each of the four MSRCs currently
operates large HPC systems that are available to the entire S&T and DT&E
user communities.

The HPCMP is executing a large acquisition that will result in complete
HPC environments at each of the four sites, including substantial
upgrades in computational capabilities at each center. The complete
environment will also include extensive capabilities to address user
requirements other than hardware, including software, programming
environments, and training. Software will include not only the usual
complement of systems software and programming tools, but also an
extensive array of commercially available applications software packages
in all computational technology areas. A unique feature at each MSRC
will be the programming environment and training component. This
component, in addition to extensive training on a wide variety of user
needs, will also include the development of systems tools and software
to facilitate the effective and efficient utilization of scalable
architectures as DoD's computational scientists and engineers transition
their workload to these promising systems.

**HPCMP Program-Level Metrics.**

+--------------+-------------------------------------------------------+
| **Metrics**  | **Description**                                       |
|              |                                                       |
| **Category** |                                                       |
+--------------+-------------------------------------------------------+
| Cost         | Are program cost increases within 10% of the Approved |
|              | Program Baseline? ( 15% is a Baseline breach)         |
+--------------+-------------------------------------------------------+
| Schedule     | Is schedule slippage within 60 days of the dates in   |
|              | the Approved Program Baseline? (90 days is a Baseline |
|              | breach)                                               |
+--------------+-------------------------------------------------------+
| Requirements | Is the fraction of sustained performance requirements |
|              | that are satisfied between 20% and 80%?               |
+--------------+-------------------------------------------------------+
| Performance  | Have Critical Operational Issues been met during      |
|              | developmental test and evaluation and operational     |
|              | test and evaluation events? (Enclosure 1)             |
+--------------+-------------------------------------------------------+
|              | Is DREN-level data transport availability, as         |
|              | experienced by HPC end-user sites and users, greater  |
|              | than or equal to 98.5%?                               |
+--------------+-------------------------------------------------------+
|              | Is total utilization by the Services/Agencies,        |
|              | aggregated across all HPCMP HPC systems, within 5% of |
|              | the officially allocated values?                      |
+--------------+-------------------------------------------------------+
| User         | In the User Satisfaction Survey, is the average of    |
| Satisfaction | responses to items 4-7 in the "General Program"       |
|              | question (Enclosure 2) "Satisfied" or higher?         |
+--------------+-------------------------------------------------------+

HPCMP program-level metrics describe the performance of the overall HPC
Modernization Program. Cost and schedule are considered at the program
level. Requirements, performance, and user satisfaction are all
addressed in terms of issues that incorporate the entire program. For
example, the fraction of sustained performance requirements that are
satisfied is determined as aggregated over all HPC systems provided by
the program, not on an individual basis. Likewise, allocations to each
Service/Agency are examined in the aggregate over all HPC systems
provided by the program, not system by system for which substantial
deviations from the overall allocation to each Service/Agency may occur.
A major segment of the performance metrics are codified in the critical
operational issues as given in the programs Test and Evaluation Master
Plan (TEMP). The user satisfaction survey is used to determine how well
the overall program is performing in disseminating required information
to its users and allowing an easy access process to its resources.

**MSRC-Level Metrics.**

  -------------- ---------------------------------------------------------
  **Metrics      **Description**
  Category**     

  Cost           Are costs at each MSRC within 10% of the Approved Program
                 Baseline? ( 15% is a Baseline breach)

  Schedule       Is schedule slippage of major events at each MSRC within
                 60 days of the dates in the Approved Program Baseline?
                 (90 days is a Baseline breach)

  Performance    Is the performance of each MSRC contractor rated as
                 "Excellent" ( 93%) in Technical Performance, Management,
                 and Cost on each semi-annual Award Fee Determination?

                 Are the systems at the MSRCs maintaining an effectiveness
                 level (operational availability) greater than or equal to
                 95%?

  Turnaround     Is the ratio of the total job turnaround time to job
  Time           computation time less than or equal to 5 for each HPCMP
                 system at each MSRC?

  User           In the User Satisfaction Survey, is the average of
  Satisfaction   responses rating the different features of each HPCMP
                 system at each MSRC (Enclosure 3) "4" or higher?
  -------------- ---------------------------------------------------------

MSRC-level metrics pertain to each individual MSRC, and, collectively,
they contribute to the program-level metrics as previously described.
Cost and schedule are examined for individual MSRCs. Performance is
measured in terms of each MSRCs contractual performance and the
availability of its systems. An analysis of turnaround time on each
system is crucial to determine whether available resources are meeting
requirements in a timely fashion, particularly for time-critical
applications. Responses to the user satisfaction survey based on each
HPC system are used to determine how those systems are individually
meeting users' needs.

**Enclosure 1: Critical Operational Issues (COI) for Developmental and
Operational Test and Evaluation.**

Criteria, based on user-validated requirements, will be developed for
each COI as test planning progresses. Supporting the evaluation of each
criterion will be one or more measures of effectiveness (MOEs) and the
associated measures of performance (MOPs) which will define the data
requirements. An actual S&T or DT&E subject matter expert has been
assigned for each COI by the HPCMO as a technical point of contact to
the Joint Interoperability Test Command (JITC). A full listing of the
criteria, MOEs, MOPs, and data requirements will be included in the
JITC\'s operational test and evaluation plan (OTEP). The JITC will
collect data during operational test and evaluation (OT&E) that will
support the resolution of the following COIs.

**Critical Operational Issues**

  --------- ------------------- -----------------------------------------------
  **COI**   **Title**           **Description**

  1         Performance         Are user operational performance requirements
                                of the DoD S&T and DT&E communities met?

  2         Compatibility,      Does the HPCMP system provide the compatibility
            Interoperability,   and interoperability required for sharing
            and Integration     resources among local and remote users; as well
                                as the capability to integrate with existing
                                HPC and planned HPCMP assets across networks?

  3         Reliability,        Does the HPCMP system operate consistently
            Availability and    enough to support user problem solving
            Maintainability     requirements?

  4         Growth Potential    Does the PL1 HPCMP system provide a
                                technologically sound foundation for migration
                                to PL2 and PL3?

  5         Programming         Do the programming environments at the MSRCs
            Environment and     promote development of new software, transfer
            Training            of existing custom software, and technology
                                transfer with academia, industry, and other
                                government HPC activities?

  6         Usability           Does the HPCMP system\'s human-computer
                                interaction (HCI) support ease of learning,
                                ease of use, safety, effectiveness and
                                efficiency and user satisfaction?

  7         Security and        Does the HPCMP system protect its information,
            Vulnerability       provide adequate system protection and possess
                                the ability to survive potential threat
                                interference?
  --------- ------------------- -----------------------------------------------

**Enclosure 2: HPCMP User Satisfaction Survey General Program Rating
Question**

Please indicate your level of satisfaction with the HPCMP in the
following areas by placing a check in the appropriate column.

  ---------------- --------- ---------------- ------------------ ----------------- --------------- -------------
                   **N/A**   **Totally        **Dissatisfied**   **Indifferent**   **Satisfied**   **Totally
                             Dissatisfied**                                                        Satisfied**

  1\. Access to                                                                                    
  the HPCMO                                                                                        
  Program Manager                                                                                  

  2\. Access to                                                                                    
  other HPCMO                                                                                      
  staff                                                                                            

  3\. Location of                                                                                  
  the HPCMO                                                                                        

  4\. Access to                                                                                    
  information                                                                                      
  about the                                                                                        
  program *via*                                                                                    
  electronic means                                                                                 

  5\. Access to                                                                                    
  information                                                                                      
  about the                                                                                        
  program *via*                                                                                    
  the HPCMO                                                                                        

  6\. Access to                                                                                    
  information                                                                                      
  about the                                                                                        
  program *via*                                                                                    
  HPC Users Group                                                                                  
  meetings                                                                                         

  7\. Ease in                                                                                      
  establishing                                                                                     
  accounts on                                                                                      
  HPCMP systems                                                                                    
  ---------------- --------- ---------------- ------------------ ----------------- --------------- -------------

![](media/image7.emf)

**Enclosure 3: HPCMP User Survey System Satisfaction Question.**

**FEDERAL AVIATION ADMINISTRATION**

***Oceanic System Performance Metrics***

**Introduction:** The Oceanic System Development and Support (OSDS)
contract will provide the Federal Aviation Administration (FAA) with
services to develop a system that automates the air traffic control in
the Atlantic, Pacific, and Arctic Oceans. Currently, with the exception
of Alaska, radar coverage is non-existent, navigation is by limited
on-board systems, and air-ground communications come through a third
party via high frequency radio. Because the existing system is manual,
the FAA has limited ability to accommodate changes in aircraft flight
paths, achieve better aircraft fuel and time efficiency, and avoid air
turbulence. The objectives of OSDS are to maintain the existing level of
safety while reducing the required distance separating in-flight
aircraft; permit frequent in-flight path changes; and increase the
productivity of air traffic controllers. The following is a description
of the OSDS performance measures:

The Oceanic and Offshore Integrated Product Team (IPT) selected
measurements to monitor and assess: product efficiency and quality,
system cost and schedule, customer/user benefits, partner satisfaction,
and team processes. Oceanic and Offshore metrics are broken out into
four key areas. A brief synopsis for each area and their components are
discussed below.

**1. 0 Oceanic Service Metrics**

The Oceanic Service Metrics consist of Oceanic Service Efficiency,
Oceanic Service Workload and Oceanic Service Fidelity. These metrics
examine the impact and benefit of oceanic automation for both our
customers and FAA partners. These customers include the airlines,
general aviation and military aircraft; our FAA partners include oceanic
controllers, system maintainers, automation staff, and flight standards.
The oceanic service metrics provide measures to assess cost and time
savings to the users and maintainers of the system, workload reduction
for the controllers, as well as the IPT's ability to manage service
efficiency and fidelity.

**Oceanic Service Efficiency**

-   Monitors the ability of the oceanic system to provide reduction in
    direct operating costs to the users. FAA will measure its success by
    how much it will:

\- save flying time

\- save fuel

-   Monitors the ability of the system to meet users\' changing needs.
    FAA will measure its success by how much it will:

\- reduce the procedural restrictions in the system

\- increase the number of user preferred route and altitude requests

\- reduce the deviations between routes and altitudes requested versus
trajectories flown

\- increase acceptable level of traffic in sectors/airspace

\- increase the number of decisions involving pilots/controllers as well
as ATM/AOCs collaboration

-   Predicts the variance in the system as experienced by the users. FAA
    will measure success by how much it will:

\- reduce the difference between estimated and actual flying times

\- reduce variation in system performance associated with changes in
weather

\- increase acceptance of users\' first preferences for oceanic routes,
entry altitudes and entry times made with direct user inputs

-   Access the ability of the users to enter the oceanic airspace and
    obtain services on demand. FAA will measure success by how much it
    will:

\- accommodate random routing and dynamic rerouting

\- decrease clearance processing time by direct pilot/controller
communications

-   Measure the amount of time over the optimum that it takes to
    complete the operation. FAA will measure success by how much it
    will:

\- reduce ground delays

\- permit enroute flying time closer to the optimum

**Oceanic Service Workload**

-   Measures change in controller workload and productivity

-   Provides insight into the establishment of priorities for new
    automation systems

-   Measures manual controller tasks associated with each sector and
    compare to the workload with automation

-   Compares estimated and predicted controller workload levels under
    specified operational environmental conditions.

**Oceanic Service Fidelity**

-   Ensures that the delivered system is providing additional measures
    of safety by monitoring the number of operational errors and
    deviations per year

-   Provides a means to determine if the delivered automation system is
    assisting in the reliability of the communications service.

**2.0 Oceanic Acquisition Metrics**

The Oceanic Acquisition Metrics consist of Oceanic Acquisition
Development Costs, Oceanic Acquisition Quality and Oceanic Acquisition
Development Time. A selected set of Oceanic Acquisition Metrics are
being used by the IPT to effectively monitor all contracts. The Oceanic
Acquisition Metrics incorporate software management indicators to
monitor acquisition development costs, quality and development time.
Data will be required as an OSDS contract deliverable, which will be
analyzed by the metrics team and presented to the IPT. The specific
software management indicators used for each area are described in the
following sections.

**Oceanic Acquisition Development Costs**

-   Monitors costs associated with the development of the oceanic
    projects

-   Software management indicators have been levied on all new contracts
    within the oceanic area. Use of software indicators will permit
    project tracking, cost, schedule and risk mitigation.

-   A specific software management indicator which will be used within
    the program is the indicator for software size. The software size
    indicator tracks changes and the magnitude of the software
    development effort. The lines of code to be developed directly
    relate to the software engineering effort necessary to build the
    system

**Oceanic Acquisition Quality**

-   Product quality will be measured throughout the life cycle to ensure
    that oceanic requirements are not abrogated by the contractor.

-   The metrics will assess oceanic quality by obtaining input from
    oceanic users, technicians, logistics personnel and quality
    reliability officers.

-   The software management indicators require the developing contractor
    to provide statistics and trend analysis charts of the project s
    software requirements and design stability. Reports generated by
    oceanic contractors, will show the contractors internal processes as
    they relate to changes in requirements down to modifications of the
    actual Computer Software Configuration Item.

**Oceanic Acquisition Development Time**

-   Provide visibility into the oceanic contractors\' adherence to their
    baselined schedules.

-   Software management indicators in contracts allow the IPT to
    determine from contract award throughout the life of the project
    significant indicators, which may cause schedule impact if not
    monitored.

-   The indicators provide visibility into software risk areas and
    software complexity, as well as examine the contractor\'s actual
    design process.

-   The main goal of these metrics will be to show any schedule impact,
    early in the project, so that the IPT could effect a resolution for
    a possible schedule delay.

**3.0 Oceanic Service Program Costs**

These metrics will establish a means to determine the change in
operational and maintenance costs through the use of new automation.

-   Monitors the actual operational and maintenance costs for oceanic
    facilities, equipment, and staffing.

**4.0 Oceanic Team Metrics**

The specific metrics outlined below measure the team dynamics and the
performance of team decision processes. These metrics will assess team
relationships and quantify the benefits derived by empowering the IPT.

These metrics will be monitored with the goals of demonstrating the
benefits from the transition from a matrix organization to an IPT and
continuing process improvements to increase team effectiveness.

**Oceanic Cycle Time**

-   Provides a means to measure the oceanic acquisition and
    implementation IPT processing times, as compared with other FAA
    projects, as well as, previous oceanic processing times before
    formulation of the IPT.

-   The goal is to reduce the cycle time by utilizing the IPT expertise
    to streamline lengthy processes.

**Oceanic Team Effectiveness**

-   Monitors IPT effectiveness and efficiency. A survey is distributed
    biannually to monitor the IPT's decision-making effectiveness, sense
    of empowerment, value of team efforts to the end product, and
    overall commonality of goals and objectives.

**Oceanic Product Quality and Satisfaction**

-   Monitors customer satisfaction by preparing surveys which are
    distributed to oceanic users throughout a project s life-cycle.

-   These monitor our partners\' satisfaction as documentation is
    delivered, systems are deployed, and training has been accomplished.

**Customer Satisfaction Survey**

+----------------------------------------------------+-----------------+
| **Airways Facility Personnel**                     | Strongly        |
|                                                    | Disagree -      |
|                                                    | Strongly Agree  |
+----------------------------------------------------+-----------------+
| 1.  The PIP clearly identified the implementation  |      NA    |
|     planning process.                              |                 |
+----------------------------------------------------+-----------------+
| 2.  Adequate funding was provided from the program |      NA    |
|     office to support implementation activities.   |                 |
+----------------------------------------------------+-----------------+
| 3.  The site implementation plan clearly           |      NA    |
|     identified space, power, HVAC, and             |                 |
|     communications requirements.                   |                 |
+----------------------------------------------------+-----------------+
| 4.  Changes to AF procedures were clearly defined. |      NA    |
+----------------------------------------------------+-----------------+
| 5.  Maintenance training materials were thorough   |      NA    |
|     and comprehensive.                             |                 |
+----------------------------------------------------+-----------------+
| 6.  Maintenance operation impacts were clearly     |      NA    |
|     defined.                                       |                 |
+----------------------------------------------------+-----------------+
| 7.  The hardware components were delivered on      |      NA    |
|     schedule.                                      |                 |
+----------------------------------------------------+-----------------+
| 8.  Delivered hardware documentation is thorough   |      NA    |
|     and comprehensive.                             |                 |
+----------------------------------------------------+-----------------+
| 9.  The delivered system is easy to maintain.      |      NA    |
+----------------------------------------------------+-----------------+
| 10. Site AF staffing impacts were clearly          |      NA    |
|     identified prior to system delivery.           |                 |
+----------------------------------------------------+-----------------+
| 11. Test requirements were clearly documented in   |      NA    |
|     the site test plan.                            |                 |
+----------------------------------------------------+-----------------+
| 12. Adequate technical resources were provided to  |      NA    |
|     test the system.                               |                 |
+----------------------------------------------------+-----------------+
| 13. Support and test equipment were provided and   |      NA    |
|     delivered on schedule.                         |                 |
+----------------------------------------------------+-----------------+
| 14. The maintenance plan was clearly defined and   |      NA    |
|     timely promulgated.                            |                 |
+----------------------------------------------------+-----------------+
| 15. Adequate supply support materials were         |      NA    |
|     delivered on schedule.                         |                 |
+----------------------------------------------------+-----------------+
| **Air Traffic Personnel**                          | Strongly        |
|                                                    | Disagree -      |
|                                                    | Strongly Agree  |
+----------------------------------------------------+-----------------+
| 1.  The new system makes my job easier.            |      NA    |
+----------------------------------------------------+-----------------+
| 2.  The new system allows me to do the same tasks  |      NA    |
|     faster.                                        |                 |
+----------------------------------------------------+-----------------+
| 3.  The training fully prepared me for using the   |      NA    |
|     new system or function.                        |                 |
+----------------------------------------------------+-----------------+
| 4.  The new system fully meets its intended        |      NA    |
|     functionality.                                 |                 |
+----------------------------------------------------+-----------------+
| 5.  Procedural changes for the new system or       |      NA    |
|     function were fully defined.                   |                 |
+----------------------------------------------------+-----------------+
| 6.  Transition to the new system or function was   |      NA    |
|     easy.                                          |                 |
+----------------------------------------------------+-----------------+
| **Air Traffic Personnel: Automation, Airspace and  | Strongly        |
| Procedures**                                       | Disagree -      |
|                                                    | Strongly Agree  |
+----------------------------------------------------+-----------------+
| 1.  The PIP accurately defined the implementation  |      NA    |
|     planning process.                              |                 |
+----------------------------------------------------+-----------------+
| 2.  The technical and training documentation was   |      NA    |
|     thorough and comprehensive.                    |                 |
+----------------------------------------------------+-----------------+
| 3.  The technical documentation accurately         |      NA    |
|     reflects the software.                         |                 |
+----------------------------------------------------+-----------------+
| 4.  The system was very stable when first          |      NA    |
|     delivered.                                     |                 |
+----------------------------------------------------+-----------------+
| 5.  Adequate resources were provided from the IPT  |      NA    |
|     for the system tests.                          |                 |
+----------------------------------------------------+-----------------+
| 6.  The data reduction and diagnostic programs are |      NA    |
|     helpful and easy to use.                       |                 |
+----------------------------------------------------+-----------------+
| 7.  The program office provided adequate           |      NA    |
|     assistance in developing the Site              |                 |
|     Implementation Plan (SIP).                     |                 |
+----------------------------------------------------+-----------------+
| 8.  The region and site personnel were involved    |      NA    |
|     early in the development and decision process. |                 |
+----------------------------------------------------+-----------------+

**Immigration and Naturalization Service**

**Information Technology Partnership**

**Introduction:** The Information Technology Program (ITP) provides the
Immigration and Naturalization Service (INS) a support services task
order contract. The ITP contractor functions as a proactive,
forward-thinking, strategically oriented partner who assists the INS to
develop information management strategies. The contractor provides
support for various tasks throughout the IT life cycle including systems
planning, analysis, design, development and maintenance. The ITP
objectives are to respond to increased workload throughout the agency by
automating information systems, and to establish a standardized systems
architecture to provide access to critical mission information. The
following describes the INS approach to performance measurement.

![](media/image8.emf){width="6.50625in" height="5.080555555555556in"}

![](media/image9.emf){width="6.50625in" height="6.4in"}

![](media/image10.emf){width="5.9944444444444445in" height="7.25in"}

![](media/image11.emf){width="5.021527777777778in"
height="6.216666666666667in"}

![](media/image12.emf){width="5.995833333333334in"
height="4.488888888888889in"}

**Social Security Administration**

**Integrated Workstation/Local Area Network**

**Introduction:** Intelligent Workstation and Local Area Network
(IWS/LAN) acquisition is the first phase of the Social Security
Administration's (SSA) infrastructure modernization program. The SSA
plans to modernize its processes to improve the disability and appeals
process, to improve public access to SSA, to turn SSA into a paper-less
agency and to establish a cooperative processing architecture. IWS/LAN
will replace SSA's existing "dumb" terminals with intelligent
workstations and LANs at approximately 1,300 district and branch
offices, program service and teleservice centers. SSA developed six
categories of measures for IWS/LAN.

**Category 1**

**[Description:]{.underline}** Productivity benefits identified for the
state Disability Determination Services (DDS) using IWS/LAN.

**[Metric:]{.underline}** A computation of the DDS productivity gain by
comparing the pre­IWS/LAN baseline data with the post IWS/LAN
implementation data.

**[The measure is:]{.underline}** The number of cases cleared on the
pre-IWS/LAN DDS\'s product

and the post-IWS/LAN DDS\'s production.

**[The target is:]{.underline}** Target productivity gains will be
established upon award of contract

The existing productivity baseline will be computed at that time

Target increase percentages or numeric projections against the

baseline will be set and tracked using the measures indicated.

**[Data Source:]{.underline}** The Office of Disability will be using
the Comprehensive Productivity

Measurement (CPM) to measure the effectiveness of IAN systems in

the disability determination services (DDSs). The CPM is the most

accurate productivity indicator available for measuring DDS

productivity. CPM is available from the Cost Effectiveness

Measurement System (CEMS) on a quarterly basis. The CEMS

tracks units of work per person-year.

**[Responsible]{.underline}**

**[Component:]{.underline}** Deputy Commissioner for Programs, Office of
Disability, Division of Field Disability Operations

**[Report]{.underline}**

**[Frequency:]{.underline}** The report is produced quarterly. SSA will
report semi-annually on the cumulative achievement of the benefits
organized on a State by State basis.

**Category 2**

**[Description:]{.underline}** SSA baseline automation savings generated
by the implementation of IWS/LAN in SSA field offices and hearing
offices.

**[Metric:]{.underline}**

a\. Reports on productivity of SSA field office employees by comparing
production data before and after IWS/LAN installation.

b\. A computation of the change in hearing level productivity by
comparing pre-IWS/LAN receipt baseline data with post-IWS/LAN receipt
data.

**[The measure is:]{.underline}**

a\. The productivity change is the earned hours (volume (total cases
completed) \* national unit rate) divided by actual hours for a control
group compared to itself before and after IWS/LAN.

The national productivity change for non-IWS/LAN field offices compared
to IWS/LAN field offices.

b\. The total OHA field office productivity index (PI) which consists of
the primary activities in the disposition of cases, i.e., case
preparation, decision drafting, and decision typing. These activities
will be weighted according to the relative \"hands-on\" time for these
activities involved in getting out a decision. To determine the PI, the
weighted \"points\" are multiplied by the \"number\" of actions for each
activity arriving at the activity points. All activities are added to
each other to compute the total points. The total points are divided by
5.7 (the value of an equivalent disposition) yielding production. The
total production is then divided by the staff work-years (total hours
worked divided by 2080) which yields the PI. A period before IWS/LAN
installation will be established as a baseline to assess impact on PI
resulting from IWS/LAN installation.

**[The target is:]{.underline}**

a\. Targeted productivity changes will be computed starting from the
baseline for the measures that exist at award of the IWS/LAN contract.

b\. Targeted productivity index will be computed starting from the
baseline at the time prior to IWS/LAN installation compared to
productivity index after IWS/LAN. Target productivity increases will be
set relational and incremental to achieving the service level objectives
outlined in the Improve the Appeals Process strategic initiative.\*

**[Data Source:]{.underline}**

a\. For field offices, productivity measures will be based on SSA\'s
Productivity and Cost Measurement System (PCMS).

b\. OHA will pull productivity data for selected hearing offices from
its Monthly Activity Report (MAR). The MAR is generated primarily from
data entered into the Hearing Office Tracking System in each hearing
office. The before and after comparison will be made for the offices on
a monthly basis.

**[Responsible]{.underline}**

**[Component:]{.underline}** Deputy Commissioner for Human Resources,
Office of Workforce Analysis \-- PCMS

Deputy Commissioner for Programs, Office of Hearing and Appeals \-- MAR

**[Report]{.underline}**

**[Frequency:]{.underline}** SSA will report semi-annually.

\* The Social Security Strategic Plan: A Framework for the Future, SSA
Publication No. 01-001, September 1991, and the Report of the Strategic
Priority Workgroup, Improve the Appeals Process, January 1992

**Category 3**

**Description:** Improvements in service delivery related to IWS/LAN
usage.

**[Metrics:]{.underline}**

a\. SSA will report on existing quality measures for the 800 number.
Based on pilots already conducted, SSA expects improved payment accuracy
and service delivery accuracy through use of the 800 Number Expert
System on the IWS/LAN. Payment accuracy reflects whether the Teleservice
Center action/information (or failure to act or provide information) has
a reasonable potential to affect payment or eligibility. Service
delivery accuracy reflects whether the Teleservice Representative
(TSR)\'s response results in the desired service delivery outcome to the
customer. SSA will compare IWS/LAN 800 Number Expert System TSR user
results to non-IWS/LAN TSR user results.

b\. SSA is implementing dial-in access as part of the IWS/LAN project.
This dial-in access gives SSA employees who are outside the full-time
offices the same functional ability and significantly improves service
levels to the SSA customers who deal with SSA at geographically remote
contact stations. This metric will look at the increase in number of
remote locations and client population served by this new capability.

**[The measure is:]{.underline}**

a\. The payment error/accuracy and service error/accuracy counts of
teleservice representative\'s use of the 800 Expert System.

b\. The number of remote service sites and the population served due to
the availability of dial-in access.

**[The target is:]{.underline}**

a\. Error/accuracy counts for both payment and service will have
improvement targets set against the current baseline for these measures
existing at the time of the IWS/LAN contract award. Statistically
significant targets will be selected for percentage or numeric measures.
As applicable, these targets will be relational and incremented to
accuracy objectives set to accomplish SSA\'s customer service goals
(e.g. \"We will provide service through knowledgeable employees\... and
meeting our customer\'s preferences for \"handling business with SSA by
telephone, a majority prefer to do so through SSA\'s 800 number.\")\*.

\* [Putting-Customers First]{.underline} SSA Publication No. 05-10030,
September 1994

b\. Targets for remote service sites and population served will vary
from month to month over the IWS/LAN implementation period. General
national targets will be to average 3.5 percent increase per month in
number of offices with remote automated services available and an
average of 175,000 increase in client population served by this new
capability per month. These targets are set to achieve the goals of
SSA\'s customer service pledge to \"provide equitable service to our
customers\" and ensure that \"our services are accessible\".\*

Improvements in service delivery related to IWS/LAN metric category 3 is
one that may be modified to reflect measurements in systems that SSA
establishes to track its performance against its new [Customer Service
Standards]{.underline}.

**[Data Source:]{.underline}**

Metric (a)\--The Office of Program and Integrity Reviews (OPIR) monitors
a sample of the 800 number calls to produce the payment accuracy and
service delivery accuracy measures.

Metric (b)\--Improved Public Access - The Office of the Deputy
Commissioner for operations (DCO) will provide information on the number
of contact stations and remote service sites that are served using the
new dial-in service capability.

**[Responsible]{.underline}**

**[Component:]{.underline}** Deputy Commissioner for Operations, Office
of Automation Support, and the Deputy Commissioner for Finances,
Assessment, and Management, Office of Program and Integrity Reviews

**[Report]{.underline}**

**[Frequency:]{.underline}** SSA will report on a semi-annual basis.

\* Putting [Customers First]{.underline} SSA Publication No. 05-10030,
September 1994

**Category 4**

**Description:** IWS/LAN Local Processing Infrastructure Capability. SSA
will report on the IWS/LAN implementation progress. During the
acceptance period, SSA will provide reports on the status of acceptance.
When the national installation begins, SSA will provide a report on the
number and location of LANs installed.

**[Metric:]{.underline}**

a\. During the acquisition phase, SSA will advise higher monitoring
authorities of the progress being made toward contract award by
complying with the special conditions set forth in the DPA (i.e.,
submitting status reports every 6 months, presenting post-bid and
pre-award briefings, etc.).

b\. When the national installation begins, SSA will provide a report on
the number and location of LANs installed by month.

**[The measure is:]{.underline}**

a\. The contract status activity report for IWS/LAN procurement.

b\. The individual office/site LAN installation by month, district
office, state, and cumulative number of LANs installed.

**[The target is:]{.underline}**

a\. Report produced every 6 months. Acquisitions completed within the
estimates established-by the Office of Acquisition and Grants.

b\. Seventy-five (75) LAN installations per month and all installations
completed in twenty-nine (29) months from first field installation date.
Targets were set to provide the most responsive and practical timeframes
for implementation of the IWS/LAN infrastructure for baseline automation
benefits and distributed data processing application platforms. The time
table also facilitates SSA\'s Customer Service Pledge objectives by
providing advanced systems to speed SSA service, such as the appointment
process, the SSA card replacement request, new SSN card requests, and
information on claims, programs, client data, and other SSA business
functions.

**[Data Source:]{.underline}**

a\. Prior to contract award, project status reports will be generated,
as required, by the contracting officer.

b\. After contract award, project status reports will be generated
monthly to the office of the Deputy Commissioner for Systems.

**[Responsible]{.underline}**

**[Component:]{.underline}** Deputy Commissioner, Office of Systems,
Office of Telecommunications; office of Finance, Assessment and
Management, Office of Acquisition and Grants; and Deputy Commissioner,
Office of operations, Office of Automation Support.

**[Report]{.underline}**

**[Frequency:]{.underline}** Semi-annually or as requested

**Category 5**

**Description:** Existing terminal redeployment and phase out. As the
IWS/LAN is installed across all SSA regions, their Terminal Acquisition
Project (TAP) terminals (i.e., IBM 3270) will be redeployed to other SSA
field offices that are not receiving IWS/LAN until late in the
installation schedule. When these relocation requirements are met, SSA
intends to release to Government excess all remaining TAP devices that
are removed from IWS/LAN installed offices.

**[Metric:]{.underline}**

The IWS/LAN installed office's existing TAP terminal base will be
redeployed to SSA field offices that have had outstanding requirements
for these resources. Once redeployment is complete, SSA will also report
on the release to Government excess of all TAP devices that are removed
from IWS/LAN installed offices..

> **[The measure is:]{.underline}** The SSA office (City/State), IWS/LAN
> install date, number of TAP terminals, number of terminals redeployed,
> number of terminals excessed, and date of the activity.

**[The target is:]{.underline}** Average redeployment of 615 TAP
workstations per month for 12 months starting at first IWS/LAN phase I
field office installation month. Average release of 820 TAP workstations
per month over a 48 month period starting at first IWS/LAN phase I field
office installation month.\*

**[Data Source:]{.underline}**

SSA will report on the number and location of redeployed and excessed
TAP devices. This report is a product of our existing property
management and maintenance contract support system.

**[Responsible]{.underline}**

**[Component:]{.underline}** Deputy Commissioner, Office of Systems,
Office of Telecommunications, Division of Integrated Telecommunication
Management

**[Report]{.underline}**

**[Frequency:]{.underline}** This report will be provided on a
semi-annual basis.

\* Metric assumes the planned ratio of workstations per employee is
implemented.

**Category 6**

**Description:** IWS/LAN contract cost and pricing details will be
provided. SSA will also report on any technology substitution provisions
or activity over the contract life.

**[Metrics:]{.underline}**

a\. At contract award, SSA will provide GSA with all pricing and cost
material for the new contract. SSA will be available to further discuss
in detail with GSA any cost provisions that GSA would want SSA to report
on over the life of the contract.

b\. SSA will provide GSA with the contract\'s technology substitution
language and will report on any technology substitution proposals that
are received and evaluated by SSA.

**[The measure is:]{.underline}**

a\. At contract award, a list of pricing and cost material for the new
contract.

b\. A report on any technology substitution proposals received by the
contracting officer.

**[The target is:]{.underline}** Report pricing structure at award of
contract. Report technology substitutions and associated pricing within
60 days of acceptance of proposals.

**[Data Source:]{.underline}**

a\. Office of Acquisition and Grants contract award.

b\. Office of Acquisition and Grants contract award.

**[Responsible]{.underline}**

**[Component:]{.underline}** Deputy Commissioner for Finance,
Assessments, and Management, Office of Acquisition & Grants

**[Report]{.underline}**

**[Frequency:]{.underline}** Frequency of report will be semi-annual.

Appendix C---Performance Measurement Legislation

This appendix provides summaries of the performance-based management
requirements from:

-   The Clinger-Cohen Act of 1996

-   Federal Acquisition Streamlining Act (FASA) of 1994

-   Government Performance Results Act (GPRA) of 1993

The Clinger-Cohen Act of 1996

(Also known as Information Technology Management Reform Act)

The Clinger-Cohen Act focuses on capital planning and performance-based
measurement. Agencies will be required to integrate their IT investment
plans and performance measures into the budget process.

Requires Director of OMB to:

-   Report to congress at budget submission on:\
    \
    (1) Net program performance benefits achieved as the result of major
    capital investments made by executive agencies in information
    systems, and\
    \
    (2) How benefits relate to the accomplishments of the goals of the
    executive agencies.

-   Compare the performance of the use of IT by executive agencies and
    disseminate to the heads of the executive agencies

-   Evaluate the information resources management practices of executive
    agencies related to the performance and results of IT investments
    made by Executive Agencies, and take appropriate steps to enforce
    accountability. Specific actions taken by the Director may include:

> \(1\) Recommending a reduction or an increase in any amount for
> information resources proposed by the head of the executive agency for
> the budget submitted.
>
> \(2\) Reducing or otherwise adjusting apportionments and
> reapportionments of appropriations for information resources.
>
> \(3\) Using other authorized administrative controls over
> appropriations to restrict the availability of resources.
>
> \(4\) Designate an executive agent for the executive agency to
> contract with the private sector for the performance of information
> resources management or the acquisition of information technology.

Requires Each Executive Agency Head to:

-   Establish a process to select, manage and evaluate the results of IT
    investments

-   Submit annual reports on progress in achieving goals with budget
    submission

-   Link IT performance measures to agency programs

-   Revise mission-related processes before making significant IT
    investments

Government Performance And Results Act (GPRA)

This provides for the establishment, testing, and evaluation of
strategic planning and performance measurement in the Federal
Government, and for other purposes. Major features include strategic
planning and the annual submission of the performance plans and reports
for fiscal year (FY) 1999.

**Strategic Planning**

Calls for submission by each agency to OMB by September 30, 1997 a
strategic plan (developed in consultation with Congress) for program
activities covering five years (and updated at least every three years),
including:

-   A comprehensive mission statement

-   Outcome-related goals and objectives

-   A description on how performance goals (see below) relate

-   An identification of key external factors

-   A description of program evaluations used

**Annual Performance Plans and Reports**

Calls for OMB to require, from each agency beginning with FY 99, an
annual performance plan covering the activities listed in the agency\'s
budget, including:

-   The establishment of performance goals for each program

-   Expressed goals in objective, quantifiable, and measurable form

-   A description of resources and activities needed to meet the goals

-   The establishment of performance indicators

-   A basis for comparing results with goals

-   Provides for alternative to objective and quantifiable form

Also, no later than March 31, 2000, each agency is to submit a report on
program performance for the previous fiscal year. The required
information may be included in the annual financial statements.

Federal Acquisition Streamlining Act (FASA)

Title V of this act applies to both defense and civilian agencies. The
Secretary of Defense is charged with implementing performance-based
management for defense acquisition programs; and the Administrator of
the Office of Federal Procurement Policy (OFPP) is responsible for
ensuring implementation by civilian programs.

> **Title V establishes a government policy:** \"that agencies should
> achieve, on average, 90 percent of the cost and schedule goals
> established for major and non-major acquisition programs without
> reducing the performance or capabilities of the items being
> acquired.\"
>
> **Title V:**

-   Requires agency heads to approve or define the cost, performance,
    and schedule goals for major acquisition programs. (Chief Financial
    Officers are required to evaluate the cost goals proposed for each
    major acquisition program.)

-   When a program fails significantly to meet established schedule,
    cost or performance goals, Title V directs the agency head to
    determine whether there is a continuing need for the program; and
    identify suitable actions to be taken, including termination.

-   Requires the Administrator of OFPP to report annually to Congress on
    agencies\' progress in implementing Title V.

-   Complements the required performance-based management system with a
    requirement that civilian agencies develop a \"results oriented
    acquisition process\" for determining whether the acquisition of
    non-commercial items satisfies agency program needs.

-   Requires the OMB Deputy Director for Management to develop an
    enhanced performance incentive system for the acquisition work force
    keyed to the achievement of the acquisition program goals.

Appendix D---Omb and GAO

> Investment Factors

OMB and GAO identified the following Overall Risk and Return
Factors.[^26]

The Risk Factors

**Investment Size** - How large is the proposed technology investment,
especially in comparison to the overall IT budget?

**Project Longevity** - Do projects apply a modular approach that
combines controlled systems development with rapid prototyping
techniques? Are projects as narrow in scope and brief in duration as
possible to reduce risk by identifying problems early and focusing on
project versus realized results. Modular approaches are considered less
risky.

**Technical Risk** - How will the proposed technology be integrated into
existing systems? Will proposed investments take advantage of
commercial-off-the-shelf software and systems? How will the complexity
of the systems architecture and software design affect the project?

The Return Factors

**Business Impact or Mission Effectiveness** - How will the investment
contribute toward improvement organization performance in specific
outcome-oriented terms?

**Customer Needs** - How well does the investment address identified
internal and/or external customer needs/demands for increased service
quality and timeliness or reductions in costs?

**Return on Investment** - Are the return on investment figures using
benefit-cost analysis thresholds reliable and technically sound?

**Organizational Impact** - How broadly will the technology investment
affect the organization (i.e., the number offices, users, work
processes, and other systems)?

**Expected Improvement** - Is the proposed investment being used to
support, maintain, or enhance existing operations systems and processes
(tactical) or designed to improve future capability (strategic)? Are any
projects required by law, court ruling, Presidential directive, etc.? Is
the project required to maintain critical operations\--payroll,
beneficiary checks, human safety, etc.\--at a minimal operating level?
What is the expected magnitude of the performance improvement expected
from the technology investment?

Appendix E---Recommended Reading List

This list includes IT and non-IT performance measures. Because the
principles are common, much can be learned from non-IT performance
measures.

Bishop, Yoes and Hamilton, *Performance Measurement for Information
Systems: Industry Perspectives*, University of Houston-Clear Lake,
October 30, 1992

Brynjolfsson, "The Productivity Paradox of Information Technology,"
*Communications of the ACM*, December 1993 (p.67)

*Information System Performance Measurement: A Method for Enhancing
Accountability in State Information Systems*, Joint Committee on
Information Technology Resources, The Florida State Legislature, January
1994

*Criteria for Developing Performance Measurement Systems in the Public
Sector,* Department of the Treasury, September 1994

Katz, "Measuring Technology\'s Business Value," *Information Systems
Management,* Winter 1993 (p. 33)

Kaplan and Norton, "The Balanced Scorecard - Measures That Drive
Performance," *Harvard Business Review,* January-February 1992 (p.71)

Kaplan and Norton, "Putting The Balanced Scorecard to Work," *Harvard
Business Review*, September-October 1993 (p.134)

Kaplan and Norton, "Using the Balanced Scorecard as a Strategic
Management System," *Harvard Business Review*, January-February 1996
(p.75)

Kaplan and Norton, *The Balanced Scorecard Translating Strategy Into
Action,* Harvard Business School Press, Boston, Massachusetts, 1996

Keen, *Shaping the Future: Business Design through Information
Technology,* Cambridge: Harvard University Press

Meyer, "How the Right Measures Help Teams Excel," *Harvard Business
Review*, May-June 1994 (p.95)

Nay and Kay, *Government Oversight and Evaluability Assessment*,
Lexington Books, 1982

National Academy of Public Administration, "Information Management
Performance Measures - Developing Performance Measures and Management
Controls for Migration systems, Data Standards, and Process
Improvement," Report by a Panel of NAPA for the US Department of
Defense, January 1996 (202-347-3190) Cost: approximately \$13

Parker and Benson, *Information Economics (Linking Business Performance
to Information Technology),* Prentice Hall, 1988

*Performance Measurement Guide*, Department of Treasury, Financial
Management Service, November 1993 (For copies: contact (202) 874-7074)

Porter, *Competitive Advantage: Creating and Sustaining Superior
Performance,* The Free Press, 1985

*A Review of the Oregon and Texas Experience in Building Performance
Measurement and Reporting Systems - Data Selection, Collection and
Reporting*, National Institute for Literacy, January 1995,
(202-632-1500)

*Measuring Business Value of Information Technologies,* International
Center for Information Technologies, ICIT Press, 1988

Remenyi, Money, and Twite, *A Guide to Measuring and Managing IT
Benefits*, Second Edition, NCC Blackwell Limited, Oxford, England, 1992

[^1]: . Robert S. Kaplan and David P. Norton, *The Balanced Scorecard:
    Translating Strategy into Action*, p. 31.

[^2]: Adapted from Robert S. Kaplan and David P. Norton, "Putting the
    Balanced Scorecard to Work," *Harvard Business Review,*
    September-October 1993, p. 139.

[^3]: . Adapted from Kaplan and Norton, p. 135-136.

[^4]: . Kaplan and Norton, p. 138.

[^5]: Robert S. Kaplan and David P. Norton, *The Balanced Scorecard:
    Translating Strategy into Action, p. 10.*

[^6]: . Adolph I. Katz, "Measuring Technology's Business Value,"
    *Information Systems Management*, Winter 1993

[^7]: . Katz, p. 79.

[^8]: Robert S. Kaplan and David P. Norton, *The Balanced Scorecard:
    Translating Strategy into Action, p. 9.*

[^9]: . Marilyn M. Parker and Robert J. Benson, *Information Economics
    (Linking Business Performance to Information Technology)*, Prentice
    Hall, 1988

[^10]: . Parker and Benson, p. 26.

[^11]: . Parker and Benson, p. 39.

[^12]: . Parker and Benson, p. 146-166.

[^13]: 13\. The table separates value and risk scores because value and
    risks need to be evaluated separately. This differs from the
    authors, Parker and Benson. They make a logical error by combining
    value and risk domain factor scores to compute a total project
    score. Their approach combines two factors that are unrelated, like
    adding apples to oranges. For example, the score for a project that
    has both high value and risk scores may be lower than the score for
    a project that has both low value and risk scores.

[^14]: 14\. Adapted from Parker and Benson, p. 226.

[^15]: . Adapted from The Department of Treasury, *Criteria for
    Developing Performance Measurement Systems in the Public Sector*,
    Office of Planning and Management Analysis Working Paper Series,
    September 1994

[^16]: . Adapted from *IRM Performance Measures and the GPRA*, Central
    Michigan University, 1996

[^17]: . To obtain a printed copy, contact GSA's IT Management Practices
    Division at (202) 501-1332. To obtain a digital copy, visit
    http://www.itpolicy.gsa.gov/mkm/bpr/gbpr.htm.

[^18]: . Adapted from Department of Treasury, *Performance Measurement
    Guide*, 1993, p. 21.

[^19]: . Russell M. Linden, *Seamless Government: A Practical Guide to
    Re-Engineering in the Public Sector,* Jossey-Bass Publishers, 1994,
    p. 14

[^20]: . Selections from Parker and Benson, p. 101 and Daniel R. Perley,
    Migrating to Open Systems: Taming the Tiger, McGraw-Hill, 1993

[^21]: ?. Remenyi, Money, and Twite, *A Guide to Measuring and Managing
    IT Benefits*, Second Edition, NCC Blackwell Limited, Oxford,
    England, p. 19.

[^22]: . Parker and Benson

[^23]: . Parker and Benson, p. 144 - 166.

[^24]: . *Evaluating Information Technology Investments: A Practical
    Guide*, Version 1.0, Office of Management Budget, November 1995, p.
    8.

[^25]: 26\. State of Texas, Office of the State Auditor*, A Guide to
    Measure the Performance of Information Systems*, April 1991

[^26]: 27*. Evaluating Information Technology Investments: A Practical
    Guide*, Version 1, OMB, Nov. 1995, p. 8.