![](media/image1.wmf){width="8.149305555555555in"
height="10.678472222222222in"}

**The Federal Enterprise Architecture Program Management Office**

![](media/image2.png){width="1.4270833333333333in"
height="0.525in"}***The Performance Reference Model Version 1.0:***

***A Standardized Approach to IT Performance***

**June 2003**

**Volume II: How to Use the PRM**

# Table of Contents {#table-of-contents .Heading}

**Table of Contents [ii](#table-of-contents)**

1\. A Process for Improvement [3](#a-process-for-improvement)

A SUGGESTED PROCESS TO USE THE PRM [3](#__RefHeading___Toc42486829)

PRM PHASE I -- ALIGN WITH THE PRM [5](#__RefHeading___Toc42486830)

PRM Phase I.1 - Determine Line of Sight from Outcomes to Inputs
[5](#prm-phase-i.1---determine-line-of-sight-from-outcomes-to-inputs)

PRM Phase I.2 - Identify and Define PRM Measurement Indicators
[8](#prm-phase-i.2---identify-and-define-prm-measurement-indicators)

PRM PHASE II - EXPLORE AND DEFINE IMPROVEMENTS
[12](#__RefHeading___Toc42486833)

PRM Phase II.1 - Conduct Baseline Analysis
[12](#prm-phase-ii.1---conduct-baseline-analysis)

PRM Phase II.2 - Set Improvement Targets
[14](#prm-phase-ii.2---set-improvement-targets)

PRM Phase II.3 - Identify, Select, and Propose Improvements
[15](#prm-phase-ii.3---identify-select-and-propose-improvements)

PRM PHASE III - MEASURE PROGRESS [17](#__RefHeading___Toc42486837)

PRM Phase III.1 - Implement improvements
[17](#prm-phase-iii.1---implement-improvements)

PRM Phase III.2 - Track progress toward improvement targets
[18](#prm-phase-iii.2---track-progress-toward-improvement-targets)

PRM PHASE IV - USE PERFORMANCE INFORMATION
[19](#__RefHeading___Toc42486840)

PRM Phase IV.1 - Evaluate progress
[19](#prm-phase-iv.1---evaluate-progress)

PRM Phase IV.2 - Inform IT project and program management
[20](#prm-phase-iv.2---inform-it-project-and-program-management)

2\. Using the PRM Within Existing Management Processes
[22](#using-the-prm-within-existing-management-processes)

Budget [22](#budget)

Government Performance and Results Act
[23](#government-performance-and-results-act)

Program Assessment Rating Tool [23](#program-assessment-rating-tool)

IT Capital Planning and Investment Control
[23](#it-capital-planning-and-investment-control)

Enterprise Architecture [24](#enterprise-architecture)

Appendix A: How to Use the PRM to Complete the OMB Exhibit 300
[27](#appendix-a-how-to-use-the-prm-to-complete-the-omb-exhibit-300)

A-11 REQUIRES AGENCIES TO USE THE PRM FOR MAJOR IT INITIATIVES THAT ARE
DME IN FY 2005 [27](#__RefHeading___Toc42486850)

AGENCIES CAN EASILY INCORPORATE THE PRM INTO EXHIBIT 300 I.C, TABLE 2
[29](#__RefHeading___Toc42486851)

Examples of IT Investments Using the PRM to Complete Section I.C, Table
2 [31](#__RefHeading___Toc42486852)

![](media/image3.wmf){width="7.768055555555556in"
height="4.434722222222222in"}

# 1. A Process for Improvement {#a-process-for-improvement .unnumbered}

This section provides an overview of a suggested process to use the PRM.
Details of Version 1.0 of the PRM are provided in *Volume I: Version 1.0
Release Document*.

[]{#__RefHeading___Toc42486829 .anchor}A SUGGESTED PROCESS TO USE THE
PRM

To help agencies use the PRM, the FEA-PMO is suggesting a high-level
process. This process has been identified through examining best
practices and builds from existing management processes that are
relevant to the PRM. This process consists of four phases that follow
the PRM from developing Operationalized Measurement Indicators through
measuring progress and making more informed management decisions. Stated
otherwise, the suggested process to use the PRM focuses heavily on its
primary purpose to develop IT performance information that can be used
to improve decision-making and performance. As with all aspects of the
PRM, this suggested process will be refined and improved as lessons
learned emerge.

The suggested PRM process described below follows the "flow" of
PRM-related performance information through the IT lifecycle. However,
the suggested process also acknowledges that IT initiatives must be
developed in a business-driven context to be truly successful and
accordingly identifies intersections with other processes, such as GPRA,
that are relevant when using IT to improve performance.

This process is not meant to replace any existing federal-level or
agency processes. Agencies are encouraged to use the PRM and the
information it produces in a way that makes sense for their specific
environment. Figure 1 on the following page shows the suggested process
phases and steps to use the PRM throughout the IT lifecycle.

FIGURE 1: SUGGESTED PROCESS PHASES AND STEPS TO USE THE PRM

![](media/image4.wmf){width="6.445138888888889in"
height="4.2756944444444445in"}

Throughout this suggested process, the degree of cross-agency
collaboration necessary will depend on how much collaboration the
specific environment requires. For example, an agency-specific IT
initiative may need to collaborate with other agencies minimally or not
at all. However, a cross-agency IT initiative may require significant
collaboration to define and measure what are the shared priorities that
the IT will help these organizations achieve.

In practice, the application of the PRM process to any specific IT
initiative will differ based on many factors, including:

![](media/image5.png){width="0.125in" height="0.125in"} The degree of
improvement that is needed;

![](media/image5.png){width="0.125in" height="0.125in"} Whether a new IT
initiative is being developed or an existing one is being modernized or
enhanced;

![](media/image5.png){width="0.125in" height="0.125in"} Whether one's
frame of reference is that of an IT project manager, a program manager,
an agency CIO or CFO, a Line of Business Owner, an OMB examiner, or
other decision-maker; and

![](media/image5.png){width="0.125in" height="0.125in"} What other
existing management initiatives, processes, and data collection efforts
are currently underway that can be leveraged.

As with any successful improvement initiative, several best practices
are critical to PRM implementation. These include:

![](media/image5.png){width="0.125in" height="0.125in"} Top leadership
support and commitment to change;

![](media/image5.png){width="0.125in" height="0.125in"} Ongoing
involvement and buy-in from front-line employees;

![](media/image5.png){width="0.125in" height="0.125in"} Identification
and engagement of other critical business partners and stakeholders from
the very beginning;

![](media/image5.png){width="0.125in" height="0.125in"} A diverse team
with a broad perspective and representation to shepherd the improvements
needed;

![](media/image5.png){width="0.125in" height="0.125in"} A strong and
ongoing partnership between business and IT communities within the
agency; and

![](media/image5.png){width="0.125in" height="0.125in"} Managing change
and communicating priorities and progress both internally and
externally.

[]{#__RefHeading___Toc42486830 .anchor}PRM PHASE I -- ALIGN WITH THE PRM

This section provides information on how agencies can align with the
PRM.

### PRM Phase I.1 - Determine Line of Sight from Outcomes to Inputs {#prm-phase-i.1---determine-line-of-sight-from-outcomes-to-inputs .unnumbered}

A critical first step in aligning with the PRM is to understand the
"line of sight" from the IT initiative through outputs, then to
outcomes. This places the IT initiative in its proper context when
assessing a program's or agency's overall progress. This is in part
because IT initiatives generally contribute to outputs and outcomes, but
rarely achieve them on their own. This step can consist of at least two
key tasks:

[PRM Phase I.1.A - Understand context and performance
drivers]{.underline}

Before attempting to draw the line of sight, additional background
information may be helpful. The type of information will vary depending
on the specific agency and/or IT initiative, as well on what
perspectives (e.g. program officials, IT project manager, budget, EA)
are represented when completing this task. Key context and background
information may include:

![](media/image5.png){width="0.125in" height="0.125in"} Relevant
legislative and Congressional mandates for the IT initiative and/or the
program it supports;

![](media/image5.png){width="0.125in" height="0.125in"} Relevant BRM
Lines of Business and Sub-functions the IT initiative supports;

![](media/image5.png){width="0.125in" height="0.125in"} Relevant agency
business lines and/or functions the IT initiative supports;

![](media/image5.png){width="0.125in" height="0.125in"} Agencies,
bureaus, and/or programs that align with those Sub-functions or
functions;

![](media/image5.png){width="0.125in" height="0.125in"} Applicable
strategic and annuals goals for these entities;

![](media/image5.png){width="0.125in" height="0.125in"} Applicable
measures or indicators for these goals; and

![](media/image5.png){width="0.125in" height="0.125in"} Existing
performance gaps or constraints that need to be addressed (GAO, IG,
Congress, OMB, PART Assessments, etc).

As discussed in *Volume I: Version 1.0 Release Document*, agencies
should leverage the work already done to align their IT initiatives to
the Business Reference Model. Determining what Line of Business and
Sub-function the IT initiatives supports is the starting point to
identify Operationalized Measurement Indicators in both the Mission and
Business Results and Processes and Activities Measurement Areas.

After agencies have provided their FY 2005 Exhibit 300s to OMB, the
Federal Enterprise Architecture Management System (FEAMS) can be a
valuable tool for agencies to identify other federal agencies that may
conduct the same or similar Lines of Business and Sub-Functions. FEAMS
is a web-based tool that will display how major IT initiatives can be
characterized through each FEA reference model. Selected agency and OMB
officials will have access to FEAMS during relevant times of the year to
identify collaboration opportunities with other federal agencies.

[PRM Phase I.1.B - Understand what contributes to performance and
how]{.underline}

The "line of sight" for the IT initiative consists of identifying its
cause and effect relationship with other inputs, then outputs, and
outcomes. This relationship should be identified by starting with the
desired outcome, then cascading down to outputs and
inputs---specifically the IT initiative. The figure below suggests key
considerations to understand WHAT contributes to performance---and HOW
inputs contribute to outputs, and by extension, outcomes.

Generally, more informative Operationalized Measurement Indicators
result when the line of sight is more detailed. This information can be
invaluable later in the IT lifecycle when agencies conduct Post
Implementation Reviews to determine whether the initiative was
successful.

Figure 2 below graphically depicts the line of sight and suggests some
simple questions that should be addressed to understand at least at a
high-level how an IT initiative contributes to outputs and outcomes.

![](media/image6.wmf){width="7.652777777777778in"
height="3.513888888888889in"}FIGURE 2: KEY QUESTIONS TO DETERMINE LINE
OF SIGHT

This type of thinking is not new. Program logic models, value chain
analysis, and theory of constraints analysis all provide the basis to
identify these relationships. At its essence the line of sight consists
of thinking through two aspects critical to identifying adequate
Measurement Indicators (1) WHAT is in the line of sight and (2) HOW does
each element in the line of sight relate?

Key questions to determine what is in the line of sight include:

![](media/image5.png){width="0.125in" height="0.125in"} What inputs (for
the PRM an IT initiative) contribute to what processes?

![](media/image5.png){width="0.125in" height="0.125in"} What customers
receive the products and services those processes produce?

![](media/image5.png){width="0.125in" height="0.125in"} What mission or
business results do the outputs of those processes contribute to or
influence?

Key questions to determine how each element in the line of sight relates
include:

![](media/image5.png){width="0.12569444444444444in"
height="0.12569444444444444in"} How does the IT initiative contribute to
processes and activities (what capabilities does it actually provide)?

![](media/image5.png){width="0.12569444444444444in"
height="0.12569444444444444in"} How do the processes and activities
impact customers and contribute to mission and business results?

It is important to remember that the line of sight often will
over-simply the true relationship between an IT initiative and the
processes, customers, and business results it supports. These
relationships will rarely be direct "cause and effect" and more often an
association or perceived relationship. As such the point of the line of
sight is not to portray that an IT initiative is completely responsible
for improvements in processes and activities. Further, the line of sight
does not suggest that the outputs from a process or activity are the
only influences on customer results or mission and business results.

However, the line of sight is a valuable exercise that both IT project
managers and key business or program stakeholders should participate in.
Drawing the line of sight will more clearly reveal what exactly the IT
initiative will provide---and what it will not provide. This will yield
important insight to leverage when using the PRM to identify Measurement
Indicators within each required Measurement Area.

In summary, the line of sight uses the PRM structure to identify what
the IT initiative does, how it does it, and how it can be measured
through the PRM. Figure 3 on the following page shows an example line of
sight using the IRS Free Filing E-Gov Initiative.

FIGURE 3: EXAMPLE LINE OF SIGHT ANALYSIS FOR IRS FREE FILING

![](media/image7.wmf){width="7.572222222222222in"
height="4.448611111111111in"}

### PRM Phase I.2 - Identify and Define PRM Measurement Indicators {#prm-phase-i.2---identify-and-define-prm-measurement-indicators .unnumbered}

Against the backdrop of the information identified in I.1, PRM
Measurement Indicators can now be chosen. This step can consist of at
least three key tasks:

[PRM Phase I.2.A - Determine and "operationalize" possible PRM
Measurement Indicators]{.underline}

The information learned about the context and line of sight should be
used to help determine which PRM indicators could be used. During the FY
2005 budget formulation process OMB Circular A-11 requires that at least
one Measurement Indicator in four Measurement Areas be chosen. These
Areas are (1) Mission and Business Results, (2) Customer Results, (3)
Processes and Activities, and (4) Technology. In each of these Areas the
inventory of Indicators should be reviewed to determine which
Measurement Indicators could possibly apply. After choosing the possible
Generic Measurement Indicators, each will then need to be uniquely
tailored or "operationalized" to reflect the specific environment. See
Appendix A for more guidance on how to complete Table 2 of the Exhibit
300 for IT initiatives seeking DME funding in FY 2005.

For example, an outcome-oriented Indicator in the Mission and Business
Results Area could be operationalized as shown below:

![](media/image8.wmf){width="6.9215277777777775in" height="0.79375in"}

![](media/image9.wmf){width="6.9215277777777775in"
height="0.9104166666666667in"}An output-oriented Indicator in the
Processes and Activities Area could be operationalized as shown below:

![](media/image10.wmf){width="6.9215277777777775in"
height="0.9104166666666667in"}An Indicator in the Technology Area could
be operationalized as shown below:

It is important to note that in the above example, only the Technology
Measurement Indicator is directly attributable to the IT initiative,
AWIPS. The Processes and Activities and Mission and Business Results
Indicators merely describe what AWIPS contributes to. So naturally,
these Indicators would not be defined by an IT project manager. Rather,
they would be created through the programmatic budget and GPRA
processes. The IT project manager would then use the line of sight
analysis to identify that AWIPS contributes to these Indicators and
identify them as part of the line of sight for AWIPS. This requires a
strong partnership between the IT and business communities within an
agency.

The degree of commonality or standardization across agencies' use of the
PRM will depend upon whether the IT initiative is being managed by a
single agency or multiple agencies. All major IT initiatives requesting
DME funding for FY 2005 will be identifying at least one indicator in
the four required Measurement Areas. However, unless OMB, a Managing
Partner, or Line of Business owner dictates collaboration at the
Measurement Category, Generic Measurement Indicator, and Operationalized
Measurement Indicator levels, agencies will independently use the PRM
during FY 2005 budget formulation. Figure 4 depicts how the degree of
cross-agency standardization using the PRM decreases with each level of
granularity of the PRM.

![](media/image11.wmf){width="4.836805555555555in"
height="3.3229166666666665in"}FIGURE 4: DEGREE OF COMMONALITY AND
STANDARDIZATION IN PRM USE ACROSS AGENCIES

FEAMS will again be a valuable resource to examine what Indicators other
agencies with similar Lines of Business, with similar customers, similar
processes, or similar IT initiatives are using.

Identifying possible Customer Results Measurement Indicators that will
not require unwarranted data collection burden can be a challenge. One
accepted way to meet this challenge is to partner with other
organizations. For example, the American Customer Satisfaction Index
(ACSI), a partnership between the University of Michigan and American
Society for Quality, helps federal agencies track annual trends in
customer satisfaction. In its 2002 survey, ACSI asked citizens about a
number of federal agencies and programs and several government Web
sites, which scored higher than the average for private industry and
news and information sites.

The plans and reports that agencies produce to comply with the
Government Paperwork Elimination Act (GPEA) can be useful resources as
well. These plans identify priority business processes that have been or
can be improved through technology. The plans also should identify the
customers of each of these processes. This documentation can help
identify the processes and customers an IT initiative contributes to.

Finally, the pre-determined goals for the IT initiative and the program
it supports should play a significant role when determining which PRM
Measurement Indicators could possibly be used.

[PRM Phase I.2.B - Select Appropriate PRM Measurement
Indicators]{.underline}

According to OMB's guidance on conducting PART assessments, the key to
assessing program effectiveness is measuring the right things.[^1] PART
guidance notes that performance measures should reflect a sense of
priorities, and should reflect both outcomes and outputs.

It is against this backdrop that PRM Measurement Indicators can be
selected. After identifying and operationalizing the range of possible
PRM Indicators, it is then important to apply criteria to both
individual Measurement Indicators and the entire list to ensure that the
appropriate set---or "vital few"---are selected. The exact number of PRM
Indicators will vary depending on many factors, including the current
data available and the size and scope of the IT initiative.

For FY 2005, OMB Circular A-11 requires that at least one
operationalized Measurement Indicator be provided for each DME IT
initiative in the (1) Mission and Business Results, (2) Customer
Results, (3) Processes and Activities, and (4) Technology Measurement
Areas. Agencies may choose to select and use a greater number of
Measurement Indicators as they see fit.

Key criteria for individual Measurement Indicators could include:

![](media/image5.png){width="0.125in" height="0.125in"} Informative -
Would the Indicator help articulate success for the initiative and the
programs it supports? Would the Indicator demonstrate progress towards
goals, closing performance gaps, and achieving critical results?

![](media/image5.png){width="0.125in" height="0.125in"} Feasible -- What
data is currently being collected and available? Would the burden or
cost of collecting new data for the Indicator be reasonable considering
how informative it would be for managers and decision-makers?

Key criteria for the entire list of Measurement Indicators could
include:

![](media/image5.png){width="0.125in" height="0.125in"} Manageable - Is
the entire list of Indicators pared down to the \"vital few\" measures
that can help drive improvement and characterize success?

![](media/image5.png){width="0.125in" height="0.125in"} Complete - Does
the entire list of Indicators collectively provide an accurate and broad
enough \"snapshot\" of performance? Could the list be further cascaded
or supplemented with additional Indicators to provide relevant
information to the public, decision-makers, and managers? Does the list
track progress towards key performance gaps or constraints identified in
I.1.A?

Finally, the point of performance measurement is to reduce
uncertainty---not completely eliminate it. This means that particularly
in the early years of PRM implementation agencies will not have perfect
measures. PRM implementation is not about perfect measures, but better
measures that reduce uncertainty for project managers and key
decision-makers. This is an important point to keep in mind when
deciding which PRM Measurement Indicators to use.

[PRM Phase I.2.C. - Define Selected PRM Operationalized Measurement
Indicators]{.underline}

After applying the criteria in 1.2.B, the selected PRM indicators then
need to be further defined and assessed. This task is critical to
ensuring that data will truly be available to track progress towards the
indicator and that someone has been given the responsibility to collect
this data on a regular basis. Key aspects about each Measurement
Indicator should be considered as shown below:

![](media/image5.png){width="0.125in" height="0.125in"} How exactly will
the Measurement Indicator be calculated?

![](media/image5.png){width="0.125in" height="0.125in"} What are the
data source(s) that will be used?

![](media/image5.png){width="0.125in" height="0.125in"} What other
federal or private sector organizations currently collect similar data
and can that can be leveraged?

![](media/image5.png){width="0.125in" height="0.125in"} When will the
data first be available and then actually collected?

![](media/image5.png){width="0.125in" height="0.125in"} How often will
data be collected thereafter?

![](media/image5.png){width="0.125in" height="0.125in"} Who is
responsible for collecting this data?

[]{#__RefHeading___Toc42486833 .anchor}PRM PHASE II - EXPLORE AND DEFINE
IMPROVEMENTS

######## This section provides information on how agencies can conduct analysis consistent with the PRM framework to drive performance improvement. {#this-section-provides-information-on-how-agencies-can-conduct-analysis-consistent-with-the-prm-framework-to-drive-performance-improvement. .unnumbered}

### PRM Phase II.1 - Conduct Baseline Analysis {#prm-phase-ii.1---conduct-baseline-analysis .unnumbered}

A baseline is a quantifiable point from which to begin an effort and
from which change can be measured and documented. Conducting baseline
analysis involves setting a reference point from which improvement
targets will be set and progress measured. Baseline analysis also
includes assessing the current capabilities that are associated with the
initiative. This is typically described as an agency's "status quo" or
"current state" environment.

This step can consist of at least two key tasks:

[PRM Phase II.1.A - Baseline performance against chosen PRM Measurement
Indicators]{.underline}

For each PRM Indicator identified in Phase I it is important to set a
baseline. This can be done using current data or previous data that
coincides with when the initiative began. There are multiple ways in
which this baseline performance can be identified:

![](media/image5.png){width="0.125in" height="0.125in"} Assess current
performance against chosen PRM Indicators. For an IT initiative, the
performance of the existing legacy IT system (if there is one) can be
identified. The initiative's current contribution to outputs and
outcomes can then also be assessed.

![](media/image5.png){width="0.125in" height="0.125in"} Benchmark
current performance of other, similar organizations. If assessing
current performance of the initiative and/or agency is too
time-consuming or not possible, other initiatives, agencies, or private
sector organizations with similar characteristics can be used to
approximate what current performance might be.

![](media/image5.png){width="0.125in" height="0.125in"} Use initial
requirements. For new programs or initiatives, the requirements for the
new program or initiative can be used as a baseline.

Table 2 of the Exhibit 300 will require agencies to discuss their
"baseline" for each PRM Measurement Indicator they report.

[PRM Phase II.1.B - Baseline current processes and
capabilities]{.underline}

In addition to identifying baselines for performance, it can be useful
to baseline current processes and capabilities. This can begin by
building from the line of sight identified in Phase I. More detailed
baselining of processes and capabilities is somewhat helpful context to
baseline performance but imperative when setting improvement targets and
identifying improvement strategies. This process can also identify
in-process Measurement Indicators that may be useful to track in
addition to end-process Measurement Indicators that agencies might
report to OMB in the Exhibit 300. Figure 5 on the following page shows
how the line of sight identified in Phase I can be expanded.

FIGURE 15 EXPANDED LINE OF SIGHT ANALYSIS TO SUPPORT PROCESS BASELINES

![](media/image12.wmf){width="7.129166666666666in"
height="4.295833333333333in"}

To the extent that similar sources of information will be used while
baselining performance, those sources should also be used to baseline
current processes and capabilities.

Processes can also be baselined through process-mapping or
information-flow analysis. Capabilities can be baselined through
conducting inventories of the existing people, technology, and other
fixed assets that contribute to the relevant outputs and outcomes. In
addition to simply identifying these capabilities, specific attributes
can also be assessed---such as the skills and competencies of key staff.

A number of existing disciplines suggest methodologies to baseline
processes and capabilities. These include Business Process
Re-engineering and the Theory of Constraints. The Theory of Constraints,
for example, focuses on identifying the existing aspect of the process
or capabilities---in other words a "constraint"---that if changed or
removed would result in the most significant improvement in
performance.[^2]

Importantly, the integration of the FEA reference models becomes crucial
during this step. For example, baselining IT capabilities against the
SRM and TRM will provide important information. This integration is
critical when seeking to use FEAMS to identify other federal agencies
that can serve as collaborative partners, for example to borrow or
re-use service components.

### PRM Phase II.2 - Set Improvement Targets {#prm-phase-ii.2---set-improvement-targets .unnumbered}

Once the relevant baselines are established, the next step is to use
that context along with some additional information to set performance
targets for each PRM Measurement Indicator identified. Performance
targets are quantifiable estimates or expected results bound to a given
time period. It is against these targets that performance improvement
will be measured. Stated otherwise, target setting provides the road map
for performance improvement. Successfully setting improvement targets
requires taking a short and long-term perspective that allows annual
progress to be assessed in the context of a longer-term vision.

This step can consist of at least two key tasks:

######### [PRM Phase II.2.A - Collect relevant data]{.underline} {#prm-phase-ii.2.a---collect-relevant-data .unnumbered}

Improvement targets must be set in the context of the requirements or
change agenda of the Congress, the Administration, and agency. In
addition, customer feedback, baselines, historical data, benchmarks, and
a variety of other information may be useful context when setting
targets. Some of this information may have been identified in Phase I,
and may include:

![](media/image5.png){width="0.125in" height="0.125in"} Directives from
Congress, the Administration, and the Agency -- The requirements or
priorities that directly or indirectly affect the initiative. These can
include legislative mandates, other Congressional directives, GAO or
Inspector General recommendations, the OMB scorecard, agency Strategic
and Annual Plans, or any other documents that may suggest improvements
or specific levels of performance that are required.

![](media/image5.png){width="0.125in" height="0.125in"} Customer
Feedback -- What customers say, through a variety of forums, about the
current level of performance and how it could be improved. Targeted
sampling surveys, focus groups, literature, and third-party research are
all ways to solicit customer feedback.

![](media/image5.png){width="0.125in" height="0.125in"} Benchmarking --
The process of rating an organization's processes, products, and
performance against high-performers. Benchmarking involves seeking the
"best-in-class" performers within and outside the organization, then
learning from them what they do well, why they do it, and how they
accomplish it. It is imperative to use standardized criteria (e.g. size,
scope, mission) to identify organizations that may serve as appropriate
benchmark partners. FEAMS can again be a useful resource to help
agencies identify other federal agencies that may serve as suitable
benchmarks.

![](media/image5.png){width="0.125in" height="0.125in"} Baselines and
Trends -- Current and/or historical performance data in the relevant
areas of performance. To some extent, this information may be useful in
projecting the range of possible improvement targets for the future.
This information must be considered in the context of the factors that
contributed to the trends and whether or not those factors will still be
relevant in the future.

[PRM Phase II.2.B - Identify long-term and annual targets]{.underline}

Once the relevant data has been collected, performance targets should be
set for each PRM Indicator. GPRA and the PART both emphasize the
importance of long-term and annual improvement targets. More broadly,
long-term and annual targets are important because they provide insight
into what the long-term vision is---as well as how progress towards that
vision will be tracked along the way. Generally, long-term targets are
three to five years away.

Particularly early in the IT lifecycle, targets will be estimates.
General directions of improvement or specific ranges are usually most
realistic and these can be refined during the IT lifecycle as more
information is learned about exactly what the IT initiative will be.

Targets should be set using a collaborative approach driven by
leadership's vision for improvement. Within these parameters,
experience, perspective, and expertise should be leveraged at all levels
of the organization---in particular from front-line employees who have
critical insight into what is truly possible to achieve. This type of
collaborative approach improves the chance that the relevant individuals
will take ownership and be willing to be held accountable for progress.
This approach also improves the chances the targets will be effective in
driving improvements. Generally, useful targets meet some or all of the
following criteria:

![](media/image5.png){width="0.125in" height="0.125in"} Specific, in
that they focus attention on exactly what level of performance is
expected;

![](media/image5.png){width="0.125in" height="0.125in"} Clarifying, in
that they help executives, managers, and front-line employees understand
priorities and set strategies and manage resources accordingly; and

![](media/image5.png){width="0.125in" height="0.125in"} Achievable, yet
challenging, in that they "stretch" the organization to improve but are
realistic enough to be attained. The targets should be realistic but
still spur improvement.

Table 2 of the Exhibit 300 will require agencies to discuss their
"planned improvements to the baseline" for each PRM Measurement
Indicator they report.

### PRM Phase II.3 - Identify, Select, and Propose Improvements {#prm-phase-ii.3---identify-select-and-propose-improvements .unnumbered}

This step involves measuring the magnitude of the "gap" between the
baseline and targets identified in the previous steps. To close this
gap, one or a series of improvement strategies are generally needed.
These strategies may span or affect any or all of the areas of the PRM.
For example, a new or enhanced IT initiative may be needed, staff may
need to be trained in a new skill area, and/or a more sweeping set of
changes to the existing process may be the only way to achieve the
improvement targets set.

This step can consist of at least three key tasks:

[PRM Phase II.3.1 - Identify potential improvement
strategies]{.underline}

There are many approaches to identify the potential range of improvement
strategies. These include simple brainstorming sessions involving key
staff and stakeholders to leveraging the bechmarking analysis performed
in II.2.A to identify how the "best-in-class" achieve such high levels
of performance. Regardless, the strategies identified should not be
constrained (at least initially) and span the entire "line of sight"
identified in Phase I. This should include innovative new thinking about
ways to partner with other federal agencies, state and local
governments, non-profit organization, or other members of the "value
chain" to achieve the performance targets.

Once this universe of potential improvements is identified, the key
aspects of each improvement strategy should be described in the context
of consistently applied and transparent criteria. These criteria can
include feasibility, benefit or value, cost, risk, and timing or payoff
period. This will provide a standard backdrop against which to select an
improvement strategy.

Once again FEAMS can prove valuable during this step. Agencies can use
the tool to identify other agencies that can be partners for
improvement. For example, an agency conducting a similar Line of
Business or Sub-function may have or be developing components within a
Service Domain that would be an efficient strategy to achieve
performance targets.

[PRM Phase II.3.2 - Select desired improvement strategy]{.underline}

There are a variety of ways to select an improvement strategy (or
strategies). Regardless, the standardized criteria identified in the
previous step should be used as the primary means to choose one of the
strategies identified. Ideally, these criteria should be weighted based
on their priority to the organization. Each improvement strategy should
then be "scored" against this set of weighted criteria. Traditional
business case development practices often refer to this as an
"Alternatives Analysis." At any point in this process, the effect of a
desired improvement strategy on the existing environment can be
projected using a modeling or scenario-based software.

The specific strategy selected may determine whether the improvement
targets set earlier are truly feasible. It is critical at this stage to
revisit the improvement targets and ensure that the selected improvement
strategies provide a reasonable chance of achieving the targets.

[PRM Phase II.3.3 - Propose improvement strategy to agency, OMB,
Congress, and other stakeholders]{.underline}

Once the desired improvement strategy has been selected, that strategy
and its benefits, cost, and risks must be approved. Depending on the
specific environment, documents must be completed within an agency,
which then may be shared with OMB, and then ultimately the Congress or
other key stakeholders, such as unions. Though there are a variety of
formal and informal structures through which this can occur, the budget
process is generally where decision-makers assess whether the
improvement strategy is appropriate and will be approved. For example,
GPRA and budget proposals agencies submit to OMB generally provide some
details on what improvements are needed, how much they will cost, and
how the agency will make them.

Improvements that involve major IT initiatives require that an Exhibit
300 be produced and submitted to OMB for approval. The specific
requirements for the Exhibit 300 are outlined each year in OMB Circular
A-11. Appendix C of this release document provides detailed information
about how to use the PRM to complete Section I.C. in the Exhibit 300 for
IT initiatives seeking DME funding in FY 2005.

[]{#__RefHeading___Toc42486837 .anchor}PRM PHASE III - MEASURE PROGRESS

At some point, the planning is complete and approvals have been given.
It is time to begin implementing improvements. As with all the preceding
phases, the exact timeframes for this phase will vary significantly
depending on the magnitude of improvement needed and the size and scope
of the initiative. Nevertheless, key steps within this phase could
include:

### PRM Phase III.1 - Implement improvements {#prm-phase-iii.1---implement-improvements .unnumbered}

Though the time for implementation has arrived, it is still necessary to
follow existing agency-level or OMB processes. These include the IT CPIC
requirements identified in OMB Circulars A-11 and A-130. Within this
framework many agencies have a variety of unique processes and
procedures to meet these requirements. However, key aspects include:

![](media/image5.png){width="0.125in" height="0.125in"} Robust program
and project management planning and monitoring throughout the IT
lifecycle;

![](media/image5.png){width="0.125in" height="0.125in"} Continued
collaboration and involvement of key business stakeholders and customers
as appropriate; and

![](media/image5.png){width="0.125in" height="0.125in"} Continued
monitoring of the external environment to ensure the planned
improvements are still viable given recent developments.

### PRM Phase III.2 - Track progress toward improvement targets {#prm-phase-iii.2---track-progress-toward-improvement-targets .unnumbered}

Within the relevant processes, again usually the IT CPIC process, the
work done in earlier phases to apply the PRM will begin to yield
benefits. The improvement targets set against each relevant PRM
Indicator can now serve as the measuring stick for progress. Progress
towards some input or output Indicators may be useful on a weekly,
monthly, or quarterly basis. Progress towards outcome Indicators may
only be available on an annual or bi-annual basis. Similarly, different
audiences will need to track progress on different timeframes. IT
project managers may wish to see weekly updates while agency CIOs or OMB
will see quarterly or annual updates.

In particular, progress towards PRM Indicators may be useful if the
program the IT initiative supports is subject to PART. More
specifically, OMB examiners and agency officials conducting PART
assessments may find the information gained from tracking progress
towards PRM Measurement Indicators useful. This will be especially true
when answering questions the Strategic Planning and Program Results
sections of the PART assessment tool.

Many E-Government initiatives managed by state governments are tracking
progress towards improvement targets. Some of these have been recognized
by the National Association of State Chief Information Officers
(NASCIO), including:

![](media/image5.png){width="0.125in" height="0.125in"} The Missouri
E-Grants project, which reported an 86 percent reduction in the time to
process grant applications;

![](media/image5.png){width="0.125in" height="0.125in"} The District of
Columbia Business Resource Center, which reported annual cost savings of
\$1.8 million; and

![](media/image5.png){width="0.125in" height="0.125in"} The
Massachusetts Educator Licensure and Recruitment Initiative, which
reported a six-month decrease in the licensing process and \$2.5 million
in additional annual revenue.[^3]

[]{#__RefHeading___Toc42486840 .anchor}PRM PHASE IV - USE PERFORMANCE
INFORMATION

One of the main purposes of the PRM is to produce enhanced performance
information to improve daily and strategic decision-making. Once
collected during Phase III, the performance information facilitated by
the PRM can be used in targeted circumstances to drive more rigorous
analysis to support decision-making. This phase is the most important of
the suggested process to use the PRM.

Key steps within this phase could include:

### PRM Phase IV.1 - Evaluate progress {#prm-phase-iv.1---evaluate-progress .unnumbered}

The information provided by tracking progress towards PRM Indicators in
Phase III will only provide a point-in-time "snapshot" of performance.
Agencies at times will wish to use this information as a starting point
for more detailed analysis. This analysis is often referred to as
program evaluation or root cause analysis. This analysis is also an area
in which agencies generally have weak or non-existent capacity, but is
none-the-less essential to truly improving performance.[^4]

The IT CPIC process' "Evaluate" phase generally includes Post
Implementation Reviews (PIR). These PIRs are detailed assessments of,
among other things, whether the IT initiative actually produced its
intended benefits. For example, an IT initiative may have used the PRM
to project it would contribute to improvements in productivity and
efficiency for the process it supports. After the IT initiative has been
implemented and enough time has passed for it to actually impact the
process, a PIR would seek to determine whether it actually did make the
contribution as planned. This can be assessed through more detailed
performance measurement, user surveys, and comparing process performance
before and after the initiative was implemented.

Using performance information in this way will require a major
transformation towards a performance-driven and evaluation culture.
Improvements in data quality, analytic expertise, and collaborative
partnerships will also need to occur for performance information to be
used to improve performance and management decision-making.[^5]

Eventually, it is envisioned that the PRM framework could be used to
create a "performance dashboard" to monitor the critical areas of
performance---and more importantly identify starting points for further
analysis to determine what strategies need to be changed to achieve even
higher levels of performance.

Figure 6 below shows an example performance dashboard using the PRM.

FIGURE 6: CONCEPTUAL PRM "DASHBOARD" TO EVALUATE PROGRESS

![](media/image13.wmf){width="6.404166666666667in"
height="4.249305555555556in"}

###   {#section .unnumbered}

### PRM Phase IV.2 - Inform IT project and program management {#prm-phase-iv.2---inform-it-project-and-program-management .unnumbered}

Building from the concept of a "performance dashboard," there are times
when either tracking progress towards Measurement Indicators or
conducting more detailed evaluations holds important lessons for project
and program management. To the extent reasonable, this performance
information should be shared with project and program managers.

These managers and at times OMB and Congress may wish to use this
information to enhance their future planning and resource allocation
decisions.

After conducting such evaluations, IT initiatives can begin the PRM
process again. This may involve identifying new PRM Measurement
Indicators in Phase I or identifying new improvement targets or
strategies in Phase II. The important point is to use the information
provided by progress towards PRM Indicators to make better decisions and
as necessary assess and re-assess the path forward.

Based on lessons learned and further outreach within OMB and to key
councils and agencies, the FEA-PMO will refine the suggested PRM process
as needed in PRM Version 2.0.

![](media/image14.wmf){width="7.768055555555556in"
height="4.434722222222222in"}

# 2. Using the PRM Within Existing Management Processes {#using-the-prm-within-existing-management-processes .unnumbered}

This section provides an overview of how the suggested PRM process
described in Section 1 is consistent with and reinforces other existing
management processes.

Though the PRM is being applied to IT, its principles and process steps
require a business and program-driven approach. By extension, this means
that the PRM has relevant intersections with other important management
processes. The "PRM Points" in the previous section identify some of
these intersections. The PRM's most important linkages with other
management processes are highlighted below.

### Budget {#budget .unnumbered}

Key intersections of the PRM and the information it can help produce and
the existing budget formulation and executive process include:

![](media/image5.png){width="0.125in" height="0.125in"} Help articulate
the value of proposed IT improvements to program performance in budget
justification documents;

![](media/image5.png){width="0.125in" height="0.125in"} Help identify
and explore improvement strategies to propose in budget justifications;
and

![](media/image5.png){width="0.125in" height="0.125in"} Use progress
towards Operationalized PRM Measurement Indicators to inform budget
decisions.

### Government Performance and Results Act {#government-performance-and-results-act .unnumbered}

Key intersections of the PRM and the information it can help produce and
the existing GPRA cycle of strategic planning and reporting include:

![](media/image5.png){width="0.125in" height="0.125in"} Goals or
priorities in Strategic Plans can inform and drive the Mission and
Business Results, Customer Results, and at times Process and Activity
Measurement Indicators that agencies "operationalize" through the PRM;

![](media/image5.png){width="0.125in" height="0.125in"} The improvement
strategies identified to achieve performance targets can be informed by
and complement those identified in Accountability Reports; and

![](media/image5.png){width="0.125in" height="0.125in"} Progress towards
relevant PRM Measurement Indicators can be included in Accountability
Reports.

### Program Assessment Rating Tool {#program-assessment-rating-tool .unnumbered}

Key intersections with the PRM and the information it can help produce
and the PART cycle of assessments and making improvements in response to
those assessments include:

![](media/image5.png){width="0.125in" height="0.125in"} Lower scoring
programs can use the PRM to identify Measurement Indicators to help
drive improvement called for in PART Assessments;

![](media/image5.png){width="0.125in" height="0.125in"} Progress towards
operationalized PRM Measurement Indicators can provide additional
information to allow more robust Assessments in the Strategic Planning
and Program Results sections. This is particularly true when determining
the extent to which IT is contributing to improved program performance
and related IT funding decisions;

![](media/image5.png){width="0.125in" height="0.125in"} Assessment
findings can help determine which PRM Measurement Indicators and targets
are used; and

![](media/image5.png){width="0.125in" height="0.125in"} Progress towards
PRM Measurement Indicators in common areas can help OMB and agencies
compare the performance of programs with similar missions. These
programs will be in the same BRM Sub-function, and can be used to help
lower-scoring PART programs learn best practices or collaborate with
higher-scoring programs.

### IT Capital Planning and Investment Control {#it-capital-planning-and-investment-control .unnumbered}

Key intersections of the PRM and the information it can help produce and
the federal and agency-level IT CPIC cycle of Select, Control, and
Evaluate include:

![](media/image5.png){width="0.125in" height="0.125in"} Identify
Measurement Indicators to help select the appropriate IT investments and
request funding for them in Exhibit 300s;

![](media/image5.png){width="0.125in" height="0.125in"} Progress towards
PRM Indicators can inform agency-level IT reviews; and

![](media/image5.png){width="0.125in" height="0.125in"} Progress towards
PRM Indicators can be the starting point for more detailed agency-level
Post Implementation Reviews.

### Enterprise Architecture {#enterprise-architecture .unnumbered}

Key intersections of the PRM and the information it can help produce and
the EA process of baselines, target architectures, and transition or
migration plans include:

![](media/image5.png){width="0.125in" height="0.125in"} Conducting
baseline analysis can inform the completion of Baseline Architectures;
and

![](media/image5.png){width="0.125in" height="0.125in"} The Target
Architecture can help drive and identify improvement strategies chosen
to meet target levels of performance for the appropriate PRM Measurement
Indicators.

Figure 7 on the following page summarizes these intersections.

FIGURE 7: KEY INTERSECTIONS OF THE PRM AND OTHER MANAGEMENT PROCESSES

![](media/image15.wmf){width="9.938194444444445in"
height="6.963888888888889in"}

Based on lessons learned and further outreach within OMB and to key
councils and agencies, the FEA-PMO will refine the suggested PRM process
and its intersections with other processes as needed in PRM Version 2.0

![](media/image16.wmf){width="7.768055555555556in"
height="4.434722222222222in"}

# Appendix A: How to Use the PRM to Complete the OMB Exhibit 300 {#appendix-a-how-to-use-the-prm-to-complete-the-omb-exhibit-300 .unnumbered}

This Appendix provides detailed guidance on how to use the PRM when
completing the Exhibit 300 portion of OMB Circular A-11.

[]{#__RefHeading___Toc42486850 .anchor}A-11 REQUIRES AGENCIES TO USE THE
PRM FOR MAJOR IT INITIATIVES THAT ARE DME IN FY 2005

The FY 2005 OMB Circular A-11, which when released can be found at:
www.whitehouse.gov/omb/circulars/index.html, requires that all major IT
initiatives that are development, modernization, or enhancement (DME)
projects align with the FEA Performance Reference Model (PRM). Section
I.C of the Exhibit 300 required for all major IT initiatives states:

![](media/image17.wmf){width="7.209722222222222in"
height="2.6395833333333334in"}Table 2 from Exhibit 300, Section I.C.
Performance Goals and Measures is shown below. The table consists of
seven columns and includes specific linkages to the PRM.

The definitions for what is expected in each column of the above table
are:

a\) Fiscal Year -- Agencies are to indicate in which fiscal year the
initiative plans to use a particular Measurement Indicator.

b\) Measurement Area -- Agencies should use the PRM to determine the
areas in which they will be measuring performance. For FY 2005, these
must include Mission and Business Results, Customer Results, Processes
and Activities, and Technology.

c\) Measurement Category -- Agencies should use the PRM to determine the
category in which they will be measuring performance. These must
correspond to the appropriate Measurement Area provided in the preceding
column.

d\) Measurement Indicator -- Agencies should use the PRM to determine
the Operationalized Measurement Indicator for which they will be
measuring performance. These must correspond to the appropriate
Measurement Category provided in the preceding column. Each of these
Measurement Indicators should be tailored or "operationalized" to the
agencies' specific environment.

e\) Baseline -- Agencies should provide the relevant baseline
information for the "Operationalized" Indicator provided in the
preceding column. Baselines generally stay the same across fiscal years
unless OMB approves a change to the baseline. See section III of this
release document for more information on conducting baseline analysis.

f\) Planned Improvements to the Baseline -- Agencies should provide the
relevant performance target for the "Operationalized" Measurement
Indicator for the appropriate fiscal year. These may, but will not
always, change over time. See section III of this release document for
more information on setting performance targets.

g\) Actual Results -- Agencies should provide the most current
information they have to demonstrate progress consistent with the
"Operationalized" Measurement Indicator. See section III of this release
document for more information on collecting and reporting actual
performance information.

Again, the only IT investments required to use the PRM for FY 2005 and
beyond are development, modernization, and enhancement initiatives. This
generally will mean that columns "f" and "g" may not be affected in the
early years of the initiative. Consequently, important project-related
milestones and performance measures that will be critical to driving the
early phases of these types of initiatives should be identified under
section I.H.2 of the Exhibit 300 that discusses important
project-related tasks and milestones.

[]{#__RefHeading___Toc42486851 .anchor}AGENCIES CAN EASILY INCORPORATE
THE PRM INTO EXHIBIT 300 I.C, TABLE 2

For each fiscal year, agencies must identify performance information for
their major IT investments in four Measurement Areas of the PRM: (1)
Mission and Business Results, (2) Customer Results, (3) Processes and
Activities, and (4) Technology. Identifying this performance information
is critical so that agencies and OMB can understand the full "line of
sight" from the proposed IT to outputs and outcomes.

![](media/image18.wmf){width="5.249305555555556in"
height="2.9631944444444445in"}

Identifying Mission and Business Results and Customer Results will
generally require coordination with agency budget and planning staff.
Agency Strategic Plans, Performance Plans, and Performance Reports will
be important resources for agencies to use and ensure they are reporting
consistent measures. Generally, the IT investment will be contributing
to improvements in these areas and simply identify measures already in
use. As mentioned in Section I, the primary Business Reference Model
alignment already identified should serve as the starting point for the
Mission and Business Results Measurement Area.

Within each of the four Measurement Areas required for FY 2005, agencies
need to then insert the Measurement Category and Measurement Indicator
in the next two columns to the right. This is shown below. The
Measurement Indicator must be the Operationalized Measurement Indicator
that fits the agency's specific environment. Sections II and III of this
release document provide more information on Operationalized Measurement
Indicators. If agencies believe their project can benefit from using an
Indicator not identified in the PRM, they can use "Other" as needed. The
FEA-PMO will review all "Other" Indicators during OMB passback review
and refine the PRM as appropriate.

![](media/image19.wmf){width="7.6125in" height="4.721527777777778in"}

![](media/image20.wmf){width="7.411111111111111in"
height="4.163194444444445in"}The remaining three columns of the table,
though not explicitly addressed in the PRM, are consistent with Phases
II and III of the suggested PRM process discussed in section III of this
release document. This is where actual data will be provided---and used
to improve decision-making and performance.

[]{#__RefHeading___Toc42486852 .anchor}Examples of IT Investments Using
the PRM to Complete Section I.C, Table 2

Below are three examples of how IT investments could complete Section
I.C of the Exhibit 300. Note that no baseline, planned improvement, or
target data is provided. This is because these examples are designed to
simply illustrate how the PRM structure can be applied to a variety of
investments.

Example Table 2 for IRS Free Filing.

![](media/image21.png){width="7.320138888888889in"
height="4.705555555555556in"}

Example Table 2 for IT initiative supporting a grants process (for FY
2005 only, but should be repeated as needed in future fiscal years).

![](media/image22.png){width="7.330555555555556in"
height="3.8541666666666665in"}

Example Table 2 for Enterprise Architecture initiative (for FY 2005
only, but should be repeated as needed in future fiscal years).

![](media/image23.png){width="7.290277777777778in"
height="4.584722222222222in"}

[^1]: "Completing the Program Assessment Rating Tool (PART) for the FY
    2005 Review Process," Budget Procedures Memorandum No. 861. U.S.
    Office of Management and Budget, May 5, 2003.

[^2]: "The Theory of Constraints and its Thinking Processes," The
    Goldratt Institute. 2001.

[^3]: "High Payoff in Electronic Government: Measuring the Return on
    E-Government Investments," Intergovernmental Advisory Board. March
    31, 2003.

[^4]: "Performance Budgeting: Current Developments and Future
    Prospects," U.S. General Accounting Office. April 1, 2003.

[^5]: "Program Evaluation: An Evaluation Culture and Collaborative
    Partnerships Help Build Agency Capacity," U.S. General Accounting
    Office. May 2003.