**Introduction and Purpose**

*\"Measure a thousand times, and cut once\" --- Ancient Turkish Proverb*

The Office of Information Systems Management (OISM) of the
Administration for Children and Families (ACF) has a continuing interest
in helping States to improve the quality and usefulness of the plans and
studies that support their development of public benefit information
systems. To this end, OISM has published the *Feasibility, Alternatives,
and Cost/Benefit Analysis Guide* (the Guide), has sponsored State
systems planning working groups, has developed a set of model
spreadsheet templates for cost/benefit analysis, and has prepared
cost/benefit training materials.

To augment these efforts, *and especially to respond to suggestions and
comments by State working group participants and reviewers of draft
versions of the Guide,* OISM has developed this *Companion Guide:
Cost/Benefit Analysis Illustrated.* This optional guidance is intended
to respond to the State working group\'s request for:

· Examples of sound cost/benefit studies,

· Clarification of what is required by law and regulation to be
submitted to ACF,

· Differentiation between what may be developed and held in State files
and what is submitted,

· Additional guidance in the area of developing benefits, and

· Further information on reporting actuals against a baseline.

This *Companion* *Guide* is, in a sense, four documents in one.

*Chapter 1: Introduction and Purpose.* This introductory chapter
provides general information to supplement the information presented in
the Cost/Benefit Guide. It also provides a section on developing
benefits.

*Chapter 2: Sample State Documentation.* This chapter provides an
example of a cost/benefit study prepared by a State and maintained in
State files. It simply serves to illustrate, when compared to Chapter 3,
that the information submitted to ACF can be a summary of or extracts
from the State\'s study.

*This section in no way implies a standard, approach, or format that
States must use.*

*Chapter 3: Sample State APD Documentation.* This chapter provides an
example of the part of the Implementation APD which addresses
cost/benefit analysis. This example illustrates the summary or key
information that ACF considers important. Among the most important
factors are:

· Detailed descriptions of benefits, and

· Clear establishment of a baseline for later cost/benefit measurement
and reporting.

This guide does not mandate a format. It does illustrate a sufficient
level of detail for ACF\'s purposes since this section (and the other
chapters) underwent review in ACF\'s program offices.

*Chapter 4: Sample State APDU Documentation.* This chapter is an example
of a cost/benefit measurement report. It is written as though reporting
in the second year of the project described in Chapters 2 and 3. This
clarifies the relationship between the planning stage studies and the
post-implementation measurement and reporting phase.

This *Companion Guide* stresses the importance of *completeness*,
*reasonableness*, and *internal consistency* in a cost/benefit analysis.
It is intended as a companion to the Guide, and not as a replacement.
The Guide remains the definitive ACF reference on the subject of
cost/benefit analysis to support State public benefit information
systems advanced planning.

**Definitions and Clarification of Terms . . . Not Policy**

Cost/benefit analysis for public benefit information system planning
purposes overlaps two distinct philosophic disciplines: economics and
finance[^1]. Each of these disciplines has been (and continues to be)
the subject of tremendous volumes of research by academics and corporate
and government scientists. As a result, and especially when discussions
of cost/benefit analysis refer to these academic roots,

endless nuances and subtleties, complex formulas, and difficult jargon
can become involved.

Practical cost/benefit analysis is, at its heart, based on a simple,
fair, common sense approach to comparing alternatives. However, because
cost/benefit analysis usually involves evaluating monetary costs and
benefits *over time*, some basic rules for handling such \"time
distributed\" monetary values must be applied. Beyond these basic rules
(which are discussed further below), practical cost/benefit analysis
involves no more than basic arithmetic: adding, subtracting, multiplying
and dividing.

The *Feasibility, Alternatives, and Cost/Benefit Analysis Guide* is
based on methodologies and formats in current and common use within the
Federal government. Most of the terminology in the Guide derives from
the usage within Office of Management and Budget Circulars[^2]. OMB
Circulars are the primary vehicle by which the Executive branch
establishes and disseminates Federal agency management policies that
derive from Public Laws.

OMB identifies cost/benefit analysis as the recommended technique to use
in formal, economic analysis of Government programs or projects. In this
context, it is *social net benefits*, and not the benefits and costs to
the government, that should be used as the basis for evaluating
alternative approaches. This means that system benefits are evaluated
not just from a State or Federal perspective, but also from a public
perspective as well. What benefits accrue to citizens?

OMB also defines a less comprehensive approach, called
cost-effectiveness analysis. An alternative is *cost-effective* if it
costs the least of the alternatives, for a given amount of benefits.
Cost-effectiveness analysis is appropriate when the benefits from
competing alternatives are identical, or where a given level of benefits
must be provided as the result of a specific new legislative or policy
decision. Cost-effectiveness analysis is a one-sided cost/benefit
analysis: only costs are analyzed.[^3]

For ACF and most organizations, the standard metric for cost/benefit
analysis of alternatives is net present value (NPV). NPV is simply the
difference between total projected benefits and total projected costs,
and may be referred to as net benefit (or net

cost). Note that NPV may be negative (a net cost) for one alternative,
and positive (a net benefit) for another alternative.

Another term sometimes used is return-on-investment (ROI). (The Guide
uses the term benefit-cost ratio.) ROI is expressed as a percentage or a
ratio --- total projected benefits divided by total projected costs. If
the NPV is positive, ROI will be greater than one and a net benefit
results. A negative NPV means an ROI of less than one (and a net cost).
Because the NPV incorporates a discount factor (to account for the time
value of money, as noted above), sometimes a third metric, internal rate
of return (IRR) is applied. The IRR is the value of the discount factor
that results in an NPV of zero (that is, when projected total benefits
*equal* projected total costs).

Although a positive net present value cannot always be demonstrated for
all alternatives, efforts to measure it can produce useful insights,
even when the monetary values of some benefits cannot be determined.
Enumerating such benefits (e.g., in terms of clients served) can be
helpful in identifying the full range of program effects, even if they
cannot be assigned a dollar value, or monetized.

**On the Importance of Being Consistent**

Perhaps more important than the form and content of a cost/benefit
analysis is its internal consistency.

If a cost/benefit analysis does not demonstrate internal consistency,
nothing else matters; its results are meaningless. Internal consistency
depends entirely on two things: that costs and benefits are assigned
properly, and that the time value of money is considered.

Costs and benefits are assigned properly if nothing significant is left
out, nothing that doesn\'t belong is included, and nothing gets counted
twice. The Guide provides an

outline of typical cost and benefit factors that *may* appear in a
cost/benefit analysis;

of course, any and all other significant factors should be included and
explained as

well. The things that don\'t belong in a cost/benefit analysis are the
things that have

already been paid for, and the gains that have already been
achieved[^4]. Counting

things twice often results when a cost or a benefit is included in two
or more

categories, or when a cost or benefit that is included within a larger
factor is included again, as a separate factor.

Future costs and benefits can be presented in terms of today\'s prices
or in terms of future (inflated) prices. Using today\'s prices (called
constant or real dollars) rather than future prices (called current or
nominal dollars) is the generally preferred approach for planning
purposes, since inflation is so difficult to predict. Whichever approach
is selected *must* be applied **consistently** throughout the analysis.

The time value of money must be accounted for **consistently**
throughout a cost/benefit analysis. Regardless of whether constant or
current dollar values are used, a *discount factor* should be applied to
future costs and benefits. The discount factor reflects a basic
principle of economic analysis --- that money today is worth more than
money in the future. Because the most obvious effect of this economic
principle is on interest rates, the discount factor is sometimes
described as reflecting the impact of interest rates on future costs and
benefits.[^5] The 7% discount rate recommended for use in the Guide is
based on the assumption that States will use constant dollars and that
the systems will generate public benefits.[^6]

Much of what comprises a cost/benefit analysis is presented in the form
of spreadsheets or tables. The information is derived by calculation,
applying formulas or rules to a given set of input values. It should be
possible for an analyst to recreate the basic results of any internally
consistent cost/benefit analysis by starting from only the initial
values, assumptions and formulas that were used in that initial
analysis. OMB puts this point more succinctly in Circular A-94:
\"Measures should be consistent with basic economic principles and
should be replicable.\"

**About the More Sophisticated Techniques**

As noted above, new, advanced methods and techniques for cost/benefit
analysis are constantly being introduced, discussed and refined.
Sometimes these techniques have a practical application to cost/benefit
analysis for systems planning; often they add little in the way of new
information or insights, and sometimes they serve only to confuse
matters.

These techniques may involve:

· Using functions to estimate uncertain risks;

· Weighting benefits according to their relative importance;

· Evaluating external economies and diseconomies, spillovers or
externalities;

· Measuring excess burdens or deadweight losses;

· Estimating prices absent market distortions such as excess burdens and
externalities (shadow prices);

· Estimating willingness to pay; or

· Other methods or algorithms for financial or economic analysis.

Traditionally (and as is true in most areas of human endeavor), the new
techniques that have real value find their way into the mainstream of
practical cost/benefit analysis, while the questionable or overly
complex techniques get left aside.

The bottom line: If you understand, and can explain, and can defend the
use of these techniques in your cost/benefit analysis, and if they
provide useful additional information, then use them. Otherwise, don\'t.

**How much effort is all this worth?**

One of the most frequently asked questions by analysts responsible for
cost/benefit analyses is \"how detailed does this thing need to be?\"
Put another way, how does one determine the \"appropriate\" scope or
level-of-effort for any given cost/benefit analysis?

There are three traditional answers to the question of scope:

· Conduct a cost/benefit analysis with detail commensurate with the size
and scope of the acquisition.

· Make it as complete and detailed as it needs to be to support the
decision it is intended to support.

· Spend up to 3% of the total projected system cost on planning
(including needs, feasibility, alternatives, cost/benefit, risk, and
requirements analyses).

Applying the \"3%\" rule: If the system is estimated to cost in the
\$6,000,000 to \$8,000,000 range, the supporting analyses \"should\"
cost from about \$180,000 to \$240,000. Cost/benefit analysis may
comprise as much as 15% of the total initial planning effort, or from
about \$27,000 to \$36,000 in this example. If a planning analyst
receives \$40,000 per year in salary and \$20,000 per year in benefits,
the total level of effort (in time) for the cost/benefit analysis should
be from about 23 work weeks up to about 31 work weeks.

From another perspective, before any manager signs off on a plan to
spend five or ten million or more taxpayers\' dollars, he or she is
going to want to see ample evidence that alternatives, costs, and
benefits were analyzed, weighed, *and documented* carefully and
completely.

Further, ACF emphasizes the importance of tracking and reporting
\"actuals\" --- the real costs and benefits that result from
implementing a new public benefit information system. In order for the
actuals to have any value or meaning, there must be a reasonably
detailed \"forecast\", or pre-implementation cost/benefit analysis.

**Assembling Data**

A cost/benefit analysis can be no better than the basic numbers that are
used to build it. In fact, a cost/benefit analysis can be *internally
consistent* (see above), and yet be *practically useless* if
unreasonable starting numbers were used.

After ensuring that a cost/benefit analysis is internally consistent,
reviewers evaluate the *completeness* and *reasonableness* of the costs
and benefits that are presented. Costs and benefits are complete when
all significant factors have been identified and evaluated. Costs and
benefits are reasonable if their sources and bases are clearly
identified, explained and justified. As a rule of thumb, costs are
simpler to identify and quantify than benefits, and require less in the
way of explanation and justification. For example, new system hardware
and software costs usually are readily available, in adequate detail,
from interested vendors. On the other hand, benefits are often
speculative or uncertain and must be developed and documented carefully
for inclusion in a cost/benefit analysis.

A good way to start is to assemble the team (sometimes a team of one!)
that is to be responsible for developing source information. Make sure
that the right people inside the organization are aware that a study is
underway, and that they know who will be contacting them for
information, and when. Much useful information can be developed through
interviews or other survey techniques. Often, important cost or benefit
factors will be identified by program managers or proposed system users
that would not otherwise have been considered. As one member of such a
cost/benefit analysis study team noted:

*We had trouble with intangibles. Everybody had a gut feeling that we
needed to do something, but they all wanted numbers before reaching
consensus. So we went to the experts --- the managers and users in areas
affected by the project. We developed the benefits with their help. The
final numbers stood up because of their source.*[^7]

Cost information sources include internal budget, finance, operations,
and human resources cost records and reports; reports to the public and
outside organizations; management and staff directly, by interview or
survey; commercial research organizations, such as DataPro or Gartner
Group; and external information sources, such as other government
organizations or vendors.

Benefit information sources include internal budget, finance, operations
and human resources forecasts, projections, record systems and reports;
management and staff directly, by interview or survey; and external
sources, such as other government organizations.

A particularly useful way to project quantified benefits (and to
determine costs for

the status quo) is to find out exactly how much time is currently being
spent, by

activity, by (for example) child support enforcement specialists.
Another method is

to use survey techniques to determine (for example) the average number
of new

cases per month, time to process new cases (by type of case), number of
referrals per month, time to complete referral forms, number of
\"locate\" cases per month, time

to complete various phases of locate activity, number of paternity cases
per month, time

to complete, etc. These \"baseline\" numbers can be used to estimate the
value of

system improvements that reduce the time necessary to perform specific
functions,

and thus increase the caseload that can be handled by an average
caseworker. The

increased caseload can be translated into dollars as cost savings or
avoidances from reduction in overtime expenditure, reduction in staff,
and / or reduction in staff growth.

Remember also that benefit information does not have to be expressed in
monetary terms to be useful or have value. Competing alternatives may
have comparable *quantifiable* (monetary) costs and benefits, and yet
differ widely in terms of non-monetary or intangible benefits.

As with most worksheet-oriented problems, an electronic spreadsheet is
the ideal tool to use in developing and producing tabular reports. A
spreadsheet can automate routine tasks, such as the distribution of
costs over time or the generation of summary or \"roll-up\" views of
information. To illustrate these and other capabilities, OISM has
developed a prototype set of spreadsheet templates and macros (with
brief instruction sheet) to automate the development and production of
cost/benefit analysis tabular reports. These templates are available in
Lotus 1-2-3, Microsoft Excel, and Borland Quattro-Pro formats. They are
recommended only for experienced users since ACF can provide only
limited phone support.

**Developing Benefits**

Cost/benefit analysis in the private sector is normally concerned with
determining whether expenditures will result in increased income. The
effect on the bottom line is the primary concern. What net profit will
result?

Only a handful of federal programs can conduct cost/benefit analyses as
\"profit\" decisions --- notably, the IRS and the Child Support
Enforcement Program. These organizations generate collections (income)
that offset (in a sense) the costs of the government\'s programs to
collect or provide welfare support. In these systems, improved and
integrated information systems can increase collections, resulting in a
net gain for the government.

However, most public sector cost/benefit analysis is concerned with net
program effect. The government does not charge for its services: public
services or benefits are required by statute to be provided. So the
government\'s obligation is not to maximize profit, but to make
cost-effective expenditures and to deliver maximum benefits within the
budget.

Therefore, most public sector cost/benefit analysis does not seek to
increase program funds --- but *to change the distribution of costs
enough to support system development*

*within the overall budgetary limitations of the organization. By this
means, public agencies prove projects to be cost-effective.*

ACF views cost/benefit analysis as serving four fundamental and equally
important

needs --- to:

· Evaluate alternative mixes of financial, human, and information
resources,

· Support wise economic decisions on proposed information system
investments,

· Establish a performance baseline against which to measure the success
of the systems project, and

· Provide fundamental management tools to maximize benefits and minimize
costs.

Therefore, cost/benefit analysis is a process of developing \"economic
indicators\" that serve as important tools in management
decision-making. These economic indicators reflect how the distribution
of costs change --- so that the net effect on the program can be
evaluated. The questions are:

· Can enough be \"saved\" in other categories to \"pay for\" the costs
of developing the new system,[^8] and

· Will the system project result in measurable improvements over current
operations?

*Public sector cost/benefit analysis is not an accounting process. When
benefits equal costs, the analysis has not proven the system will cost
nothing.* It has proven that the organization will remain within the
overall, projected program budget --- and that the projected benefits
are sufficient to warrant the expenditure for the system project.

Given these precepts, ACF permits States to develop benefits in several
ways, listed below in approximate order of preferability:

· Increased collections,

· Program cost savings,

· System cost savings,

· Program cost avoidances, and

· System cost avoidances.

In this context, *cost savings* apply when benefits are developed from
firm, fixed costs approved in the State\'s law or budget (such as
cost-of-living increases) or set in a contract to which the State is a
party. *Cost avoidances* are more speculative and are based on reasoned
projections of costs expected to develop and affect future budgets and
expenditures. *The inherent nature of a cost avoidance calculation
requires more rigorous analysis and justification*, because it is based
on assumptions (estimated future staff needs) rather than facts
(budgeted staff costs). \[Compare, for example, the detail in Benefits 2
(cost savings) and 3 (cost avoidances) in Chapters 2 and 3.\]

*System costs* are those costs that directly relate to the systems
project and are included in the \"cost side\" of the cost/benefit
analysis. *Program costs* are those *affected by the project but not
obligated for the systems project itself.* Program costs are not on the
\"cost side\" of the cost/benefit analysis, but may be the basis for
claiming benefits. The table below shows the characteristics of each
type of benefit.

+----------------+---------+---------+---------+---------+----------+
| **Type of      | *       | **      | *       | **      | **System |
| Benefit /**    | *Collec | Program | *System | Program | Cost     |
|                | tions** | Cost    | Cost    | Cost    | Avoi     |
| **Ch           |         | Sa      | Sa      | Avoid   | dances** |
| aracteristic** |         | vings** | vings** | ances** |          |
+----------------+---------+---------+---------+---------+----------+
| Cost in        |         |         |         |         |          |
| Budget, Law,   |         |         |         |         |          |
| or Contract?   |         |         |         |         |          |
+----------------+---------+---------+---------+---------+----------+
| Cost for       |         |         |         |         |          |
| System         |         |         |         |         |          |
| Development or |         |         |         |         |          |
| Operation?     |         |         |         |         |          |
+----------------+---------+---------+---------+---------+----------+
| Cost for       |         |         |         |         |          |
| Program        |         |         |         |         |          |
| Operations?    |         |         |         |         |          |
+----------------+---------+---------+---------+---------+----------+
| Could Affect   |         |         |         |         |          |
| Cost Side of   |         |         |         |         |          |
| Cost/Benefit   |         |         |         |         |          |
| Analysis?      |         |         |         |         |          |
+----------------+---------+---------+---------+---------+----------+
| Increased      |         |         |         |         |          |
| Revenues?      |         |         |         |         |          |
+----------------+---------+---------+---------+---------+----------+

**Benefits Based on Increased Collections**

For child support systems, States can base benefits on the expected
effect of the new system on caseworker productivity, measured in terms
of increased collections or other indicators. Claiming these benefits
requires several steps, which begin during the feasibility and
alternatives analysis. These steps include:

· Determining current problems and setting system objectives,

· Setting performance goals for collections or other indicators,

· Determining how the system will support the desired improvement,

· Calculating the probable effect on collections, and

· Determining how the improvement will be measured.

During the feasibility study, States identify system problems and set
objectives. For example, the State could express system problems in
terms of inadequate access to statewide or nationwide databases required
to find absent parents, procedural or operational inefficiencies,
unacceptable quality and timeliness of services to the public, or
administrative overhead cutting caseworkers\' available productive time.
To fix those problems, the State develops objectives for the new system,
such as online access to statewide and national databases, specific
procedural and operational improvements, and the reduction of manual or
administrative tasks that the caseworker must perform.

In order to determine how these improvements will affect performance,
States must first have good information on what the current performance
indicators are. These might include:

· Total successful \"locates,\"

· Average number of \"locates\" by caseworker,

· Rate of \"locates\" (in percentage) by total caseload,

· Average rate of \"locates\" (in percentage) by caseworker,

· Total collections (in dollars),

· Average collections (in dollars) by caseworker,

· Rate of collections (in percentage) by total caseload, and

· Average rate of collections (in percentage) by caseworker.

In addition, the State should collect or develop timeliness records on
elapsed time until absent parents are located and until collection
begins, stated in terms of time ranges. For example, the current
system\'s \"successful locate rate\" is 10% within 30 days, 25% by 60
days, 40% by 90 days, and so forth.

Once the State has this information, the effect of system improvements
can be evaluated and new performance objectives set which are both
specific and measurable. At the same time the performance objectives are
set, a plan to measure performance under the new system should be
developed.

Since the cost/benefit analysis developed for this *Companion Guide* is
\"generic,\" it does not include a sample benefit for increased
collections. However, States may want to consider that this benefit can
be evaluated in two ways: the social net benefits[^9] resulting directly
from the collections and indirectly from reductions in welfare payments.

**Benefits Based on Program Cost Savings**

Benefits based on program cost savings evaluate the effect of the
project, such as improved efficiency or elimination of costs, on
currently budgeted or obligated funds.

The difference between costs budgeted for program operations under the
*current* system and costs projected for program operations under the
*new* system is claimed as a benefit for the alternative. This benefit
helps offset the project costs.

Note that the budgeted and projected program costs are *not* included as
costs in the analysis --- only the difference as a benefit.

An example of this type of benefit is shown in Benefit 2, set forth in
both Chapters

2 and 3. In this benefit, improved caseworker efficiency supported by
system improvements is expected to reduce caseworker overtime pay. The
program cost

savings of \$150,000 are listed as a benefit. Caseworker costs are not
included on the cost side of the cost/benefit analysis. See the table
below.

  ----------- ----------------------------- -----------------------------
  **Benefit 2                               
  --- Program                               
  Cost                                      
  Savings**                                 

  **Year**    **Cost Side**                 **Benefit Side**

  1                                         \$ 150,000

  2                                         150,000

  3                                         150,000

  4                                         150,000

  5                                         150,000

  **Total**                                 \$ 600,000
  ----------- ----------------------------- -----------------------------

**Benefits Based on System Cost Savings**

Benefits based on system cost savings evaluate the projected effect of
the project on currently budgeted or obligated funds supporting systems
operations.

Examples of this type of benefit include elimination of staff positions
for systems operators and programmers, lower utility costs resulting
from lower power consumption, and reduced costs for space when moving
from centralized to distributed processing.

There are three ways to evaluate this \"benefit\" in a cost/benefit
analysis. None is perfect: all are acceptable.

Conventional wisdom suggests that the effect of the benefit be applied
to either the cost *or* the benefit side of the cost/benefit analysis,
not both. Those two alternatives are addressed first.

*[Benefits on the Cost Side Only]{.underline}.* If the savings were
shown only on the cost side, then reduced costs for the alternative
would be revealed in a side-by-side comparison of current and proposed
system costs, not by reviewing the benefits identified for the system.

There are three problems with this approach.

The first is that a benefit counted in the cost side can get \"lost\"
when combined with other cost elements in the same category and may not
be measured during systems implementation and operation. While this may
not be important for low dollar cost reductions such as utility costs,
it may be quite significant when professional salary reductions are
involved. For example, costs for systems programmers (expected to be
reduced with the new system) would be combined in the cost analysis with
other personnel costs, such as system operators, data entry staff, and
security officers. When costs are combined with others on the cost side
and no specific benefit is claimed, they are not identified as specific
measurement goals.

Benefits are not \"lost\" when identified on the benefits side. They
become discrete, performance management goals against which improvements
achieved by the new system are measured.

The second problem is that measuring a benefit on the cost side reflects
only the projected cost, not the improvement achieved over the status
quo by the new system. For example, if the old system required \$100,000
of systems programmers and the new system is projected to require
\$50,000, then actual costs of \$75,000 are evaluated only from the
perspective of a \$25,000 cost overrun above projected cost --- not of
the \$25,000 improvement over prior system operations. This is because
actuals are compared to projected costs for the alternative --- not the
status quo. See the table on the next page.

Third, not only is the cost reduction from the status quo not available
for performance measurement, it also is not available for offsetting the
system development costs. If the intent is to evaluate *overall program
effect,* this method does not assess the cost savings from the old
system to the new system.

While this may not matter for low dollar cost savings between cost
elements, it may be quite significant in areas where major reductions
are projected.

  ----------------- -------------------------- --------------------------
  **Benefits on the                            
  Cost Side Only**                             

  **For:**          **Cost Side**              **Benefit Side**

  Status Quo        \$100,000                  

  Alternative\*     50,000                     

  Actual\*          75,000                     
  ----------------- -------------------------- --------------------------

\*Figures in these rows are compared during cost measurement. Note that
cost side figures would typically be combined with other staff costs in
the cost analysis and lose their effectiveness as discrete, measurable
goals.

*[Benefits on the Benefits Side Only]{.underline}.* Some of the problems
cited above are eliminated when a system cost savings benefit appears
only on the benefits side: benefits are not lost, are measurable goals,
and offset system development costs. In addition, the improvement over
the status quo would be evident because the narrative benefit
description would establish the basis for the \$50,000 benefit --- that
is, \$100,000 *status quo* costs less \$50,000 projected costs for the
alternative.

*However, one major new problem is introduced.* A cost element central
to the systems project is not counted on the cost side --- with the
result that the total systems cost is not truly a total systems cost.

  ----------------- -------------------------- --------------------------
  **Benefits on the                            
  Benefits Side                                
  Only**                                       

  **For:**          **Cost Side**              **Benefit Side**

  Status Quo                                   0

  Alternative\*                                \$ 50,000

  Actual\*                                     \$25,000
  ----------------- -------------------------- --------------------------

\*Figures in these rows are compared during cost measurement.

*[Benefits on the Cost and Benefit Side]{.underline}.* The third method
shows the respective costs for the current and proposed system and
claims the difference as a cost savings on the benefits side of the
analysis.

Viewed from an accounting perspective, some claim this is double
counting. Using the systems programmer example, this argument suggests
that if you project \$50,000 in costs for systems programmers on the new
system, then claim a cost savings benefit of \$50,000 over the status
quo,[^10] you get a net cost of \$0 for systems programmers, which is
clearly incorrect.

However, the effect can be interpreted differently.

Taking a net program effect approach, the data would mean that under the
new system \$50,000 is projected for systems programmers. Further, a
\$50,000 cost savings from the status quo helps offset new system costs
and provides a measurable goal for new system operations. The
perspective is --- what can you buy with available funds? This helps the
State evaluate the overall program effect of redistributing
expenditures.

This method is sometimes used by federal agencies and contractors. Given
the importance of evaluating overall program effect and of establishing
a measurable baseline, States may also use this method.

An example of a system cost saving calculated for net program effect is
shown in Benefit 4 in Chapters 2 and 3. In this benefit, the current
system is not able to meet peak processing loads, requiring the State to
shift some of the processing load to outside service bureaus. The new
system would handle all peak loads, so that budgeted costs for the
service bureau would not have to be spent. These funds could offset
system development costs.

The table below illustrates how the system programmer example would be
calculated using both the cost and benefit side of the analysis.

  ----------------- -------------------------- --------------------------
  **Benefits on the                            
  Cost and Benefit                             
  Sides**                                      

  **For:**          **Cost Side**              **Benefit Side**

  Status Quo        \$ 100,000                 0

  Alternative\*     50,000                     \$ 50,000

  Actual\*          75,000                     25,000
  ----------------- -------------------------- --------------------------

\*Figures in these rows are compared during cost measurement. Note that
cost side figures would typically be combined with other staff costs in
the cost analysis and would not serve as discrete performance management
goals. On the benefits side, benefits have been achieved, although not
as much as anticipated.

**Benefits Based on Program Cost Avoidances**

Benefits based on program cost avoidances calculate the effect of the
project, such as improved caseworker efficiency or staffing for future
caseloads, in increasing or decreasing budgetary spending levels. In
other words, cost avoidances do not deal with fixed budgetary dollars,
but on projected increases (or decreases) likely to be required in the
budget.

As indicated previously, cost avoidances are more speculative than cost
savings. They are based on reasoned projections of costs expected to
develop and affect future budgets and expenditures. *The inherent nature
of a cost avoidance calculation requires*

*more rigorous analysis and justification.*

The difference between costs projected for program operations under the
*current* system and costs projected for program operations under the
*new* system is claimed as a benefit for the alternative. This benefit
helps offset the project costs. Note that the projected program costs
are *not* included as costs in the analysis --- only the difference as a
benefit.

An example of this type of benefit is shown in Benefit 3 in Chapters 2
and 3 and summarized in the table below. In this benefit, staffing
increases in the number of caseworkers are projected to be higher and
start earlier for the current system than for the chosen alternative. In
this case, the difference between the projected total annual salaries
for caseworkers under the status quo and alternatives is claimed as a
program cost avoidance benefit. Note that the explanation of the
derivation and calculation of this benefit is far more thorough than
that of Benefit 2, which claims a program cost savings.

  -------------- ----------------------------- -----------------------------
  **Benefit 3                                  
  --- Program                                  
  Cost                                         
  Avoidances**                                 

  **Year**       **Cost Side**                 **Benefit Side**

  1                                            0

  2                                            \$ 481,920

  3                                            1,034,120

  4                                            1,257,510

  5                                            1,420,660

  **Total**                                    \$ 4,194,210
  -------------- ----------------------------- -----------------------------

**Benefits Based on System Cost Avoidances**

Benefits based on system cost avoidances calculate the effect of the
project, such as eliminating courier service fees by enhanced
telecommunication capability, in increasing or decreasing budgetary
spending levels. In other words, cost avoidances do not deal with fixed
budgetary dollars, but with increases (or decreases) likely to be
required in the budget.

To repeat, cost avoidances are more speculative than cost savings and
require more rigorous analysis and justification.

As discussed in detail under the system cost savings section, there are
three ways to evaluate this \"benefit\" in a cost/benefit analysis:

· Benefits on the cost side only,

· Benefits on the benefits side only, or

· Benefits on the cost and benefits sides.

None is perfect: all are acceptable.

A system cost avoidance calculated for net program effect (both cost and
benefit sides) is shown in Benefit 5 in Chapters 2 and 3 and summarized
in the table below. In this benefit, the State has begun to incur new
expenses resulting from using courier services to deliver information to
meet new program and timeliness mandates. The new system would meet this
requirement through a new telecommunications network, so that funds
would not have to be redirected and budgeted to cover these costs. This
cost avoidance allows those funds to spent *differently,* supporting
system development costs.

  ----------------- -------------------------- --------------------------
  **Benefit 5:                                 
  System Cost                                  
  Avoidances**                                 

  **For:**          **Cost Side**              **Benefit Side**

  Status Quo        \$ 650,000                 0

  Alternative\*     65,000                     \$ 585,000

  Actual\*          65,000                     585,000
  ----------------- -------------------------- --------------------------

\*Figures in these rows are compared during cost measurement. Note that
cost side figures would typically combined with other costs in the cost
analysis and would not serve as discrete performance management goals.
On the benefits side, benefits have been achieved as anticipated.

**Sensitivity Analysis: Dealing with Uncertainty and Risk**

Because cost/benefit analysis deals primarily with future events, many
costs and most benefits are based on predictions. The degree of
certainty of these predictions can vary: from \"near certain\", such as
in estimating future costs for hardware maintenance on a new system, to
highly speculative, such as in estimating the dollar value of an
electronic mail system that *should* displace a substantial amount of
memo and letter writing.

Most cost/benefit analyses are based on a single set of numbers, which
usually represent the \"best guess\" by analysts as to the value of
uncertain outcomes. In fact, the degree of risk that is inherent in a
cost/benefit analysis depends entirely on the degree of confidence in
such \"best guesses\". Sometimes, varying a single estimated value by a
significant amount can alter the basic outcome of a cost/benefit
analysis, by shifting the optimum net present value from one alternative
to another. How should a cost/benefit analysis deal with such
situations, when so much is at stake over what are essentially informed
guesses? What is the effect on a cost/benefit analysis of altering the
basic estimates or assumptions that went into its development?

The basic technique for resolving such issues is to perform a
*sensitivity analysis*. A sensitivity analysis is a methodical approach
to varying the basic or underlying assumptions contained within a
cost/benefit analysis. Sensitivity analysis is the game of --- *what
if?* --- and serves two important purposes. It validates the conclusions
of a cost/benefit analysis by examining the effects of changes to the
basic, underlying assumptions that formed the analysis, and it provides
a framework for subsequent recording and analysis of actual incurred
costs and realized benefits.

A common approach to sensitivity analysis involves two steps. First,
identify the range of possible values for each key variable[^11]. A
typical approach is to identify \"high\", \"low\", and \"expected\"
values. Second, examine the effect on the outcome of the cost/benefit
analysis (e.g., the net present value) as the high, low and expected
values are plugged in, *while each of the other variables is held
constant at its \"expected\" value*[^12].

The sensitivity analysis may identify a *critical variable:* one that
has the effect of shifting the optimum NPV from one alternative to
another when its initial value is changed. When this occurs, it is
usually an indication that additional research and analysis is needed,
either to refine the estimate (i.e., narrow the expected range of
values) for that critical variable, or to quantify other costs or
benefits that may help differentiate between the alternatives.

The results of the sensitivity analysis should be a part of any
cost/benefit analysis. As a system is implemented and becomes
operational, the actual costs incurred and benefits realized will
provide a confirmation (or refutation) of the \"best guess\" estimates
that were used in the original planning.

When actual outcomes vary substantially from those projected, the
sensitivity analysis can be revisited to determine whether the outcomes
are within the range of possibilities that were identified by the
sensitivity analysis. It\'s better to be wrong than to be surprised!

**Actuals and Updates**

Systems designers refer to a system\'s \"life cycle,\" which underscores
the fact that modern information systems are expected to serve a
particular purpose for a *limited* period of time, after which
(presumably) new technologies will have evolved such that the following
generation system will cost less, run faster, be easier to use, and so
forth.

Regardless of the stage in the life cycle of a public benefit
information system, at some point it will be necessary to begin the
planning process anew. For cost/benefit analysis, the best place to
start is with the \"status quo\" --- the existing system. The costs and
benefits of the status quo provide a baseline from which alternative
costs and benefits are estimated, and against which the alternatives
will be compared.

During systems design, development, implementation, and operation, costs
and benefits should be tracked and recorded in as much detail as
possible (commensurate with the level of detail of the original
cost/benefit analysis). These \"actuals\" should be recorded and updated
over time, to provide a year-by-year (or quarter-by-quarter,
month-by-month, etc.) view of how costs and benefits actually accrue.

Tracking and updating actuals provides a vital feedback-loop for
management\'s use in controlling and maximizing the cost effectiveness
of the project. In addition, the process will support the development of
better cost/benefit analyses over time. Initial assumptions and formulas
can be evaluated and adjusted as suggested by comparison with the
results. This should be an iterative process; one of constant
refinement. The methods and outcomes of cost/benefit analysis will be
improved and management control will be enhanced by access to
information systems planning projections and forecast-versus-actual
reporting.

**The Reviewer\'s Perspective**

As the Introduction to the *Feasibility, Alternatives, and Cost/Benefit
Analysis Guide* points out, it was developed for use by Federal
personnel in evaluating State submissions, as well as for use by States
in developing their submissions. In general, Federal reviewers of State
cost/benefit analysis submissions look for three things: *completeness*,
*reasonableness*, and *internal consistency:*

· **Completeness** in that all of the important cost and benefit factors
have been identified, and that adequate discussion is provided to
explain their sources and derivation.

· **Reasonableness** in that the assumptions, rationale, sources,
derivations, justifications, explanations, and projections are sound and
defensible.

· **Internal consistency** in the assignment of costs and benefits, in
the treatment of real or nominal dollars, and in the application of
present value discounting.

Finally, and to sum up, the review process is certainly simplified when
submitters have followed the guidance and used the worksheet formats
presented in the Guide!

[^1]: **^?^** Of course, to a lesser extent it also involves the
    disciplines of politics, sociology, computer and communications
    sciences, psychology, probability, statistics, etc.

[^2]: **^?^** In particular, OMB Circular No. A-94: *Guidelines and
    Discount Rates for Benefit-Cost Analysis of Federal Programs*, 57
    Federal Register No. 218, November 10, 1992.

[^3]: **^?^** Note that this paragraph discusses OMB\'s policy on
    cost-effectiveness analyses conducted by Federal agencies --- not
    ACF\'s for the States. Federal cost-effectiveness analysis is most
    typically used in weapons acquisitions.

[^4]: **^?^** Generally referred to as \"sunk costs\" and \"realized
    benefits\", respectively.

[^5]: **^?^** Note, as a point of interest only, that the discount
    factor used in a nominal (current) dollar analysis will be different
    from that used in a real (constant) dollar analysis.

[^6]: **^?^** The foundation for the 7% policy is OMB Circular A-94.

[^7]: **^?^** *ROI? There is a better way.*, Marc Dodge, **Corporate
    Computing**, May 1993, p. 109.

[^8]: **^?^** This process is sometimes referred to as work process
    re-engineering. Examples in this document include elimination of
    courier services and staff reassignment (Benefits 5 and 1,
    respectively, in Chapters 2 and 3).

[^9]: **^?^** OMB Circular A-94 provides: \"Social net benefits, and not
    the benefits and costs to the Federal Government, should be the
    basis for evaluating Government programs or policies that have
    effects on private citizens or other levels of Government.\" ACF
    permits the calculation of social net benefits.

[^10]: **^?^** If the old system required \$100,000 of systems
    programmers and the new system is projected to require \$50,000,
    then the projected cost *savings* are \$50,000 and the projected
    *costs* are \$50,000.

[^11]: **^?^** There should be no more than five or six \"key\"
    variables; more than this number adds complexity to the analysis
    without providing any additional meaningful information. If a
    cost-benefit analysis contains more than five or six highly
    uncertain variables, more research and data analysis is needed.

[^12]: **^?^** To examine every possible combination of variables (e.g.,
    a high, low and expected value for each of six variables) would
    result in 720 (6 factorial) separate outcomes! Such an exercise
    would likely be counter-productive.