# 19. SOFTWARE PRICING {#software-pricing .Style2 .unnumbered}

Table of Contents

19.1 Introduction [19-1](#introduction)

19.1.1 SEI Requisites [19-2](#sei-requisites)

19.1.2 Complexities of Software Cost Estimating
[19-2](#complexities-of-software-cost-estimating)

19.2 Software and the Life Cycle [19-5](#software-and-the-life-cycle)

19.2.1 Types of Software [19-5](#types-of-software)

19.2.2 Software Language [19-6](#software-language)

19.2.3 Software Development Cycle [19-7](#software-development-cycle)

19.2.4 Additional Modes of Software Development
[19-11](#additional-modes-of-software-development)

19.3 Software Cost Estimating Process
[19-19](#software-cost-estimating-process)

19.3.1 Process for Software Estimation Activities
[19-20](#process-for-software-estimation-activities)

19.3.2 Software Development Standards
[19-27](#software-development-standards)

19.3.3 Benefits of SW Estimation Process
[19-28](#benefits-of-sw-estimation-process)

19.4 Estimating Software Size [19-28](#estimating-software-size)

19.4.1 Heuristic Approaches (\"Rule of Thumb\")
[19-29](#heuristic-approaches-rule-of-thumb)

19.4.2 Metric Based Models [19-30](#metric-based-models)

19.4.3 Reused Code [19-42](#reused-code)

19.4.4 COTS Sizing [19-44](#cots-sizing)

19.4.5 Summary of Metric Based Sizing Models
[19-44](#summary-of-metric-based-sizing-models)

19.5 Software Cost Estimating Methodologies
[19-45](#software-cost-estimating-methodologies)

19.5.1 Analogy or Comparative [19-46](#analogy-or-comparative)

19.5.2 Grass Roots, Engineering Estimate or Bottoms-up
[19-47](#grass-roots-engineering-estimate-or-bottoms-up)

19.5.3 Top-Down Method [19-48](#top-down-method)

19.5.4 Expert Judgment [19-48](#expert-judgment)

19.5.5 Parametric Modeling or Algorithms
[19-48](#parametric-modeling-or-algorithms)

19.6 Life cycle Costs [19-50](#life-cycle-costs)

19.6.1 CDRL Deliverables Estimates [19-50](#cdrl-deliverables-estimates)

19.6.2 Software Maintenance Estimates
[19-50](#software-maintenance-estimates)

19.7 Software Pricing Process [19-52](#software-pricing-process)

19.7.1 Software Price Analysis [19-53](#software-price-analysis)

19.7.2 Price Analysis Techniques [19-56](#price-analysis-techniques)

19.7.3 Price Analysis Tools [19-57](#price-analysis-tools)

19.7.4 Software Cost Analysis [19-59](#software-cost-analysis)

19.7.5 Considerations for Software Cost Analysis
[19-60](#considerations-for-software-cost-analysis)

19.7.6 Cost Analysis of a Parametric Estimate
[19-65](#cost-analysis-of-a-parametric-estimate)

19.7.7 Rules Of Thumb Use in Cost Analysis
[19-67](#rules-of-thumb-use-in-cost-analysis)

19.7.8 Parametric Cost Analysis Example
[19-68](#parametric-cost-analysis-example)

19.7.9 COTS Special Pricing Considerations
[19-69](#cots-special-pricing-considerations)

19.7.10 Software Pricing Summary [19-69](#software-pricing-summary)

19.8 Model Selection [19-70](#model-selection)

19.8.1 Step 1: Determine Needs [19-70](#step-1-determine-needs)

19.8.2 Step 2: Select Candidate Model
[19-73](#step-2-select-candidate-model)

19.8.3 Step 3: Choose Appropriate Model
[19-73](#step-3-choose-appropriate-model)

19.8.4 Step 4: Reevaluate Model Choice
[19-74](#step-4-reevaluate-model-choice)

19.8.5 Model Selection Summary [19-74](#model-selection-summary)

19.9 Intelligent Use of Models [19-75](#intelligent-use-of-models)

19.9.1 Input Data [19-75](#input-data)

19.9.2 Model Accuracy [19-76](#model-accuracy)

19.9.3 Model Calibration [19-77](#model-calibration)

19.10 Software Project Tracking And Measurement
[19-78](#software-project-tracking-and-measurement)

19.10.1 Cost and Schedule Tracking [19-79](#cost-and-schedule-tracking)

19.10.2 Performance Measurement Indicators
[19-80](#performance-measurement-indicators)

19.10.3 Software Quality Metrics [19-83](#software-quality-metrics)

19.10.4 Data Collection [19-85](#data-collection)

19.10.5 Software Tracking and Measurement Summary
[19-86](#software-tracking-and-measurement-summary)

19.11 Summary [19-87](#summary)

[Appendix 19A. Acronyms 19A-1]{.smallcaps}

Appendix 19B. Software Cost Estimation Terminology 19B-1

Appendix 19C. Sample Software Work Breakdown Structure 19C-1

Appendix 19D. Popular Parametric Software Cost Models 19D-1

Appendix 19E. Commercial Off-The-Shelf (COTS) Software 19E-1

Appendix 19F. Bibliography 19F-1

# Software Pricing

## Introduction

Software is a set of programs and accompanying documentation that
directs computers to perform desired functions. In simple terms, a
software program is a set of instructions for a computer. Within the
Federal Aviation Administration (FAA), for example, software controls
air traffic control systems, drives simulators, directs surveillance
systems, and controls accounting, inventory and Management Information
Systems (MIS). Software is a critical component in virtually every
current FAA system, and will grow in importance as future FAA systems
are developed and fielded. Between 1982 and 2003, the FAA will spend
\$21 billion on software-intensive Air Traffic Control (ATC) information
systems, according to the General Accounting Office (GAO) report, "Air
Traffic Control: Improved Cost Information Needed to Make Billion Dollar
Modernization Investment Decisions." Barry Boehm, in his book, *Software
Engineering Economics*, suggests software [costs]{.underline} are
contributing to an increasing proportion of overall system costs. The
FAA recognizes the need for accurate software estimates as well as
performance measurement of software development programs. Software
pricing is based upon the same steps as software cost estimating.
Therefore, the software cost estimating process and techniques will be
described first in this chapter and then the software pricing process.
**Those familiar with software cost estimating can jump directly to
section 19.7, Software Pricing Process. Note also that the subjects of
software work breakdown structure (WBS), software cost models, and
commercial-off-the-shelf (COTS) software are dealt with in detail in
Appendices C, D and E, respectively.**

A significant portion of this chapter is drawn from the following
sources: United States Air Force (USAF) software primer, *Space Systems
Cost Analysis Group (SSCAG), Software Methodology Handbook* (published
in June 1995), United States Navy's Naval Command, Control, Surveillance
Center, RDT&E Division Software Engineering Process Office (NRaD),
*Software Size,* *Cost and Schedule Estimation Process* manual, dated
June 1996; the National Aeronautics and Space Administration (NASA)
*Parametric Cost Estimating Handbook, dated Fall 1995m* and DoD's
Software Technology Support Center's (STSC) *Report on Project
Management and Software Cost Estimation Technologies*, dated April 1995.
These documents will be referenced in this chapter as SSCAG Handbook,
NRaD's Manual, NASA's Handbook and the STSC Report respectively. The DoD
community has extensive experience in the research and development of
sophisticated software estimating tools and techniques. Although some
DoD specific information is included, the concepts discussed are
applicable in the FAA environment.

### SEI Requisites

This chapter also embodies the philosophy of the Software Engineering
Institute (SEI) as published in their "Checklists and Criteria for
Evaluating the Cost and Schedule Estimating Capabilities of Software
Organizations." SEI's research has concluded that developing credible
software estimates is a function of how thorough and disciplined an
organization's estimating processes are. Accordingly, SEI has developed
six institutional process requisites that organizations in the business
of building or acquiring software-intensive systems must possess if they
are to consistently produce reliable cost estimates. The activities
necessary to ensure that the SEI requisites are met span the software
program life cycle and are discussed in this chapter. These requisites
are listed below:

-   A corporate memory, or historical database(s), for cataloging cost
    estimates, revisions, reasons for revisions, actuals, and other
    descriptive information, such as any constraints or trends that
    affect the project;

-   Structured processes for estimating software size and the amount and
    complexity of existing software that can be reused;

-   Cost models calibrated/tuned to reflect demonstrated accomplishments
    on similar past projects;

-   Audit trails that record and explain the values used as cost model
    inputs;

-   Processes for dealing with externally imposed cost or schedule
    constraints to ensure the integrity of the estimating process; and

-   Data collection and feedback processes that foster capturing and
    correctly interpreting data from work performed.

### Complexities of Software Cost Estimating 

According to Richard D. Stutzke in his article "Software Estimating
Technology: A Survey" in STSC's *Crosstalk* magazine, the software cost
estimation process is difficult for several reasons. The first problem
is that projects often must satisfy conflicting goals. Projects to
develop or maintain software must provide specified functionality within
specified performance criteria, within a specified cost and schedule,
and with some desired level of quality (absence of defects). Software
engineering processes can be chosen to meet any one of these project
goals. Usually, however, more than one goal must be satisfied by a
particular project. These multiple constraints complicate the estimation
process.

The second problem is that estimates are required [before]{.underline}
the product is well defined. Software functionality is difficult to
define, especially in the early stages of a project. The basis for the
first good cost estimate is usually not available for totally new
systems until the top-level design has been defined. This level of
design is only defined at the preliminary design review in many
government contracts and sometimes not even then, which leads to
undesirable consequences. This milestone is reached after about 20
percent of the total effort and 40 percent of the total duration have
been expended by the project staff. At this point in a project, typical
accuracies for the estimated effort and duration are within 25 percent
of the final project actuals. In general, since more information becomes
available, the accuracy of estimates increases as a project proceeds. To
reduce costs, as well as to improve quality and reduce development
times, some projects employ pre-defined \"domain specific software
architectures\". The development costs for such projects can be
estimated more accurately than for projects that build a totally new
product since more information about the product is available earlier.
In general, however, the estimator must apply considerable skill to
estimate project cost and schedule early in a project.

More data are available for modifications of existing code than when the
existing code was written, so more accurate estimates are possible for
this kind of project compared to totally new development. A large
portion of all software maintenance work is a response to changes in the
original requirements or in the system\'s external environment (mission,
interfaces to other systems etc.). Software processes must produce
software that can be gracefully evolved at reasonable costs. The choice
of the software architecture significantly influences modifiability and
hence maintainability. New ways to estimate the costs of such projects
continue to be developed.

Modification of code is closely tied to software reuse. Various cost
estimation models have been developed to qualify the economic costs of
building and using reusable components. For example, Richard Selby
analyzed costs at NASA\'s Software Engineering Laboratory and found that
there is a large increase in programmer effort as soon as a programmer
has to \"look inside\" the component. The cost to reuse a component
depends on its suitability for the intended application, its structure,
and other factors. The cost to reuse software compared to the cost to
develop new software is a function of the amount of existing code that
must be modified to install the code in a new system. A model developed
using data from Richard Selby, Gerlich Rainer, Ulrich Denskat and Barry
Boehm, indicates several important aspects of reused software costs.
First, the cost is not zero even if no code is modified. Second, the
costs increase faster than linearly at first. Third, the cost to modify
all of the existing code is more expensive than to develop the code from
scratch. (Effort is wasted to understand, then discard the existing code
before the new code is written. This effort is never expended if the
decision is made to develop totally new code from the start.) For the
worst case, the economic break-even point occurs when only 20 percent of
the code is modified; reuse is not cost effective above the break-even
point. Such nonlinear behavior was not handled in previous cost models.

A third complication arises because the nature of software building is
changing. New development processes emerge and new ways are needed to
estimate the costs and schedules for the new processes. These processes
endeavor to provide higher quality software (i.e., fewer defects),
produce more modular and maintainable software, and deliver software
(products and prototypes) to the end user faster. To meet these and the
other objectives (stated previously), developers use combinations of
pre-built code components and labor-saving tools. For example, many
programming tasks are being put into the hands of the users by providing
macro definition capabilities in many products. These capabilities allow
users to define sequences of frequently used commands. A slightly more
sophisticated approach is to allow domain experts to construct
applications using special tools such as fourth-generation languages and
application composition tools. Larger systems intended for specialized
(one of a kind) applications are often built using
commercial-off-the-shelf (COTS) products to provide functionality in
areas that are understood well enough to be standardized. Examples are
graphical user interfaces and relational database management systems.
The trend toward object oriented languages supports this by making it
easier to develop \"plug compatible\" components. Thus, reuse of code
becomes an increasingly large factor in cost estimation, and a good
understanding of the cost factors associated with software reuse becomes
even more important.

Lastly, R. Selby is one of the authors that advocate of \"measurement
driven development processes\" wherein process activities are adapted
during the course of a project based on measurements of process
performance. Planning and costing such processes prior to the start of
the project is impossible.

The pricing process, terms and principles are described in the other
chapters of this handbook. Since software pricing is dependent upon an
understanding of the software cost estimation process, this chapter will
discuss the issue of software cost estimation. First, software itself is
explained so that the reader may fully understand what costs are being
estimated. Next, the software life cycle is described, which includes
the software development and software support phases. The specifics of
the software cost estimating process follow, which will contain sections
on the software work breakdown structure, software sizing, and
estimating methodologies. Following the discussion of cost estimating
will be a discussion of the software price analysis and cost analysis
process. Finally, a discussion of parametric cost models and project
tracking and metrics is included.

## Software and the Life Cycle

Computers are dominating every aspect of modern lifestyles. Computers
range in size and complexity from large mainframe computers used in
major companies to the small personal computers which have become
household items. Microcomputers are now being used in such commonplace
consumer goods as automobile engines, televisions, and microwave ovens.
To function, however, computers must be programmed using a language they
can interpret. This computer language consists of electronic signals
represented by the binary numbers "1" and "0". Groups of these binary
numbers are interpreted by a computer as data or as instructions to be
acted upon. Sets of related instructions form computer programs, which
constitute a basic element in the concept of software. Computer software
may be defined as "computer programs, procedures, rules, and possibly
associated documentation and data pertaining to the operation of a
computer system," per IEEE-STD-729, "IEEE Standard Glossary of Software
Engineering Terms."

### Types of Software

The three basic types of software, all of which are procured by the FAA,
are listed in Table 19-1. Most estimators are familiar with all three
types of software because all are used in personal computers. MS-DOS
(Microsoft Disk Operating System), for example, is a trade name for the
operating system for IBM and IBM compatible personal computers. Listing
a directory of user files is an example of a utility software function.
Lotus 1-2-3 and Microsoft Excel are examples of application software.
The software systems procured by the FAA are usually complex
combinations of all three types of software.

Table 19-1. Types of Software

  --------------------- -------------------------------------------------
  **Type of Software**  **Function**

  System software, also A collection of programs that manages all the
  known as the          concurrent tasks being performed by a computer,
  operating system      including the execution of application software
                        programs.

  Utility software      A set of programs that perform routine tasks,
                        such as listing or compressing data, copying
                        files etc.

  Application software  Software that performs specialized functions like
                        controlling air traffic systems and MIS functions
                        mentioned above, or other useful work not related
                        directly to the operation of the computer itself.
  --------------------- -------------------------------------------------

### Software Language

Machine language, sometimes referred to as object or machine
instructions, is the actual binary instructions that a computer will
execute. Assembler code usually corresponds to machine instructions on
an equal basis, i.e. one assembler line of code usually generates one
machine instruction. Although assembler code is easier to write than
machine code, most programmers find it relatively cumbersome. Next is a
**high-order language (HOL)** instruction, or line of code. High level
languages translate human instructions into machine

language. These languages are a number of levels (at a high level) away
from the actual bit manipulation. Popular HOLs currently in use include
BASIC, JOVIAL, C++, JAVA and Ada. Compared to assembler language, HOLs
have the advantages of being much easier to read and write, but do not
always make optimal use of a computer\'s capacity. An instruction
written in assembler or HOL is often referred to as a **source line of
code (SLOC)**, to differentiate it from a machine instruction. This term
will become important in the subsequent discussions on software sizing
and software cost models.

Although most large programs today are written in assembler language or
HOL, there is a growing trend toward programs that even more closely
resemble the spoken language. Programs for spreadsheets, word
processors, and similar applications are often written in a **very
high-order language (VHOL)**. VHOLs allow a person with little or no
programming background to interact with a computer. There are programs
that enable a computer to accept vocal instructions directly. Scientists
in the field of cybernetics postulate that in the future computers will
have the capability to accept thought commands as inputs.

### Software Development Cycle 

Regardless of how software is programmed, it must proceed through
certain steps, or phases, of development. This section defines the basic
software development cycle. Major phases and software development
activities are defined, as well as key milestones for the measurement of
progress.

An example of a popular software development process is the eight-phase
process used in developing software for most **Department of Defense
(DoD)** weapon systems, as described in DoD-Standard 2167A, "Defense
System Software Development." This process, when done sequentially, is
based on the waterfall model of software development, as described by
Barry Boehm. Per NASA's Handbook*,* the process depicted by the model
involves development through specific, sequential stages. There are
specific objectives to be accomplished in each phase; each activity must
be deemed successful for work to proceed to the subsequent phase. The
process is usually considered non-iterative. Each phase requires the
delivery of particular documentation products (**Contract Data
Requirements List (CDRL)** Items). Many of the phases require successful
completion of a government review process. Critics of the "Waterfall"
Model, in fact, find that the model is geared to recognize documents as
a measure of progress rather than actual results.

The eight major activities described in DoD-STD 2167A are shown Figure
19-1 as a schematic overview of the Waterfall Model, representing
concurrent hardware and software development. The maintenance phase was
added to the end of the software development phases to represent the
operation and support needed for the complete software life cycle.

Figure 19-1. Software Waterfall Model Development Phases

**Error! Not a valid link.**

It is evident from Figure 19-1 that software development comprises much
more than writing code. In fact, according to William G. Cheadle in his
article entitled "DoD-STD-2167 Impacts on Software Development," writing
code accounts for only about twenty percent of the entire software
development process and phases 2 through 4 can account for 60% of the
entire SW development effort.

An understanding of the above process requires a comprehension of the
software hierarchy and terminology used. A system is partitioned into
various subsystems, which are further partitioned into **computer
software** **configuration items (CSCIs)** and **hardware configuration
items (HWCIs)**. A CSCI is a program, or group of programs which
satisfies a common end-use function and is managed separately. Since
CSCIs may contain over 100,000 lines of code, they are further
partitioned into **computer software components (CSCs)** and **computer
software units (CSUs)**. CSUs are the lowest-level software entities and
usually contain between 100 and 200 lines of code.

#### Systems Concept/System Requirements Analysis

The initial System Requirements Analysis activities involve exploring
and defining concepts to satisfy a valid requirement using hardware,
software, and personnel resources. The system operational requirements,
as defined by the Government, are translated into preliminary
specifications. During system design activities, the contractor may
investigate several alternative approaches that would satisfy Government
requirements, but generally selects and documents one specific system
architecture to be used. The system may be designed to use one central
computer, or a number of computers distributed throughout the system
could share the processing tasks. The contractor will identify the
computer resources he will require and the database(s) that will be used
or that will need to be developed.

The **Software Development Plan (SDP)** is one of the deliverable
products that result from the analysis and design activities. This
document presents an integrated plan for managing the development
effort. It usually summarizes the overall software structure and basic
functions, as well as the planned approach to project management. It can
be a useful document for obtaining an overview of the effort to be
estimated.

The **Systems Requirements Review (SRR)** is held at the conclusion of
requirements definition activities. The objective of this review is to
determine how adequately the contractor has identified and defined
overall system requirements.

#### Software Requirements Analysis

The primary objective of this phase is to define requirements for each
CSCI as completely as possible in order to eliminate uncertainty at the
beginning of the design phase. The customer\'s requirements are assigned
to CSCIs, and the pattern of interaction (interface requirements) among
the various components is specified. The requirements that each CSCI
must meet during testing are also specified. System analysts assess the
information flow with the software; that is, how data move and are
transformed during its input, process, and output. They also look at the
structure of data, or how various elements of data are logically
related. This involves understanding how data will be organized,
accessed, modified, and processed.

The Software Requirements Specification (SRS) is the primary deliverable
product developed as a result of software requirements analysis. An SRS
will be developed for each CSCI. This specification will describe the
goals and objectives of the software, and the problems the software is
designed to solve. It will provide details of each function that the
software will perform, and what procedures it will use to perform them.
It should provide specific criteria to be used to validate the software,
that is, specify the criteria to be used to determine whether or not the
software is "successful".

The **Software Specification Review (SSR)** marks the end of this phase.
This review evaluates the SRS for each CSCI, and the **Interface
Requirements Specification (IRS)**. The objective of this review is to
determine if the contractor is ready to proceed with preliminary design.

#### Preliminary and Detailed Design

During software design, the objective is to develop functionality based
on the requirements determined above, so that the writing of code will
easily follow. Software design is divided into two phases, Preliminary
and Detailed design; each CSCI is defined in successively greater
detail. The model will be quite general at first, but as design
proceeds, the procedural detail will become more and more similar to
source code.

In the Preliminary Design Phase, the contractor develops an overall
design approach including mathematical models and determines how control
of various functions and data will flow through the software. The
overall structure or architecture for each CSCI is determined. Each CSCI
will be divided into **Computer Software Components (CSCs)**. The
general processing that will be performed by each CSC is determined, and
test plans and requirements for each CSC are developed. **Preliminary
Design Review (PDR)** is held at the end of this phase, and is intended
to determine if the contractor is ready to proceed with Detailed Design.
Separate PDR reviews are sometimes held for each CSC.

In the Detailed Design Phase, each CSCI is further subdivided by
splitting the CSCs into **Computer Software Units (CSUs)**. Ideally,
each CSU should perform only a single function, and should consist of
approximately 100 SLOC. Data requirements, algorithms, interfaces, and
required testing are all defined down to this unit level. The end result
of detailed design is a "code to" document, a complete modular lower
level design for each CSCI. This detailed design is a "blueprint" from
which to write code. **Critical Design Review (CDR)** is held at the end
of the detailed design phase. The design phase ends with successful
completion of the **System Design Review (SDR)**. This review is held to
evaluate how well the contractor has allocated technical requirements to
the hardware and software.

#### Coding and Computer Software Unit (CSU) Testing

During Coding and Unit Test, the actual programming of deliverable code
occurs. Pseudocode is converted into formal Order Language Statements or
Source Code. This Source Code is then compiled (translated into machine
readable form) and each unit is tested as planned in previous phases.
This code and unit test activity is the effort most people associate
with software development. In terms of proportion of schedule and cost,
however, programming probably represents only 17 to 25 percent of the
overall effort. Large, complex software projects do tend to experience
manpower peaks in this phase, as the activity lends itself to
subdivision and parallel effort. On large projects, the components can
be subdivided, with many people working on small subtasks
simultaneously.

#### Computer Software Component (CSC) Integration and Testing

The previous phases define requirements and allocate those requirements
to successively lower level components of the overall software system.
Once the coding or programming of the lower level units is complete,
this process reverses itself, and effort is focused on integrating and
testing successively the higher levels of the system. The software
development contractor now has to check the algorithms and logic for
each CSC, and integrate the functions that were specified for each CSC
in its respective requirements document.

#### Integration Testing

CSCIs are tested during the next phase, and the tested software is then
integrated with the hardware it controls. This integration process and
testing of the integrated system is conducted by the contractor. This
phase concludes with the **Test Readiness Review (TRR)**.

#### System Testing

The integrated system will have to undergo System Testing. The objective
of testing in this phase is to determine if the system meets the
customer's requirements and user needs. If the system as a whole meets
all the customer\'s requirements as defined in the system
specifications, then it is ready to be delivered. The final reviews in
the process are the **Functional and Physical Configuration Audits
(FCA/PCA)**. The final test in the development phase is the Operational
Test and Evaluation, which is conducted by the organizations that are to
use the system in order to determine how well the fielded system will
actually work.

#### Software Development Cycle Summary

During the first phase, Systems Concept/System Requirements Analysis,
system level requirements are partitioned to CSCI and HWCI-level
requirements. Each CSCI now follows its own development cycle following
the phases shown in Figure 19-1 as CSCI-level activities. During the
Software Requirements Analysis phase, the specific requirements of a
CSCI are defined in detail. During the next two phases, Preliminary
Design and Detailed Design, the requirements are refined to the CSC and
CSU levels respectively. Ideally, the software is now completely
designed and is ready to code. The last three phases involve writing
source code (e.g., FORTRAN language statements) for each CSU and testing
the CSU; testing aggregates of CSUs which form CSCs, and testing the
overall CSCI to insure the CSCI requirements are met. After CSCIs are
tested, the last phase, System Test, determines whether the aggregate of
hardware and software meets system requirements. Once the system becomes
operational at a site, development is essentially complete and the
program enters the maintenance part of the life cycle.

Per the Computer Science Telecommunications Program course material
website at the University of Missouri, Kansas City, the term waterfall
is an analogy to the fact that the development process is allowed to
move in a forward direction only. When a software developer takes on a
contract to build a software system using the waterfall model it usually
agrees to provisions to deliver documents at the completion of each
phase by a particular date. Therefore, the model has been termed a
legalistic document driven approach. Obviously there are problems with
the strict adherence to the waterfall model.

The development phases shown in Figure 19-1 do not always occur
sequentially. Many modern development practices, such as prototyping or
incremental development, will result in a different order of activities.
(Also, these phases may be modified or combined in some instances. For
example, in Military Standard 498, "Software Development and
Documentation" (which replaced DoD-STD-2167A), the preliminary and
detailed design phases have been combined into a single "software
design" phase, and the requirement to have CSCs has been eliminated.

### Additional Modes of Software Development

According to Boehm, a new generation of software processes and products
are changing the way organizations develop software. The eight-phase
process described above is tailored to the standard waterfall, full
scale development method that is most commonly used when discussing
project planning and management concepts. However, other types of
development that include COTS integration, prototyping, evolutionary
acquisition, incremental development, spiral development, and technology
insertion, plus reuse-driven software procurement approaches; and
software process maturity initiatives lead to significant benefits in
terms of improved software quality and reduced software cost, risk, and
cycle time. The detailed nuances of how development proceeds under each
of these modes of development is beyond the scope of this document.
However, some general considerations that should be made for the
estimation process under each of these modes is discussed below since
the different software cost estimating models/tools described in
Appendix 19D refer to these processes.

#### COTS Integration

According to Richard Stutzke's paper "Cost Factors for COTS Integration"
presented at the Tenth International COCOMO User's Conference, Software
developers are using increasing amounts of commercial-off-the-shelf
(COTS) software products to construct new systems. These COTS products
may be either components or standalone packages. They are considered to
be ready to use and have been documented at some level of detail. They
are also supposedly well tested, making them more reliable.

Software developers choose to use COTS products to decrease development
costs, schedules and technical risks. Unfortunately, these goals are not
achieved in many cases due to an incomplete understanding of all the
factors involved. These factors include features provided and various
associated costs, including license fees. For a more in-depth discussion
of the many considerations peculiar to COTS software integration, cost
estimating, and pricing, refer to Appendix 19E of this chapter.

The model pictured in Figure 19-2 is called the five-panel model and is
extended from SEI's Special Report \[SEI-97-SR-019\] titled "Workshop on
COTS-Based Systems." The five panels represent various activities that
address different aspects of turning a set of COTS components into a
COTS-based system.

> Figure 19-2. COTS Five Panel Model

**Error! Not a valid link.**

The **COTS Market** panel deals with the market survey and analysis
activities that determine what are the viable candidates for a
particular component, from both a business and a technical perspective.

The **Qualify** panel activities investigate the hidden interfaces and
other characteristics and features of the candidate products. The most
difficult problem in this phase is to determine the characteristics of
the available COTS products so that a product that best meets the
requirements may be selected. The result of this discovery process is to
reveal the necessary information to make a selection and identify
possible sources of conflict and overlap, so that the component can be
effectively assembled and evolved. Per the March 1998 STSC Crosstalk
article on COTS, "Experience shows that the selection process for one
major product can require three to six months of calendar time, multiple
engineers and programmers, access to sophisticated suites of hardware
and software environments, and will likely entail the purchase of
vendor-provided training classes." Per Stutzke, "this phase concludes
with the purchase of the selected products which can be accomplished
using a wide variety of licensing schemes. The licensing scheme can
affect the overall cost of the system. The biggest difficulty in dealing
with licenses is there is no standard terminology in use by the various
vendors, for example "annual" and "perpetual" are used by different
vendors to mean the same thing."

The **Adapt** panel activities amend the selected components to address
potential sources of conflict. The figure implies a kind of component
"wrapping," but other approaches are possible (e.g., mediators and
translators). According to the Crosstalk article, since COTS software
does not require coding but does require integration with other
components, it starts the life cycle as a partially developed component.
The design, construction, and integration and test development stages
must be recast to accommodate early COTS software integration and
testing as well as to develop "glue" code: interface software,
configuration files, scripts, utilities, and data files required to make
the COTS software deliver its intended functionality. The proper
development and testing of the glue code to make a COTS package work may
not be a trivial undertaking. For more complex COTS software, the
development of glue code might need to be treated in the same manner as
the development of a traditional custom-coded software module.

The **Assemble** panel shows the integration of the adapted components
into an architectural infrastructure. This infrastructure will support
component assembly and coordination, and differentiates architectural
assembly from ad hoc "glue". Referring again to the Crosstalk article,
waiting until late in the development process to test and integrate COTS
products, particularly those that are complex, will not give adequate
time to master all their intricacies and complexities. COTS product
testing and integration activities must be interwoven into more of the
development process stages.

The **Update** panel acknowledges that new versions of components will
replace older versions; in some cases, components may be replaced by
different components with similar behavior and interfaces. These
replacement activities may require that wrappers be rewritten, and they
suggest the advantage of well-defined component interfaces that reduce
the extensive testing otherwise needed to ensure that the operation of
unchanged components is not adversely affected.

#### Rapid Prototyping Development

The Computer Science Telecommunications Program course material website
at the University of Missouri, Kansas City describes prototyping as
developing an initial "model", providing the prototype to the intended
users, gathering feedback from the users, and including any revisions or
refinements. If this process continues until the required system is
developed, then the process would be considered evolutionary
development. However, when the objective of the prototype is to
determine or validate system requirements, then the process illustrated
in Figure 19-3 (extended from James E. Tomayko's course material website
at Carnegie Mellon University) applies and is sometimes referred to as
throw-away prototyping.

> Figure 19-3. Rapid Prototyping Development Model

**Error! Not a valid link.**

NRaD's Manual refers to prototyping as a method to evaluate the
feasibility of technical ideas and theories that has become increasingly
popular and is a widely used development mode at various defense R&D
centers. Developing a prototype is usually a distinct portion of the
life cycle. Just as the prototype will provide insight into the design
and implementation issues, the estimate and cost of producing the
prototype will provide insight into the cost of the overall project. The
University of Cincinnati, College of Business Administration course
material website states that typically the prototype represents about
15% of the final project.

Developing a cost estimate for a Prototyping project should be
accomplished in the same manner as for a full-scale development project.
The primary cost savings resulting from prototyping is the lack of
formal reviews and documentation required. Also, there is a minimum of
systems engineering and formal testing involved.

#### Evolutionary Acquisition (EA)

Figure 19-4 (extended from the University of Illinois Department of
Computer Science course material website) shows the basic elements of EA
development.

> Figure 19-4. Evolutionary Acquisition Model

**Error! Not a valid link.**

The Computer Science Telecommunications Program course material website
at the University of Missouri, Kansas City states that one of the
shortcomings of the Waterfall Model is its incapacity to analyze the
crucial activity of risk analysis. The Evolutionary Acquisition Model is
pretty similar in content to the Waterfall Model except it encourages
prototyping. Per the NRaD Manual, EA is becoming the rule more than the
exception in large-scale development efforts, especially when the
detailed specification is difficult or impossible to create. The thesis
of \"build a little, test a little\" is seen as a way to develop systems
that better satisfy user requirements and more easily take advantage of
technical advances. The underlying factor in EA is to field a
well-defined core capability quickly in response to a validated
requirement, while using a phased upgrade program to eventually enhance
the system to provide the full system capability. This process is also
referred to as evolutionary prototyping. Documentation and the software
structure may suffer with this approach, especially if the prototypes
are created rapidly. EA can be considered a risk reduction approach and
effective prototyping can increase the software quality.

The process for estimating an EA project can be accomplished in the same
manner as for a standard waterfall project. An EA project requires the
same formality as any other method of development. Estimates should be
developed for each phase or development cycle of an EA project just as
they would be for a large Full Scale Development project. During each
phase of an EA project, two separate estimates will be required, an
estimate for the current phase and an estimate for the total project as
it is understood and defined at that time.

#### Incremental Development

The NRaD Manual refers to the incremental development approach as a top
down implementation of distinct functional elements of the product. The
development of each increment is accomplished as a separate waterfall
type of development as illustrated in Figure 19-5 (extended from James
Tomayko's course material website at Carnegie Mellon University). This
strategy allows visibility into potential design, interface problems
etc., early in the development cycle. It also provides the opportunity
to incorporate user experience into the final product. The incremental
development methodology differs from the evolutionary approach in that
under the incremental strategy the end product is well-defined.

> Figure 19-5. Incremental Development Model

**Error! Not a valid link.**

Estimates should be developed for the total project as well as for each
of the increments. Each increment should be identified and sized. The
estimates for each increment should be accomplished as for an individual
waterfall development project. The estimate for the total project will
have to show the overlapping of the increments. How the individual
increments overlap may differ from project to project.

The establishment of size, cost, and schedule estimates for each
increment will provide an overall estimate of the total project. As the
first increments are completed, then the assumptions and estimates for
the subsequent increments as well as for the total product can be
revised to reflect a more accurate assessment.

#### Spiral Development

Barry Boehm created the spiral model in 1988 for software engineering
because of the perceived inadequacies of the waterfall life cycle. The
Computer Science Telecommunications Program course material website at
the University of Missouri - Kansas City states that the spiral model
encompasses features of the phased life cycle as well as the prototype
life cycle. However, unlike those life cycles, the spiral model uses
risk analysis as one of its elements. It also uses the waterfall model
for each step so as to avoid any risks. It is a complex model that
requires complicated diagrams (such as Figure 19-6) to be fully
understood.

> Figure 19-6. Spiral Development

**Error! Not a valid link.**

The radial dimension in the figure shows the cumulative cost of the
accomplishment of the steps at that time. The angular dimension shows
the progress made in the completion of each cycle of the spiral. Every
cycle of the model represents the same sequence of steps in the
completion of the software. Each cycle of the spiral begins with the
identification of the objectives of that portion of the product, an
alternative way of implementing this portion, and the constraints on the
application of the alternatives.

The first step is used to evaluate the risks involved with the
objectives and constraints. The next step is determined by the remaining
risks. If a detailed prototype is useful enough for future product
evolution with low risks, then the next steps would be the series of
evolutionary prototypes going toward the right in the figure. However,
if previous prototyping has already resolved all of the performance
risks, then the next step uses the waterfall approach.

The primary advantage of the spiral model is that it has a wide range of
options to accommodate the good features of other life cycle models. It
becomes equivalent to another life cycle model in appropriate
situations. Also the risk-avoidance approach keeps it from having
additional difficulties. The spiral model focuses its early attention on
the option of reusing existing software. It prepares for life cycle
evolution, growth, and changes of the software product. The spiral model
has some difficulties that need to be resolved before it can be a
universally applied life cycle. These difficulties include matching to
contract software, relying on risk assessment expertise, and the need
for more detailed steps. The risk-driven model is dependent on the
developers\' ability to identify project risk and therefore depends on
the risk assessment skills of the developer. It is still fairly new
compared to other models, so it has not been used significantly and the
problems associated with it haven\'t been widely tested and solved.

The spiral model is best used for the development of large-scale
systems. It can be used for reliable and mission critical systems. The
risk-driven nature of the spiral model is more adaptable to the plethora
of software project situations than the document-driven life cycles
(i.e., the waterfall model) or the primarily code-driven life cycles
(i.e., the evolutionary model). The spiral model would be inappropriate
for use in small projects.

#### Technology Insertion

Referring to the NRaD Manual, Technology Insertion is the enhancement of
an existing system through the introduction of new technology. The new
technology for software systems is often the result of improved
hardware. The new hardware is usually characterized by higher
throughput, or increased memory size, and will often have an improved
operating system.

New hardware often provides the opportunity to also introduce a new
version of the application software. The new version is usually
characterized by improved functionality. The new software version is
typically comprised of a significant amount of new code and modified
existing code.

Technology Insertion projects are usually conducted the same as new
development projects and thus will follow the generic estimation process
contained herein. The primary difference, however, is the amount of
existing software that is planned for reuse. The software developers are
often optimistic regarding the amount of existing code that can be used.
Estimators should be conservative in regards to reused code estimates.

The developers may state that the target code to be reused will only
need a 10-20% modification when in reality some of it may require a
complete rewrite. One of the most common reasons for this development of
new code is to take advantage of operating system and hardware
enhancements. Another reason is that programmers who are new to the
project may recode in accordance with more modern software engineering
practices.

The uncertainty in the amount of effort for the reused code should be
reflected in the cost/schedule risk analysis. A range of estimates
should be generated that reflects the optimistic viewpoint and the
conservative viewpoint. A detailed analysis of the existing code should
be performed as early in the project as possible to ascertain the degree
of rework actually required. An accurate estimate of the reuse effort
will not be possible until the existing code is actually inspected by
the programmer/analysts.

#### Software Maintenance

After software has been developed, it must be maintained or supported.
Software maintenance involves much more than correcting errors not
discovered during testing. It also involves adding additional
capabilities, deleting obsolete capabilities, modifying the software to
address a change in the environment or to better interface with the host
computer, and other activities that are necessary after software is
developed. Therefore, software maintenance may also be called "software
redevelopment" since the activities are reiterations of all or some of
the software development phases shown in Figure 19-1.

Software maintenance is defined as the process of modifying existing
operational software while leaving its primary functions intact.
Software updates result in a changed functional specification for the
product but do not include major recoding of the software that
substantially change the functionality of the available software. Major
recoding is normally best treated as a separate, follow-on project.

According to Robert L. Glass and Ronald A. Noiseux in their book
*Software Management Guidebook*, almost all software maintenance can be
partitioned into three categories: corrective, adaptive, and perfective.
Table 19-2 briefly explains each category and gives the relative percent
of maintenance effort expended for each category. Note that corrective
maintenance, the closest analogy to traditional "maintenance", accounts
for only about 17% of the overall maintenance effort.

> Table 19-2. Software Maintenance

  ---------------- ------------------------------------- ----------------
  **Category**     **Definition**                        **Percentage**

  Corrective       Corrects error                        17%

  Adaptive         Accommodates environmental changes    18%

  Perfective       Makes enhancements                    60%

  Other            \-\--                                 5%
  ---------------- ------------------------------------- ----------------

Software maintenance is not inexpensive. According to L. H. Putnam and
A. Fitzsimmons in their article "Estimating Software Costs" and other
sources, software maintenance costs often exceed the total cost of
software development. Consequently, a discussion of software cost
estimation would be incomplete without considering maintenance costs
(see section 19.6.2). Therefore, in the subsequent discussions of
commercially available software cost models, an assessment of their
ability to estimate software maintenance costs is included.

## Software Cost Estimating Process

The overall process of developing a cost estimate for software is no
different from the process for estimating any other element of cost.
There are, however, aspects of the process that are peculiar to software
estimating. Some of the unique aspects of software estimating are driven
by the nature of software as a product. Many of the problems that plague
the software development effort itself are responsible for the
difficulty encountered in estimating that effort.

One of the first steps in any estimate is to understand and define the
system to be estimated. Software, however, is intangible, invisible, and
intractable. It is inherently more difficult to understand and estimate
a product or process that cannot be seen and touched. Software grows and
changes as it is written. Also, when hardware design has been
inadequate, or when hardware fails to perform as expected, the
"solution" is often attempted through changes to the software. This
change may occur late in the development process and sometimes results
in an unanticipated software "build." In this type of situation, where
the product is highly ambiguous, it is more important than ever to
develop graphical representations of the final software product. This
could be as simple as creating a product tree line drawing to represent
the software development effort, or developing a software **work
breakdown structure (WBS)**, which is discussed in section 19.3.1 and
19.3.3. Other problems are created by the metrics used to size software
and the nature of the software estimating methodologies themselves.
These topics are discussed in sections 19.4 and 19.5, respectively.

Per NRaD's Manual*,* projects should produce and document project plans,
which include estimates of product size, resources, staffing levels,
schedules, and key milestones. Historically, the costs and schedules for
most software projects have been greatly underestimated. There are many
reasons for this: costs and schedules are often pre-determined by an
outside source; a real in-depth analysis of the software development
process was not taken into consideration or in many cases not fully
understood; and there is a general lack of acceptance of the concept
that developing software is an expensive endeavor.

The software estimation process in the following sections describes the
steps required for establishing initial software **Life Cycle Cost
(LCC)** estimates and then tracking and refining those estimates
throughout the life of the project. Establishment of this process early
in the life cycle will result in greater accuracy and credibility of
estimates and a clearer understanding of the factors that influence
software development costs. This process also provides methods for
project personnel to identify and monitor cost and schedule risk
factors.

Following the sections on software cost estimating will be a discussion
of the software price and cost analysis process. Software pricing is
dependent upon most of the same input parameters that are required to
perform a valid cost estimate. The objective of price analysis is to
determine if the cost estimate contained in the contractor's proposal is
fair and reasonable. The objective of cost analysis is to analyze the
cost elements contained in the contractor's proposal to form an opinion
on the degree to which individual cost elements and associated proposed
profit or fee represent what the work should cost, given reasonable
economy and efficiency. It involves a review of the judgment factors
used in projecting the estimated costs. Therefore an understanding of
the software cost estimating process is necessary for performing
software pricing.

### Process for Software Estimation Activities

Referring to NRaD's Manual, software estimation is a continual process
that should be used throughout the life cycle of a project. The process
activity for developing the cost estimate is shown before the schedule
estimate in Figure 19-7 because this is the sequence often used by the
cost models. However, a development schedule is often mandated before
the scope of the effort is clearly understood. The early establishment
of a WBS helps to divide the effort into distinct work segments that can
be scheduled and prioritized. A sample detailed software WBS is provided
in Appendix 19C.

Figure 19-7. Software Cost Estimation Process

**Error! Not a valid link.**

The following subsections provide an overview of the steps used to
develop a thorough software project estimate. Detailed explanations of
estimation requirements for each phase of a project are in subsequent
sections. Detailed explanations of specific estimation methods and
automated tools are contained in section 19.5 and Appendix 19D
respectively.

STSC's Report states that software estimation should be approached as a
major process; it should be well planned, reviewed often, and
continually updated. The basic steps required to accomplish software
estimation are:

#### Step 1: Identify project objectives and requirements 

The key point that needs to be made here is that project objectives and
requirements must be clearly and precisely identified to develop good
estimates of effort and costs early in the project\'s life cycle. The
objectives of the project define what the end product is, what the end
product\'s intermediate steps are, and when and for whom the project
will be accomplished. The objectives also help define how and by whom
the project will be carried out. Constraints are restrictions that
affect the completion of project objectives. Constraints may arise from
several factors. The start date or completion of the project may impose
constraints. Other constraints are imposed by the application of
specific resources and their associated availability. Constraints can
also be policies and procedures that require justifications or
explanations of implementation actions.

NRaD's Manual states that defining requirements early on in a project
can be a very difficult task. However, without knowing what the
requirements are, it is impossible to accurately estimate a project\'s
cost and schedule. If all of the requirements are not known, then the
estimate should be based on the known requirements and make sure
everyone understands that the estimate is based on only those known
requirements. If using an incremental or evolutionary development
strategy, then base the estimate on the requirements that have been
defined for that increment.

#### Step 2. Plan the activities

The Work Breakdown Structure (WBS) is created in this step and is used
in the software estimation steps that follow. WBS is an excellent tool
for visualizing the software product. The WBS need not be complex, nor
does it need to be highly detailed. The hardware WBS can be a useful
tool in developing the initial WBS for software. There is usually a
software CSCI or similar module associated with each HWCI. As the
program evolves, the initial or draft WBS should include all software
associated with the program regardless of whether it is developed,
furnished, or purchased. If furnished or purchased software were
omitted, it would not be possible to capture the cost of integrating
pre-existing or purchased software with the development software.
Appendix 19C contains a sample software WBS.

During each of the phases of software development, numerous activities
will be performed in disciplines such as software development, software
project management, software configuration management, and software
quality assurance. As discussed by Boehm and by Donald J. Reifer in his
*SoftCost-R User\'s Manual*, the activities performed for each
discipline during each phase can be organized into an activity WBS for
each CSCI. This WBS can be used with the product WBS as a basis for
management reporting and tracking for a CSCI. A manager must understand
that a software WBS contains different activities than a traditional
hardware WBS (e.g., coding versus fabrication).

The WBS should depict only major software functions, and major
subdivisions. It should not attempt to relate the software to the
hardware it controls. Each of the major software functional units can be
modeled as a CSCI. Lower level WBS elements can be modeled as a
component. Once the WBS is established, the next step is to determine
the magnitude of the software development/support effort and decide on
which estimating technique should be used for deriving the estimate.

#### Step 3. Estimate product size

STSC's REPORT describes how using the requirements and WBS, the product
size can be estimated by estimating the sizes of the components
identified in the WBS. Estimating product size is the basis of software
cost estimation. Size is considered by software project managers to be a
major technical performance or productivity indicator allowing them to
track a project during development. Estimating product size is not
simple to do; typically, it is very dependent on the experience of the
persons doing the estimating. Since estimating size is so important to
the overall cost estimate, section 19.4 describes in detail the various
methods to estimate product size.

#### Step 4. Estimate Cost And Effort

Using size estimates as input, estimates of cost and effort are
prepared. Many methods can be used (see Section 19.5), and the use of
more than one method is strongly recommended. It is in this step that
software estimation tools should be used to assist with the estimates.
Many of the available tools use models that have been developed over the
years using historical data from hundreds, even thousands, of projects.
One widely used model is the **COnstructive COst MOdel (COCOMO)**. For
information on individual models see Appendix 19D.

In the NRaD Manual, the first consideration in estimating cost and
effort (man-hours or man-months) is to choose the estimation method. If
a parametric cost model (see Appendix 19D) is being used, then the
environmental parameters for the project must be determined and entered
into the model.

Examples of parameters are program complexity, programming language,
requirements volatility, analyst capability, and execution time
constraint. The estimate should include all labor activities charged
directly to the task. These activities normally include:

-   Engineering labor charges for System/Software Requirements Analysis,
    Design, Code, Test, and Integration.

-   Documentation effort

-   Configuration Management

-   Software Quality Assurance

-   Management effort charged directly to the task

#### Step 5. Estimate schedule

Using estimates of cost and effort, a tentative, projected schedule is
developed. Continuing with the NRaD Manual, Schedule Estimates can be
derived either manually or by using automated estimation models. A
combination of both manual and automated methods is recommended. Manual
methods should be based primarily on past experience. One or more
software engineers with experience with the specific application under
development should develop a schedule estimate as follows:

-   The WBS should be expanded to delineate the order in which
    functional elements will be developed. The order of development will
    define which functions can be developed in parallel as well as
    dependencies that drive the schedule.

-   A development schedule should be derived for each set of functions
    that can be developed independently, i.e., a schedule for each build
    of an incremental development.

-   The schedule for each set of independent functions should be derived
    as the aggregate of the estimated time required for each major phase
    of the development: analysis, design, code & unit test, integration
    and test.

-   The total project schedule should reflect the aggregate of the
    product development, including documentation and formal review
    requirements.

The steps outlined above are typical of manual estimates. Most of the
available tools provide a standardized schedule estimate based on
waterfall milestones shown in Figure 19-1. Automated tools allow the
user to tailor the schedule in order to observe the impact on cost.
However, most automated tools allow only a small amount of flexibility
in shortening schedules. REVIC (REVised Intermediate COCOMO), for
instance, allows the user to shorten the schedule by 25% from the
nominally derived schedule.

#### Step 6. Risk Assessment

The STSC Report states that all known risks associated with a software
development project should be defined and weighed, and impacts to the
project costs should be determined. This information should always be
included as part of the software estimation process. Poor software
estimates generally result from four major risk areas, which are shown
in Table 19-3.

Table 19-3. Risk Resulting from Poor Software Estimates

  --------------------------- -------------------------------------------
  **Risk Area**               **Factors Associated with Risk**

  Size of the software        Software size estimates tend to be
  project                     optimistic resulting in underestimation.
                              Since size estimates are central to other
                              project estimates, poor size estimates can
                              cause numerous problems such as cost and
                              schedule overruns.

  Development Environment and An inadequate development environment or
  Process Stability           changes in the environment or processes on
                              which estimates are based can result in
                              cost and schedule overruns.

  Staff Skills                Misalignment of skills to tasks can result
                              in miscalculations of schedules and amount
                              of effort required as well as poor
                              estimates of project staffing requirements.

  Requirements Growth         Unconstrained growth of requirements during
                              the software development life cycle results
                              in changing project goals that can lead to
                              frustration, customer dissatisfaction, and
                              ultimately, cost and schedule overruns.
  --------------------------- -------------------------------------------

Comparing and iterating the results from different estimation methods
will also help identify risks. More than one software estimation method
should be used and the results should be compared, since there is no one
method that is clearly superior. The strengths and weaknesses of the
different methods tend to complement each other. Significant differences
in cost/schedule estimates should be iterated for two reasons:

-   Similar components of a project may have widely varying estimates
    due to differences in the estimator's personalities (pessimistic vs.
    optimistic), roles, and incentives. Iteration of the estimates helps
    to resolve these differences.

-   A few components may dominate the SW costs, in which case, the
    estimates for those components are critical to the project's success
    and therefore can be risk areas. Iteration can be used to scrutinize
    the critical estimates.

Once the potential risks are identified, risk factors should be recorded
and tracked in the **Software Development Folder (SDF)** or **Software
Estimation File (SEF)**, the WBS and the **Software Development Plan
(SDP)**.

#### Step 7. Inspect/Approve

Referring to the NRaD Manual again, the purpose of this step is to
improve the quality of the estimate and get upper management commitment.
The objectives of the inspection and approval of the estimate are:

-   Confirm the software architecture and functional WBS.

-   Verify the methods used for deriving the size, schedule and cost
    estimates.

-   Ensure that the assumptions and input data used to develop the
    estimates are correct.

-   Ensure that the estimate is reasonable and accurate given the input
    data.

-   Formally confirm and record the official estimates for the project.

This step is an ideal place to incorporate the **Software Engineering
Institute (SEI)** software cost estimating requisites of audit trails
and constraint processes. Audit trails record and explain the values
used as cost model inputs. SEI suggests processes for dealing with
externally imposed cost or schedule constraints that help ensure the
integrity of the estimating process. Per the GAO *Report on Air Traffic
Control* in January 1997, SEI, along with other experts in systems and
software engineering advocate qualifying early project estimates by
disclosing the level of uncertainty associated with them, and making the
estimate more precise as the project is completed and the uncertainties
eliminated. The GAO recommended to the FAA to disclose the inherent
uncertainty and range of imprecision in all Air Traffic Control
project's official cost estimates presented to executive oversight
agencies or the Congress.

Software estimators, the project manager, and quality assurance or upper
management should sign the estimate after the inspection is complete and
all defects have been resolved. Inspection and approval activities can
be as formal or informal as you want them to be. What is important is
that the estimates are reviewed independently. Estimates can be improved
simply by having the appropriate personnel participate in the validation
process.

#### Step 8. Track Estimates

The NRaD Manual recommends tracking estimates over time in order to
develop a historical file of your estimates, to compare the current
estimate to previous estimates, to resolve any discrepancies with
previous estimates and to document the tracking data. Comparing planned
versus actual estimates over time allows the estimators to see how well
they are estimating and also to see how their project is changing over
time. Also, tracking allows the estimator to develop a historical
database of estimates. Estimators can use this historical database to
calibrate cost models or to compare past estimates to future projects.

This step allows the accomplishment of the SEI requisites of having a
corporate memory or historical database(s), for cataloging cost
estimates, revisions, reasons for revisions, actuals, and other
descriptive information. Examples of other descriptive information
includes: trends that affect the project; having the ability to
calibrate/tune cost models to reflect demonstrated accomplishments on
similar past projects; and data collection and feedback processes that
foster capturing and correctly interpreting data from work performed.

#### Step 9. Process Measurement and Improvement 

The NRaD Manual suggests that metrics should be collected during each
step of the software estimation process to improve the process. Two
forms of process measurement are recommended. The first, called process
effectiveness metrics, is used to track the effects of the process on
the project. The second set of metrics, called process cost metrics, is
used to provide management with insight into the cost of implementing
and performing this process. The long- term benefit of collecting these
metrics is to determine a correlation between the overall accuracy of
the estimates and the cost of developing the estimates. This information
is another input to the resource requirements of a project and should be
identified as such.

The collection of these metrics should begin as soon as the process is
implemented and will continue throughout the development cycle.
Collection of the metrics will continue through the final phases of
development even though estimation activity declines in those phases.
These metrics are described below.

-   **Process Effectiveness Metrics.** The purpose of the Process
    Effectiveness Metrics is to identify the elements of the estimation
    process that enhance the estimation process and those elements that
    are of little or no value to the planning and tracking processes of
    a project. Process elements or activities that enhance the process
    are those that provide the greatest accuracy regarding the actual
    cost and schedule, as well as cost and schedule to completion of a
    project. Conversely, those elements or activities that do not
    enhance the accuracy of the estimates need to be isolated and
    eliminated.

> The Process Effectiveness Metrics consist of the variances between the
> most recent estimates and the baseline estimates. A format for
> recording the data items for which the variances are recorded is shown
> in Table 19-4 below.

Table 19-4. Process Effectiveness Metrics

  ----------------------------------- ------------ ---------- ------------
                                      Baseline     Update     Variance

  Size Estimate: New Code                                     

  Size Estimate: Reused Code                                  

  Size Estimated: Modified Code                               

  Productivity (Hours/SLOC)                                   

  Cost Estimate                                               

  Schedule Estimate                                           

  Number of CSCIs                                             

  Number of CSUs                                              

  Documentation page count                                    
  ----------------------------------- ------------ ---------- ------------

Section 19.10 goes into greater detail concerning metrics for software
tracking and measurement. Section 19.7.5 discusses the importance of
tracking productivity and code condition (code size by percent new,
reused, modified) when pricing to insure consistency between program
size and proposed effort/schedule.

-   **Process Cost Metrics.** The purpose of the Process Cost Metrics is
    to quantify the cost of the software cost/schedule estimating
    process and identify ways to increase the cost effectiveness of the
    process. Elements or activities that cost-effectively enhance the
    project planning and tracking process will remain intact while those
    that are of little or no value to the planning and tracking
    processes of a project will be eliminated. Process elements or
    activities that are cost effective are those that provide meaningful
    input to the planning and tracking process with a minimum amount of
    work effort. Conversely, those elements or activities that require
    effort but do not return meaningful data need to be isolated and
    eliminated.

> The elements and definitions of the Process Cost Metrics are divided
> into two categories: effort to perform the estimates; and the cost of
> tools. The effort metrics include the person hours for Size Estimates
> and Cost/Schedule Estimates. The cost of the tools is a summation of
> the following:

-   Purchase Price

-   Training Cost

-   License Renewal

-   Special Hardware

-   Consulting Fees

> These metrics should be collected for each estimate developed.

### Software Development Standards

As stated earlier, very often the government requires software
development to follow DoD-STD-2167A/498, which is the Department of
Defense standard that specifies the overall process for the development
and documentation of mission critical software systems. This standard
also requires technical reviews and audits to be conducted in accordance
with DoD-STD-1521B.

There are many non-DoD standards in use currently that fall into the
general category headings of Independent (IEEE, ISO), Commercial (ANSI
J-STD-016, Commercial High/Low, IS, MIS) and Project (Space Shuttle).
The SEER-SEM User's Manual provides a good overview of the definitions
and uses of many of these standards. The Cost Xpert *User's Manual*
contains matrices listing the title and description of the various
documents required by many of these standards.

Other standards that may affect the estimating process are:
MIL-STD-499A, Engineering Management; MIL-STD-490A, Specification
Practices; MIL-STD-480B, Configuration Control-Engineering Changes,
Deviations and Waivers; DoD-STD-1703, Software Products Standards.
Software developed in accordance with these standards generally requires
more resources and is more time consuming. Therefore, the software
estimation process must include and adjust for these requirements where
applicable.

### Benefits of SW Estimation Process

When the software estimation process is performed correctly, the
benefits realized far outweigh the cost of performing an estimate. Some
of the major benefits include lowering the cost of doing business,
increasing and broadening the skill-level of key staff members,
acquiring a deeper knowledge of the proposed project prior to beginning
the software development effort, and understanding, refining, and
applying the proper software life cycle mode.

As these software estimating components are enhanced, refined, and
continually applied, the software estimating process, associated tools,
and resulting products attain higher levels of quality and ultimately
benefit all.

## Estimating Software Size

Software size is a key input to most software cost estimating
methodologies. However, it is not always easy to determine size,
especially during the early stages of a program. Currently, there are
numerous tools available for size estimation; many of which are included
in, or related to, currently available software cost models. Current
software cost models usually accept input from more than one sizing
approach. An SEI requisite is that there needs to be a structured
approach for estimating size and the amount and complexity of existing
software that can be reused. This section will describe the various
methods for software size estimating, including reused and COTS
software.

The Cost Xpert *User's Manual* recommends that for small projects
(generally, less than \$50K) typically either a bottoms-up or top down
estimate is used to generate a size estimate. For medium sized projects
(\$50K to \$1M), probably one of the metric based approaches should be
employed (lines of code, function points, or object metrics). Analysts
can supplement this estimate with one of the two heuristic approaches
(bottoms-up or top down) in order to validate the numbers. For larger
projects, it is recommended to use two or more of the metric based
approaches and correlate the results.

### Heuristic Approaches (\"Rule of Thumb\")

Although this section will concentrate on metric based approaches to
size estimation, the two heuristic approaches will be briefly discussed
first for completeness. The Cost Xpert *User's Manual* is referred to
for the remainder of the Heuristic subsection. Heuristic approaches to
estimating the program size rely on the fact that the same basic
activities will be required for a typical software development project
and that these activities will require a predictable percentage of the
overall effort. For example, traditional projects typically include
activities that can be categorized as requirements analysis, design,
coding, testing, and documentation. The percentage of effort required
for these activities is predictable, varying based on the type of
project and the size of the project. This knowledge is the basis of
heuristic based estimating, whereas metric based approaches rely on a
correlation between the metric being employed and the ultimate effort
involved in building the software.

#### Bottoms-Up

For the bottoms-up approach, divide the program into modules. For each
module, directly estimate the level of effort required in terms of
person months or person hours. This provides the estimated total effort
for the coding portion of the development. Quite often, this portion of
the estimate relies heavily on input from the actual programming team.

Then look at the other life cycle activities that are required. This
list of activities will vary based on the life cycle standard being used
during this development effort, plus there may be some project specific
activities. For each of these activities, estimate the effort required.

Finally, compare the percentages of effort for each activity in the
project plan with standard percentages. If the percentages roughly match
the standard percentages, then there is a relatively high degree of
confidence that the allocation of effort between activities is
reasonable. If the percentages deviate from the standard percentages
significantly, this may indicate an unreasonable amount of effort,
either too high or too low, for one or more activities.

In summary for bottoms-up cost estimating, start with a list of all
required activities, estimate the effort for each, compare these
estimates with typical percentages, then roll the costs up to a total
project cost.

#### Top Down

For top down estimating, begin with an estimate of the total effort for
the entire project. Then use typical percentages for each life cycle
activity to calculate out the effort for each of those activities. Then
compare the predicted effort for each activity with the required work to
determine if the numbers seem reasonable. If not, the total effort can
be adjusted up or down as required.

### Metric Based Models

Three currently popular-metric based approaches for estimating software
size are **source lines of code (SLOC)**, function points and object
points. The Cost Xpert *User's Manual* provides a good summary of the
strengths for each of the different approaches for metric based size
estimates. Lines of code are particularly valuable for real time and
embedded systems with little user interface. This was the original
metric based approach and was popularized by COCOMO. The basic COCOMO
model has been incorporated into literally dozens of costing tools, and
remains the most common approach.

Function points were introduced by IBM in the early 1980s. This approach
employs user interface features to estimate program size. It is the most
common technique for estimating **management information system (MIS)**
application size. Object metrics have become feasible only with the
popularization of object oriented development and use objects as a
predictor of program size.

For these metric based approaches, there are several estimating models
and techniques available. A summary of software sizing approaches with
appropriate examples follows.

#### Source Lines of Code (SLOC)

Per STSC's Report, estimating SLOC is the traditional and most popular
technique used to estimate the size of a software project. The appeal of
estimating SLOC is that it is a simple technique, at least in concept,
and often is automated. However, because determining which lines of code
to count can be ambiguous, rules are needed for which types of lines to
count in a

programming language. Two of the difficulties with using SLOC as an
estimate in project planning are:

-   SLOC cannot be estimated reliably in the early phases of the
    development cycle unless SLOC data are available from similar,
    completed projects.

-   The work content of a line of code varies widely among higher-order
    programming languages.

Boehm in his briefing "An Overview of the COCOMO.II Software Cost Model"
stated that defining a line of code is difficult due to conceptual
differences involved in accounting for executable statements and data
declarations in different languages. To minimize these problems, the SEI
definition checklist for a logical source statement was used in
COCOMO.II for defining the line of code measure. (COCOMO.II refers to
version 2.0 released in 1997.) Figure 19-8 shows a portion of the
definition checklist as it is being applied to support the development
of the COCOMO.II model. Each check mark in the \"Includes\" column
identifies a particular statement type or attribute included in the
definition, and vice-versa for the excludes. Other sections in the
definition clarify statement attributes for usage, delivery,
functionality, replications and development status.

Figure 19-8. Source Lines of Code Sizing

  -- -------------- ----------------- -------------- ------------ --------- ------------- ---- ---------- ---------- --
     **Definition                                                                                                    
     Checklist for                                                                                                   
     Source                                                                                                          
     Statements                                                                                                      
     Counts**                                                                                                        

                    Definition name:  Logical Source                        Date:                                    
                                      Statements                                                                     

                                      (basic                                Originator:        COCOMO 2.0            
                                      definition)                                                                    

                    **Measurement     **Physical                                                                     
                    unit:**           source lines**                                                                 

                                      **Logical                             **4**                                    
                                      source                                                                         
                                      statements**                                                                   

     Statement type                   Definition     ****        Data                         Includes   Excludes   
                                                                  array                                              

                    When a line or                                                                                   
                    statement                                                                                        
                    contains more                                                                                    
                    than one type,                                                                                   

                    Classify it as                                                                                   
                    the type with the                                                                                
                    highest                                                                                          
                    precedence.                                                                                      

     1              Executable                       Order of                             1    ****                 
                                                     precedence                                                      

     2              Nonexecutable                                                                                    

     3              Declarations                                                          2    ****                 

     4              Computer                                                              3    ****                 
                    directives                                                                                       

     5              Comments                                                                                         

     6              On their own                                                          4               ****      
                    lines                                                                                            

     7              On lines with                                                         5               ****      
                    source code                                                                                      

     8              Banners and                                                           6               ****      
                    non-blank spacers                                                                                

     9              Blank (empty)                                                         7               ****      
                    comments                                                                                         

     10             Blank lines                                                           8               ****      

     11                                                                                                              

     12                                                                                                              

     How Produced                     Definition     ****        Data                         Includes   Excludes   
                                                                  array                                              

     1              Programmed                                                                 ****                 

     2              Generated with                                                                        ****      
                    source code                                                                                      
                    generators                                                                                       

     3              Converted with                                                             ****                 
                    automated                                                                                        
                    translators                                                                                      

     4              Copied or reused                                                           ****                 
                    without change                                                                                   

     5              Modified                                                                   ****                 

     6              Removed                                                                               ****      

     7                                                                                                               

     8                                                                                                               

     Origin                           Definition     ****        Data                         Includes   Excludes   
                                                                  array                                              

     1              New work: no                                                               ****                 
                    prior existence                                                                                  

     2              Prior work: taken                                                                                
                    or adapted from                                                                                  

     3              A previous                                                                 ****                 
                    version, build,                                                                                  
                    or release                                                                                       

     4              Commercial,                                                                           ****      
                    off-the-shelf                                                                                    
                    software (COTS),                                                                                 
                    other than                                                                                       
                    libraries                                                                                        

     5              Government                                                                            ****      
                    furnished                                                                                        
                    software (GFS),                                                                                  
                    other than reuse                                                                                 
                    libraries                                                                                        

     6              Another product                                                                       ****      

     7              A vendor-supplied                                                                     ****      
                    language support                                                                                 
                    library                                                                                          
                    (unmodified)                                                                                     

     8              A vendor-supplied                                                                     ****      
                    operating system                                                                                 
                    or utility                                                                                       
                    (unmodified)                                                                                     

     9              A local or                                                                            ****      
                    modified language                                                                                
                    support library                                                                                  
                    or operating                                                                                     
                    system                                                                                           

     10             Other commercial                                                                      ****      
                    library                                                                                          

     11             A reuse library                                                            ****                 
                    (software                                                                                        
                    designed for                                                                                     
                    reuse)                                                                                           

     12             Other software                                                             ****                 
                    component or                                                                                     
                    library                                                                                          

     13                                                                                                              

                                                                                                                     
  -- -------------- ----------------- -------------- ------------ --------- ------------- ---- ---------- ---------- --

Some examples of SLOC estimating models in each model category are
summarized in the following sections.

##### Analogy Models

These models estimate program size by comparison with one or more
similar programs. Currently popular are database analogy models, such as
the Aerospace Software Size Estimator as discussed by Marilee J. Wheaton
in her article on "Functional Software Sizing Methodology". This model
contains over 1,000 historical programs, primarily in the military
space, avionics, and ground-based environments. The user can specify his
or her specific application category and obtain a list of historical
projects with sizes and languages. The user can further specify a subset
of an application, such as mission control. This will result in fewer
similar programs, but the programs will be more relevant.

Analogy sizing models possess the advantage of using historical data for
comparison. They are also useful very early in a program. However, they
are of limited use for any programs outside of the historical database.
For example, the Aerospace model would be ineffectual for business
applications.

##### Functionality (Grass Roots) Models

These models compute SLOC from components of known size from a database.
If the user knows what functions a program will perform, they can obtain
a size estimate for each function and sum the function estimates to
estimate the size of a program. The Software Architecture, Sizing, and
Estimating Tool (SASET), as discussed in the *SASET 3.0 User\'s Guide*
by Robert W. Ratliff, et. al., allows size to be input based on
functionality. The user specifies the functions performed by the
program, and the model obtains a SLOC estimate for each function from
the database. For example, if the user specifies a telemetry processing
function, the model provides an estimate of 5,950 HOL statements. The
Quantitative Software Management (QSM) Size Planner, defined in QSM
Corporation's *Size Planner User\'s Manual*, has a "standard components"
option which also estimates the size of each function specified,
primarily for business applications. For example, if the user specifies
a screen function, the model provides an estimate of 2,535 lines of
COBOL code.

Like analogy models, functionality sizing models estimate size based on
historical data. In contrast to traditional bottoms-up models, they are
often useful early in a program since many functions are defined during
software requirements analysis or preliminary design. However, like
analogy models, these models are database dependent; similar functions
with historical sizes must be available. A further limitation of models
such as SASET is that the language is not specified. This can be
problematic since there are significant differences among HOLs.

##### Expert Judgment Models 

These models estimate SLOC based on the opinions of one or more experts.
A simple example of an expert judgment technique is a **Program
Evaluation and Review Technique (PERT)** for sizing used in several cost
models. A more refined example is the SEER Software Sizing Model as
discussed in the SEER-SSM User's Guide, which is based on an expert
judgment model developed by George Bozoki. In SEER-SSM, the user
specifies at least one "reference" program, then performs a number of
"structured judgments" to determine the size of the new program(s). The
user performs a pair-wise comparison of program sizes, inputs a PERT
estimate for all new programs, selects the most probable size range from
a set of ranges generated by the model, and performs a ranking to verify
the pair-wise comparison made initially.

In the sizing arena, expert judgment models are either used alone or as
aids in determining inputs for other categories of models. Expert
judgment models have a disadvantage in that the views of experts are
often biased. Despite their subjectivity, however, expert judgment
models are often the only choice for new or unique situations.

##### Parametric Models

These models use inputs consisting of numerical or descriptive values to
compute program size in SLOC. They are developed using regression
analysis and other methods, such as numerical analysis or management
surveys. One subset of parametric sizing models are regression-based
models. These are similar to database analogy models in that historical
data are used in model development. Instead of using direct comparisons,
however, these models use equations, sometimes called **size estimating
relationships (SERs)** developed from regression analysis of the
historical data. Minoru Ikatura and Akio Takayanagi, in their paper "A
Model For Estimating Program Size and Its Evaluation" developed a
regression-based SER for business programs which has the necessary high
correlation for a valid equation. While regression-based parametric
sizing models transcend the need for direct comparison, they are usually
limited to the scope of the database used. Furthermore, as illustrated
by Mark J. Whetstone's research and later by Frank Albanese Jr., it is
often difficult to obtain the requisite high correlation for model
validity. The Ikatura and Takayanagi SER is one of the few successful
efforts.

Another example of a parametric sizing model is the PRICE-S Sizer, which
is included in the *PRICE-S Reference Manual*. The PRICE-S Sizer was not
developed solely from regression analysis; other techniques were also
used. This model has multiple inputs plus a size calibration factor,
SICAL, which can be used to adapt the model to the user\'s environment.

Parametric sizing models generally possess the advantages noted for the
PRICE-S Sizer model and for parametric models in general. They are fast
and easy to use and are generally useful early in a program, but may be
challenging to use for new or unique programs.

#### Function Point 

Function point is a method of estimating size during the requirements
phase based on the functionality to be built into the system. Referring
to the STSC Report, using function points as a basis for size estimates
grew out of the difficulties with SLOC mentioned above. Function points
are derived by:

-   First, counting the number of external inputs, inquiries, outputs,
    master files, and interfaces;

-   Second, adjusting the counts for complexity; and

-   Third, summing the results into a **Function Point Count (FPC)**.

The FPC is independent of the programming languages used on the project.
Tables of source statements per function point have been developed
through research for most programming languages and major dialects. This
technique, known as **Function Point Analysis (FPA)**, grew out of
estimating the size of **Information System (IS)** software development
projects.

FPA has proven useful because:

-   It is based on information that is available early in a project\'s
    life cycle.

-   It can be used on projects with multiple programming languages and
    multiple platforms.

Criticisms of using FPA are:

-   It is hard to use.

-   It does not measure mathematical optimization, various embedded
    systems, scientific and real-time process control applications as
    well as it measures information systems.

Function point sizing models were popularized by the extensive research
of A. J. Albrecht and J. E. Gaffney for data processing programs written
in COBOL and PL/1 programming languages as documented in their article
"Software Function, Source Lines of Code, and Development Effort
Prediction: A Software Science Validation." They are parametric models
in that they use design parameters to estimate size; however, the
parameters were not developed from regression analysis. Instead,
Albrecht hypothesized that five program functions; inputs, outputs,
inquiries, interfaces, and master files; could be used to estimate size.
The researchers performed function point analysis on more than thirty
data processing programs and concluded that function points are not only
a valid predictor of software size, but were also superior to SLOC as a
predictor of software development cost or effort. The CHECKPOINT model,
as discussed in Software Productivity Research, Inc.'s *CHECKPOINT For
Windows User\'s Guide*, employs function points as a primary input for
effort and schedule estimation. Other models such as PRICE-S, Cost Xpert
and QSM Corporation's **Software Life Cycle Model (SLIM)** allow
function points as an alternative to SLOC for program size.

A brief definition of the five attributes used for function points, as
described by Donald J. Reifer in the *ASSET-R Version 2.0 User\'s
Manual*, is shown in Table 19-5.

Table 19-5. Function Point Attributes

  ----------------- -----------------------------------------------------
  **Attributes**    **Definition**

  External inputs   All unique data or control inputs that cross the
  (EI)              system boundary and cause processing to occur.
                    Examples are input screens and tables.

  External outputs  All unique data or control outputs that cross the
  (EO)              system boundary after processing has occurred.
                    Examples are output screens and reports.

  External          All unique transactions which cross the system
  inquiries (EQ)    boundary to make active demands on the system.
                    Examples are prompts and interrupts.

  Internal files    All logical groupings of data that are stored within
  (ILF)             a system according to some pre-defined conceptual
                    schema. Examples are databases and directories.

  External          All unique files or programs that cross the system
  interfaces (EIF)  boundary and are shared with at least one other
                    system or application. Examples are shared databases
                    and shared mathematical routines.
  ----------------- -----------------------------------------------------

CASE STUDY 19-1, extended from SSCAG's *Software Methodology Handbook*
shows how traditional function points (sometimes called Albrecht
function points) are computed. The user must determine the number of
**external inputs (EI), external outputs (EO), external inquiries (EQ),
internal files (ILF)**, and **external interfaces (EIF)** in the
program. This will give a measure of \"basic\" function points. The user
can refine this measure by considering the complexity level of each
function point and fourteen complexity adjustments factors related to
the overall program, as shown in CASE STUDY 19-1.

CASE STUDY 19-1. Function Point Sizing Equations

-   **"Basic" Function Points:** BFP = 4 EI + 5 EO + 4 EQ + 10 ILF + 7
    EIF

> (With + / - 25% Complexity Adjustment)

-   **Unadjusted Function Points (UFP):** Weight Five Attributes as
    Simple, Average, or Complex

  --------------- -------------- ------------- ------------- -------------
                  **Complexity                               
                  Weighting**                                

  **Attribute**   **Simple**     **Average**   **Complex**   **Total**

  EI              3              4             6             

  EO              4              5             7             

  EQ              3              4             6 (or 7)      

  ILF             7              10            15            

  EIF             5              7             10            
  --------------- -------------- ------------- ------------- -------------

-   **Adjusted Fraction Points (FP):** UFP (0.65 + \[0.01 x CA\])

> (*CA is Complexity Adjustment; Sum of 14 Factors, Rated 1 to 5 for
> Influence)*
>
> *\[0 - None, 1- Little, 2 - Moderate, 3 - Average, 4 - Significant,
> 5 - Strong\];*
>
> *Ratings Defined for Each Factor)*
>
> **[14 Factors]{.underline}**

  ------------------------------------ ----------------------------------
                                       

  Complex Processing                   Multiple Site Usage

  Conversion and Installation Ease     On-Line Data Entry

  Data Communications                  On-Line Update

  Distribution Data Processing         Operational Ease

  End-User Efficiency                  Performance Objectives

  Facilitate Change                    Reusability

  Heavily Used Configuration           Transaction Rate
  ------------------------------------ ----------------------------------

A marked strength of function points is the excellent results obtained
from the research of Albrecht and Gaffney. Ongoing research is being
performed by the **International Function Points User\'s Group
(IFPUG)**, which meets twice a year and periodically publishes a guide
to counting and using function points. Proponents of function points
models also state that function point counts can be made early in a
program; during requirements analysis or preliminary design. Another
strength, according to Capers Jones, is that they give a more realistic
measure of productivity; SLOC-per-person-month measures tend to penalize
HOLs. However, function points have disadvantages compared to SLOC as a
measure of size. They are often harder to visualize since a SLOC can be
seen (e.g., on a code listing) while a function point is only a concept.
Furthermore, function points have only been extensively studied for
business or data processing applications, although attempts to adapt the
function point concept to real-time and scientific environments have
been made. Some function points issues are variants, real-time
adaptations, and function point-to-SLOC conversion.

##### Function Point Variants 

There have been several attempts to modify the traditional or IFPUG
function points shown in CASE STUDY 19-1. In the CHECKPOINT model, for
example, the fourteen complexity adjustment factors are not normally
used (although the user has the option of using them). Instead the user
rates the overall complexity of the problem, code, and documentation
from \"l \" to \"5\", and these complexities are used to compute
adjusted function points. The function points used in the QSM Size
Planner also do not use complexity adjustment factors; however, they use
five attribute ranges instead of three. In SEER-SEM, the user can use
either IFPUG or Galorath function points instead of SLOC. Galorath
function points add a sixth attribute, internal functions, to the five
attributes used in traditional function points. Probably the most
radical departure from traditional function points is Symon\'s Mark II
Function Points. Mark II function points use only three attributes;
inputs, outputs, and entities. An entity is any item about which the
system provides information. Mark II function points were standardized
in England.

##### Real Time Adaptation

There have been several attempts to adapt function points to scientific
and real-time programs. Some of these endeavors are now briefly
described.

##### Capers Jones\' Feature Points

For some time, function point practitioners recognized that some classes
of applications did not profit from function point analysis as much as
they hoped. These application classes included real-time process
control, mathematical optimization and various embedded systems. Feature
points add a sixth attribute, algorithms, to the five used in
traditional function points. According to Jones, an algorithm is a \"set
of rules which must be completely expressed in order to solve a
computational problem\". The basic feature points (BFP) equation is: BFP
= 3AL + 4EI + 5EO + 4EQ + 7ILF + 7 EIF. Note that, compared to the basic
function points equation in Case Study 19-1, the coefficient for ILF has
been decremented by three; all others remain the same. The CHECKPOINT
model includes a complexity adjustment for algorithms.

##### ASSET-R Function Points 

The ASSET-R model uses three additional attributes to those used in
traditional function points: operating modes, rendezvous, and
stimulus/response relationships. Operating modes are time-dependent
end-to-end processing flows to which software performance can be
related. Rendezvous function points are a measure of concurrency in
real-time systems; and stimulus/response relationships measure the
amount of sequencing and control in real-time systems. For scientific
programs, ASSET-R only adds operating modes to the five additional
attributes. Unlike traditional function points and feature points,
ASSET-R does not weigh attributes differently; it assigns a coefficient
of \"l\" to all attributes and uses other inputs to adjust the total for
complexity. The ASSET-R manual shows that, for a sample of 34 data
processing, scientific, and real-time programs, the model was accurate
to within 20% of actual SLOC values based on software requirements
specification data.

##### Real-time SLOC from Externals

Gaffney and Wehling studied the feasibility of estimating SLOC for
real-time programs with the four \"external\" attributes of traditional
function points: EI, EO, EQ, and EIF. They also studied this issue with
only three of the external attributes: EI, EO, and EQ. Their sample
involved nineteen military real-time programs, mostly written in
FORTRAN. The resultant equations relating \"externals\" to SLOC showed a
correlation of .89 when four externals were used, and .87 when three
externals were used.

According to the "Function Point FAQ" (Frequently Asked Questions)
homepage, when the function point technique was originally developed, it
was not applied to systems with **Graphical User Interfaces (GUIs)**,
since there were no GUIs. In 1995, IFPUG completed Release 4.0 of their
*Function Point Counting Practices Manual*. It contains the official
rules for, and extensive examples of, counting GUI based systems.
Subsequently, IFPUG produced a case study that contained the count of an
entire GUI-based system.

Function points can be used to measure object oriented development
applications and their size is the same as if object oriented
development had not been used. This is because the function point count
for an application is independent of the technology used to develop the
application.

#### Function Point-to-SLOC Conversion

Referring to the SSCAG Handbook*,* it is sometimes necessary to convert
from SLOC to function points, or vice-versa. For instance, consider that
a user of SLIM only knows the number of function points in his or her
program but does not know SLOC. According to QSM Corp's SLIM 3.0 for
Windows User Manual, since the SLIM algorithms are based on SLOC that
user must specify a ratio or "gearing factor", to convert the number of
function points to the number of SLOC. The opposite situation occurs in
CHECKPOINT where SLOC inputs must be converted to function points. To
help in this conversion, sets of SLOC-to-function point ratios have been
developed for various languages based on research by Jones, Reifer, and
Dan Galorath and Karen McRitchie in their paper "Software Size Measures:
Source Lines of Code and Functions." Jones also specifies language
levels that, in effect, show the number of equivalent assembler language
SLOC generated by one SLOC of the specified language. Jones further
states that language levels are useful for converting size from one
language to another, and for assessing relative language productivities
(although the relationship between language level and productivity is
not linear).

While function point-to-SLOC conversion ratios are useful, and sometimes
necessary, they must be used with caution. While the researchers cited
above agree on the ratios for some languages such as Ada, they differ,
sometimes significantly, on the ratios for others such as Pascal and
PL/l. For the COBOL language, research by Garland S. Henderson, in his
thesis *The Application of Function Points to Predict Source Lines of
Code for Software Development* showed that ratios are inconsistent both
within and among databases. For two databases maintained at the Air
Force Standard Systems Center, Henderson found that, for the military
database of 26 COBOL programs, the average SLOC-to-function point ratio
was 14:1, while for a database of 31 commercial COBOL programs, the
average SLOC-to-function point ratio was 165:1. Furthermore, there was
considerable variance for these ratios within the databases, and the
results differ significantly with the ratios developed by Jones, Reifer,
and Galorath. For some languages, therefore, it appears that ratios
should not be used and, for cost estimation, it may be best to use a
model for which the algorithms are based on the user\'s size measure.
Table 19-6 compares the FP to SLOC ratios for some sample languages.

Table 19-6. Function Point to SLOC Conversion

  -------------- ------------------ ------------- ------------- ----------------
  **Language**   **Jones Language   **SLOC/FP**   **Reifer      **Galorath
                 Level**                          SLOC/FP**     SLOC/FP**

  Assembler      1.0                320           400           320

  COBOL          3.0                105           100           58

  FORTRAN        3.0                105           105           61

  Ada            4.5                71            72            73

  PROLOG         5.0                64            64            N/A

  Pascal         3.5                91            70            71

  PL/1           4.0                80            65            71
  -------------- ------------------ ------------- ------------- ----------------

#### Object Points 

With the advent of **Object Oriented (OO)** programming/development
(Software programs are built from prefabricated, pretested building
blocks (objects) combined together to produce a fully written software
application. The idea of OO programming is to make the writing of
complex computer software easier.), there has been a growing demand for
new ways to measure software size in an OO environment.

Per the Cost Xpert *User's Manual*, object metric based estimating is
seeing an increased following among companies using object oriented
techniques throughout the software life cycle (e.g., object oriented
analysis, design, and programming). Although still relatively new, it
was applied on a massive scale within Hewlett Packard with great
success. One very nice thing about the object metric approach is that it
works equally well with both MIS type applications and with the
embedded, scientific, and system level programming projects that are not
well served by function points. Of course, a disadvantage is that it
only works if the software development is done using object oriented
techniques.

According to the STSC Report, objects typically are screens, reports,
and third generation modules. (By definition, they are not directly
related to \"objects" in an object-oriented methodology.) Getting the
Object Point Count is very similar to getting the FPC described above;
adjustments are made to the raw count for complexity and summed to get a
final count. The advantage of counting object points is that it usually
takes less effort than function points and the results are comparable.

CASE STUDY 19-2 (extended from the "COCOMO.II Software Cost Estimation
Model" abstract) presents the baseline COCOMO.II Object Point procedure
for estimating the effort involved in prototyping projects. Definitions
of terms are provided in Table 19-7.

Table 19-7. Terms

  ------------ -----------------------------------------------------------
  **Term**     **Definition**

  **NOP**      New Object Points (Object Point count adjusted for reuse)

  **Srvr**     Number of server (mainframe or equivalent) data tables used
               in conjunction with the SCREEN or REPORT.

  **Clnt**     Number of client (personal workstation) data tables used in
               conjunction with the SCREEN or REPORT.

  **%reuse**   The percentage of screens, reports, and 3GL modules reused
               from previous applications, pro-rated by degree of reuse.
  ------------ -----------------------------------------------------------

Note that the use of the term \"object\" in \"Object Points\" defines
screens, reports, and 3GL modules as objects. This may or may not have
any relationship to other definitions of \"objects\", such as those
possessing features such as class affiliation, inheritance,
encapsulation, message passing, and so forth. Different counting rules
for \"objects\" of that nature are required when used in languages such
as C++.

CASE STUDY 19-2. Steps for Object Point Estimate

-   **Assess Object-Counts.** Estimate the number of screens, reports,
    and 3GL components that will comprise this application. Assume the
    standard definitions of these objects I in your ICASE environment.

-   **Classify each object instance into simple, medium and difficult
    complexity levels depending on the values of characteristic
    dimensions.** Use the following scheme:

+--------+-------+--------+-------+--------+-------+--------+-------+
| **For  |       |        |       | **For  |       |        |       |
| Scr    |       |        |       | Scr    |       |        |       |
| eens** |       |        |       | eens** |       |        |       |
+========+=======+========+=======+========+=======+========+=======+
|        | **\#  |        |       |        | **\#  |        |       |
|        | and   |        |       |        | and   |        |       |
|        | s     |        |       |        | s     |        |       |
|        | ource |        |       |        | ource |        |       |
|        | of    |        |       |        | of    |        |       |
|        | data  |        |       |        | data  |        |       |
|        | tab   |        |       |        | tab   |        |       |
|        | les** |        |       |        | les** |        |       |
+--------+-------+--------+-------+--------+-------+--------+-------+
| **     | **    | *      | **    | **     | **    | *      | **    |
| Number | Total | *Total | Total | Number | Total | *Total | Total |
| of     | \<    | \< 8   | 8     | of     | \<    | \< 8** | 8**   |
| Views  | 4**   | (2/3   | +(\>  | Se     | 4**   |        |       |
| Conta  |       | srvr   | 3     | ctions |       | **(2/3 | *     |
| ined** | **(\< | 3-5    | srvr  | Conta  | **(\< | srvr   | *+(\> |
|        | 2     | c      | \> 5  | ined** | 2     | 3-5    | 3     |
|        | srvr  | lnt)** | cl    |        | srvr  | c      | srvr  |
|        | \< 3  |        | nt)** |        | \< 3  | lnt)** | \> 5  |
|        | cl    |        |       |        | cl    |        | cl    |
|        | nt)** |        |       |        | nt)** |        | nt)** |
+--------+-------+--------+-------+--------+-------+--------+-------+
| \< 3   | S     | Simple | m     | 0 or 1 | s     | simple | M     |
|        | imple |        | edium |        | imple |        | edium |
+--------+-------+--------+-------+--------+-------+--------+-------+
| 3 - 7  | S     | Medium | diff  | 2 or 3 | s     | medium | Diff  |
|        | imple |        | icult |        | imple |        | icult |
+--------+-------+--------+-------+--------+-------+--------+-------+
| \> 8   | M     | Dif    | diff  | 4 +    | m     | dif    | Diff  |
|        | edium | ficult | icult |        | edium | ficult | icult |
+--------+-------+--------+-------+--------+-------+--------+-------+

-   **Weigh the number in each cell using the following scheme.** The
    weights reflect the relative effort required to implement an
    instance of that complexity level.:

  -------------------------- ----------------------- -------------- ----------------
                             **Complexity-Weight**                  

  **Object Type**            **Simple**              **Medium**     **Difficult**

  Screen                     1                       2              3

  Report                     2                       5              8

  3GL Component                                                     10
  -------------------------- ----------------------- -------------- ----------------

-   **Determine Object-Points.** Add all the weighted object instances
    to get one number, the Object-Point count.

-   **Estimate percentage of reuse expected to be achieved in this
    project.** Compute the New Object Points to be developed,

> NOP = (Object
> Points)$´\frac{(\text{100} - \text{\%}\text{reuse})}{\text{100}}$

-   **Determine a productivity rate.**

> PROD = NOP/person-month, from the following scheme.

  ------------------------------ --------- ----- --------- ------- ----------
  **Developers\' expertise and   Very Low  Low   Nominal   High    Very High
  capability**                                                     

  **ICASE maturity and           Very Low  Low   Nominal   High    Very High
  capability**                                                     

  **PROD**                       4         7     13        25      50
  ------------------------------ --------- ----- --------- ------- ----------

-   **Compute the estimated person-months.**

> PM = NOP/PROD.

### Reused Code 

According to NRaD's Manual reused code is considered in most current
software cost models. The only way to estimate the amount of code to be
reused is for the programmer/analysts to examine the existing code in
detail. The examination should derive estimates for three factors to
help estimate the effort: 1) the percentage of code that will be
redesigned, 2) percentage to be recoded or modified, and 3) percentage
to be retested. Cost models use these estimated percentages to arrive at
an equivalent line of code estimate.

The equivalent lines of code estimate represents the amount of new code
that could be developed for the same effort it would take to modify the
existing code. Some models allow the user to enter these three
percentages while other models assume a standard percentage of effort to
develop equivalent lines of new code. The REVIC and SEER models allow
the user to enter the three factors. SASET uses a default of 73%, i.e.,
it assumes any reused code will consume 73% of the effort that it would
take to develop equivalent new code.

When the three percentages are used, the equivalent new lines of code is
calculated as follows:

(\[%Redesign + %Recode + %Retest\]/3) \* Existing Code = Equiv LOCs

For example, equivalent LOCs for a program that had 10,000 LOCs and was
assumed to require 40% redesign, 50% recode, and 60% retest would be
calculated as:

(\[40% + 50% + 60%\]/3) \* 10,000 = 5,000 Equiv. LOCs

According to the "COCOMO.II Software Cost Estimation Model" abstract,
the COCOMO.II model uses a nonlinear estimation model for estimating
size in reusing software products. Analysis of reuse costs across nearly
3000 reused modules in the NASA Software Engineering Laboratory
indicates that the reuse cost function is nonlinear in two significant
ways.

It does not go through the origin. There is generally a cost of about 5%
for assessing, selecting, and assimilating the reusable component.

Small modifications generate disproportionately large costs. This is
primarily due to two factors: the cost of understanding the software to
be modified, and the relative cost of interface checking.

The results of a study conducted by Parikh and Zvegintzov indicates that
47% of the effort in software maintenance involves understanding the
software to be modified. Thus, as soon as one goes from unmodified
(black-box) reuse to modified-software (white-box) reuse, one encounters
this software understanding penalty. Also, research by Gerlich and
Denskat shows that, if one modifies k out of m software modules, the
number N of module interface checks required is N = k \* (m-k) + k \*
(k-1)/2.

The size of both the software understanding penalty and the module
interface checking penalty can be reduced by good software structuring.
Modular, hierarchical structuring can reduce the number of interfaces
that need checking. Software that is well structured, explained, and
related to its mission will be easier to understand.

### COTS Sizing

Software developers are using increasing amounts of
commercial-off-the-shelf (COTS) components to construct new systems.
Their motivation is to decrease development costs, schedule and
technical risks. Richard D. Stutzke's paper "Cost Factors for COTS
Integration" refers to a study done comparing how three different models
estimate the size and cost of COTS software. Two models, COCOMO.II and
SEER-SEM employ algorithms similar to those used to estimate the costs
of reusing software. The COCOMO.II model addresses the case where one
has the source code for the COTS product and represents the nonlinear
dependence of adaptation costs with the amount of existing code that
will be modified. For those cases without the source code, it is
believed that the size can be measured in function points and then
converted to SLOC for the computations. A LORAL COTS model used function
points as a basis for its sizing. It used standard function point
counting rules to size the functionality that is incorporated, including
the interfaces between the COTS components and the custom code. Only the
portions of the COTS components that perform functions that are either
used or interface with the custom code are counted. Of the three models
evaluated in the study, only SEER-SEM addressed the full range of COTS
products. SEER-SEM was also the only model to address differences in
code type (custom code (developmental), glue code and COTS code) and
three levels of reusability for COTS components \[code not reused (new
code), existing code not designed for reuse, and existing code designed
for reuse.\] Depending upon the category of the COTS type, either SLOC
or function point is used for sizing. The bottom line is that COTS
software must be evaluated and any coding required must be sized during
the estimation process, just like any other software. Appendix 19E,
"COTS Software" discusses the peculiar aspects of using COTS software
components in a system.

### Summary of Metric Based Sizing Models

The Cost Xpert *User's Manual* compared the strengths of metric based
sizing models as follows: SLOC sizing methods are particularly useful
for real time and embedded systems applications. Function point based
estimates are one of the best approaches for a wide range of MIS
applications that have significant interaction with the environment and
where this environmental interaction is the major cost driver. Function
points are significantly less effective for applications where the
environmental interactions are secondary to the algorithmic requirements
of the program. If object oriented software development is being used on
the program, the object metric approach works equally well with both MIS
type applications and with the embedded, scientific, and system level
programming projects that are not well served by function points.

Current sizing models usually include more than one of the metric based
approaches mentioned in this section. Other sizing models are already
available or currently being developed in one or more of the categories
of models discussed. For example, the QSM Size Planner is a composite
model which contains an expert judgment model ("Fuzzy Logic"), a
functionality model ("Standard Components"), and a function points model
for size estimation. The user can select any combination of models or
use all three and compute a weighted size estimate. Undoubtedly, more
sizing models will become available in the future. No single sizing
model has been shown to be accurate, or even useful, for a wide range of
programs. Each model has a unique set of advantages and limitations.

## Software Cost Estimating Methodologies

Once the basic system being estimated has been defined, planned and
sized, the next step is to select an estimating methodology. There are
five basic methodologies for estimating software.

-   Analogy (Comparative)

-   Grass Roots (Engineering Estimate or Bottom-Up)

-   Top-Down

-   Expert Judgment

-   Parametric (Algorithms)

Table 19-8 provides a brief description of these methodologies along
with their advantages and limitations. Each of the methodologies will be
defined and summarized below. An extensive appendix (Appendix 19D) is
devoted to parametric software estimating models due to their frequent
use in estimating software development costs.

  --------------------------------------------------------------------------------------------------------------------
  Table 19-8.                                                                                                       
  Software Cost                                                                                                     
  Estimating                                                                                                        
  Methodologies                                                                                                     
  --------------- ----------------- --------------- ----------------- -------------- --------------- -------------- --
                  **Methodology**   **Analogy or    **Grass Roots,    **Top-Down**   **Expert        **Parametric   
                                    Comparative**   Engineering                      Judgment**      or             
                                                    Estimate, or                                     Algorithms**   
                                                    Bottom-up**                                                     

                  **Description**   Compare project Individuals       Project        Consult with    Perform        
                                    with past       assess            partitioned    one or more     overall        
                                    similar         components, then  into lower     experts.        estimate using 
                                    projects.       component         level                          design         
                                                    estimates are     components &                   parameters and 
                                                    summed to obtain  life cycle                     mathematical   
                                                    total estimate.   phases                         algorithms.    
                                                                      beginning at                                  
                                                                      highest level                                 

                  **Advantages**    Estimates are   Accurate          More           Little or no    Models are     
                                    based on actual estimates are     applicable to  historical data usually fast   
                                    experience.     possible because  early project  are needed;     and easy to    
                                                    of detailed basis estimates.     good for new or use, and       
                                                    for estimate;     Considers      unique          useful early   
                                                    also promotes     system level   projects.       in a program.  
                                                    individual        activities,                    They are also  
                                                    responsibility,   faster, easier                 objective and  
                                                    supports project  to implement.                  repeatable.    
                                                    tracking.                                                       

                  **Limitations**   Truly similar   Methods are time- Less accurate  Experts tend to They are often 
                                    projects must   consuming.        than other     be biased, and  inaccurate and 
                                    exist.          Detailed data     methods, tends knowledge level unstable.      
                                                    needed may not be to overlook    is sometimes    Also,          
                                                    available,        lower-level    questionable.   calibration to 
                                                    especially early  components,                    historical     
                                                    in a program.     provides                       data may not   
                                                    Integration costs little detail.                 be relevant    
                                                    may be                                           for new        
                                                    disregarded.                                     programs.      
                                                    Process is costly                                               
  --------------------------------------------------------------------------------------------------------------------

### Analogy or Comparative

According to NASA's Handbook, estimating by analogy is the act of
comparing the proposed project to previously completed similar projects
where project development information is known. Actual data from the
completed projects are extrapolated to estimate the proposed project.
Estimating by analogy can be done either at the system level or the
component level.

For example, consider the situation in which an agency wanted to develop
a new payroll program that serves 5,000 people and contains 100,000
lines of COBOL code. A second agency had developed a similar
100,000-line program for two million dollars. It could be expected that
the second agency\'s program, ignoring inflation, would also cost two
million dollars. The advantage of the analogy method is that it is based
on actual experience. However, it is limited because, in most instances,
no truly similar software programs exist. For example, the cost of
100,000 lines of C++ code for a radar program would not be the same as
that of 100,000 lines of COBOL code for payroll software.

The main strength of this method is that the estimates are based on
actual project data and past experience. Differences between completed
projects and the proposed project can be identified and impacts can be
estimated. One problem with this method is identifying those
differences. This method is also limited because similar projects may
not exist, or the accuracy of available historical data are suspect. The
analogy or comparative technique uses parametric approaches including
the use of CER\'s. The analogy methodology should not normally be used
alone to estimate most software costs, except as a last resort, although
it can be useful in checking other estimates for reasonability.

### Grass Roots, Engineering Estimate or Bottoms-up

Grass roots estimating, also called "engineering estimating" or
"bottoms-up", provides a more sophisticated means of analyzing costs. It
involves estimating software costs by a detailed analysis of the cost of
each unit (or CSU), then summing unit costs to determine the cost (or
effort) for each CSC, each CSCI, and then the software cost for the
overall system. Grass roots estimating is sometimes used during proposal
preparation and, subsequently, for tracking costs of software projects.

An advantage of this method is that it provides a detailed basis for
cost estimating and, consequently, tends to be more accurate and stable
than other methodologies because it deals with low-level components.
Also, since unit costs are assessed by personnel responsible for the
development of the unit, this type of estimating is a good managerial
tool for promoting individual responsibility for keeping units within
the estimated costs. It supports project tracking more directly than
other methods because its estimates usually address each activity within
each phase of the software development life cycle.

However, grass roots estimating has several disadvantages. Since
detailed information about each unit is required, it may be difficult to
use in certain circumstances, especially during the early phases of the
life cycle where detailed information is often unavailable. Using grass
roots techniques may not be appropriate until detailed design has been
completed. Also, grass roots estimating is usually very time-consuming,
which makes it inappropriate when either time or available estimating
personnel are limited. For these reasons, grass roots estimating is
infrequently used within Government agencies where time and personnel
are usually scarce. Another limitation of grass roots estimating is that
it does not automatically capture the costs associated with integrating
units to form higher-level components and CSCIs. Integration costs must
be separately estimated.

### Top-Down Method

NASA's Handbook summarizes the top-down method of estimation, which is
based on overall characteristics of the software project. The project is
partitioned into lower-level components and life cycle phases beginning
at the highest level. This method is more applicable to early cost
estimates when only global properties are known. Advantages include
consideration of system-level activities (integration, documentation,
project control, configuration management etc.), that may have been
ignored in other estimating methods. The top-down method is usually
faster, easier to implement and requires minimal project detail.
However, it can be less accurate and tends to overlook lower-level
components and possible technical problems. It also provides very little
detail for justifying decisions or estimates.

### Expert Judgment 

The STSC Report and SSCAG Handbook summarize this method. Expert
judgement uses the experience and knowledge of experts to estimate the
cost of a software project. An advantage of this method is the
experience from past projects that the expert brings to the proposed
project. The expert also can factor in project impacts caused by new
technologies, applications, and languages. Examples of popular expert
judgment techniques include the Delphi and Wideband Delphi methods.
Expert judgment techniques are suitable for assessing the differences
between past and future programs; and are especially useful for new or
unique programs for which no historical precedent exists. However, the
expert\'s biases and sometimes insufficient knowledge may create
difficulties. It can be hard to document the factors used by the expert
who contributes to the estimate. Although Delphi techniques can help
alleviate bias problems, experts are usually hard-pressed to accurately
estimate the cost of a new software program. Therefore, while expert
judgment models are useful in determining inputs to other types of
models, they are not frequently used alone in software cost estimating.

### Parametric Modeling or Algorithms

Parametric modeling, as defined here, is a combination of algorithmic
and top-down models as described by Boehm. The algorithmic method
involves the use of equations to perform software estimates. The
equations are based on research and historical data and use such inputs
as SLOC, number of functions to perform, and other cost drivers such as
language, design methodology, skill-levels, risk assessments etc.
Parametric models and techniques for software projects generally
estimate overall system or CSCI costs based on design characteristics,
or parameters, of a software program. The overall costs can later be
partitioned among lower-level components (CSCs) or among life cycle
phases.

Advantages of this method include being able to generate repeatable
results, easily modifying input data, easily refining and customizing
formulas, and better understanding of the overall estimating methods
since the formulas can be analyzed. Also, they are usually fast and easy
to use, require little detailed information, and capture system or
CSCI-level costs. The disadvantages are that they tend to be less
accurate and less stable than the grass roots methodology; and they do
not inherently foster individual responsibility. The STSC Report adds
that the results can be questionable when estimating future projects
that use new technologies. Also, the formulas generally are unable to
deal with conditions such as exceptional personnel, exceptional teamwork
and exceptional matches between skill levels and tasks.

Parametric modeling is usually the methodology of choice for Government
agencies because of the minimal time and effort required and because it
is useful in early phases of a software development program when little
is known about the program. Also, attempts are continually being made to
calibrate available models to insure greater accuracy. Because of their
widespread use, parametric models will be discussed in general here and
some currently popular software parametric cost models will be discussed
in Appendix 19D.

All parametric models use one or more design parameters to estimate cost
or level of effort. The earliest parametric models, some of which are
still in use, were simple models that estimated cost based only on the
size of the software in SLOC, machine instructions, or other measures.
They were often of the form E = AI^B^ where E is the effort required in
dollars, man-months, or similar measure; I is the size of the program; A
is a coefficient; and B is an exponent. The coefficient and exponent are
usually computed using mathematical regression analysis on historical
size and effort data. This simplistic model generally shows considerable
variance, even within the database from which the model was formulated,
which is one reason why simple single-parameter parametric models, such
as those based solely on program size, are not generally considered to
be accurate.

Another limitation of these models is that the coefficients and
exponents vary significantly with the database used, indicating that the
models are highly database dependent. This suggests that other factors,
besides the size of the software program, affect the effort or cost
required. Also, other methods besides regression may be used in model
development. Popular parametric cost models are discussed in the
Appendix 19D.

## Life cycle Costs

In addition to the costs associated with software development, the costs
for the **Contract Data Requirements List (CDRL)** items such as
manuals, specifications, plans, reports and training, plus the
maintenance of the system after it becomes operational needs to be
considered for the complete Life cycle costs of the project.

### CDRL Deliverables Estimates

As summarized in the Cost Xpert *User's Manual*, many models provide an
estimate of the deliverables (such as the Software Requirement
Specification (SRS)) that should be produced (based on the software
development standard used, (see section 19.3.2) and an estimated page
count. The page count is useful for two reasons. First, if the delivered
document deviates significantly from the estimated page count, it raises
the question whether the document is sufficiently complete. Second, the
numbers are useful when planning to review documents in order to
estimate the magnitude of the work that will be required. Capers Jones
determined that on the average, companies spend between 2 and 4 hours
per document page for technical documents.

### Software Maintenance Estimates

The SSCAG Handbook defines the term "software maintenance" as those
software engineering activities that occur following delivery of a
software product to the customer. The maintenance phase of the software
life cycle is the time period in which a software product performs
useful work. Typically, the development cycle for a software product
spans 1 or 2 years, while the maintenance phase spans 5 to 10 years.
Estimates of the magnitude of software maintenance costs range from 50
to 75% of overall software life cycle costs. In 1985 it was estimated
that software maintenance typically requires 40 to 60% of the total life
cycle effort. And a widely used rule of thumb for the distribution of
maintenance activities is 60% for enhancements, 20% for adaptation and
20% for error correction.

#### Factors Influencing Maintenance Costs

Boehm stated that for each dollar spent on software development, another
dollar needs to be budgeted just to keep the software viable over its
life cycle and another dollar can be spent on desirable enhancements
over the life cycle.

Boehm\'s and Nippon\'s Software Productivity Range Figures shows that
the most important maintenance cost driver is personnel. Maintenance
work is generally perceived to be less interesting, resulting in higher
personnel turnover rates than new system development work. Demotivated
personnel can have a huge adverse impact on a project\'s cost/schedule
compliance.

The maintainability of software is a qualitative measure of the
following factors:

-   Ease of understanding that is reflected by the ability of an outside
    auditor to understand the structure, interfaces, functions and
    internal procedure of the software. Modularity, detailed design
    documentation, internal source code documentation and structured
    design all contribute to ease of understanding.

-   Ease of diagnosis and testing depend upon ease of understanding and
    again good documentation is essential.

-   Ease of change that is directly related to design criteria.

High-quality products require less maintenance and enhance both
development and maintenance productivity. Investment in software
reliability and modern programming practices for software development
have a strong payoff during maintenance.

Software development cost models are typically ineffective in their
ability to project resources for software maintenance activities for two
fundamental reasons. First and foremost, they use delivered effective
SLOC as the independent variable in the algorithms used to predict
effort and/or schedule. The second reason is because most models define
software maintenance to be only repairing errors in the code and do not
include adaptive or perfective activities. (See Table 19-2.)

A lack of attention to software maintainability during the requirements
specification, design, and coding phases generally leads to excessive
software maintenance costs. Software changes occur not only as a
consequence of detecting and correcting errors not discovered during
development, but also as a consequence of postponing development tasks
until the maintenance phase. This can be caused by tight schedules and
as a consequence of changing system requirements. Figure 19-9, from
NASA's Handbook, indicates the relative impact of the penalty (cost) for
delayed error correction during a software project's life cycle, from
almost no cost during preliminary design to high cost impacts during
validation and operative stages. Business process re-engineering focuses
on the later stages (maintenance) and the reduction of errors at those
times by re-engineering the early phases.

> Figure 19-9. Impact of Delayed Software Errors Detection

**Error! Not a valid link.**

A study referred to in the SSCAG Handbook using data from 346 large
electronics manufacturing firms' programs showed that the occurrence of
software bugs was strongly related to measures of size and complexity
and less strongly to the program's usage. These findings appear to
vindicate those who advocate limiting program module size in order to
help reduce delivered bugs. Repairing a defect has a significant (20% to
50%) chance of introducing another defect. So the entire process can be
thought of as \"two steps forward and one step back\".

#### Software Maintenance Conclusions

The following suggestions are ways of improving the three types of
maintenance:

-   **Corrective maintenance:** use of high-level languages, keep
    modules small, and use structured techniques,

-   **Adaptive maintenance:** use a portable high-level language and
    strive for hardware independence, and

-   **Perfective maintenance:** only improve modules that have a high
    usage level, have a reasonable life span and have a high cost for
    corrective or adaptive maintenance.

Programmers should be directed to develop reusable code. Although this
may increase development costs, it will reduce maintenance costs, as
reusable code is easier to maintain, partly because it is of higher
quality and partly because there is less of it.

Structure of the software, the quality of available documentation, the
experience and application knowledge of maintenance programmers, the
extent and types of maintenance, and the management attitude, should be
considered in estimating the maintainability of software.

## Software Pricing Process

Pricing generates the data needed by procurement officials to make the
most advisable decisions possible. The Federal Aviation Administration's
Acquisition Management System (FAA AMS) provides direction on the use of
pricing within the acquisition process. Pricing, as defined in the FAA
Pricing Handbook (FAA PH), is the techniques used by the procuring
official to establish a "fair and reasonable price". Price analysis and
cost analysis are the two basic techniques used to accomplish this
purpose. [Price analysis should always be performed]{.underline}. [Under
certain circumstances, cost analysis also needs to be
performed]{.underline}. This section will summarize selected information
concerning Pricing in general and then the particulars of Software
Pricing. Software pricing is based upon the same principles as software
cost estimating, so an understanding of the previous sections is
important. Detailed information regarding pricing in general is
contained in the other chapters of the FAA PH.

### Software Price Analysis

Price analysis is the process of examining and evaluating a proposed
price [without]{.underline} evaluating its separate cost elements and
proposed profit. It may be accomplished by: 1) a comparison of submitted
quotations; 2) a comparison of price quotations and contract prices with
current quotations for the same or similar items; 3) the use of
parameters (SLOC per person-month, for instance); and/or 4) a comparison
of proposed prices with independently developed cost estimates.

In order to conduct software price analysis, the terms for comparing the
data must be properly defined. Referring to The Department of the Navy
Center for Cost Analysis (NCCA) Software Development Estimating
Handbook, Phase One, dated February 1998, there are two basic types of
information required to develop a quality software price analysis: 1)
technical and programmatic information for the program proposal being
analyzed and 2) technical and programmatic information for the
analogous/similar historical programs used to develop the estimate/ICE
that will be compared to the proposal.

Based on the analysis in NCCA's Handbook, a software development
estimate requires, at a minimum, the following information for the
estimates being analyzed:

-   Some measure of the work to be performed with associated units
    (i.e., SLOC counts, words, function points, etc.)

-   If SLOC is utilized as the unit of measure, the associated counting
    convention (i.e., physical, physical with comments, logical, etc.)

-   The programming language utilized (at a minimum Assembly versus
    Higher Order Languages (HOL (e.g., FORTRAN, Jovial, CMS-2, etc.))
    versus Fourth Generation Languages (4GL)).

-   The condition of the code (i.e., percent new, percent reused
    (modified, verbatim, translated, rehosted, COTS, etc.)), with
    associated definitions.

-   The phases of the software development life cycle to be estimated
    (e.g., System Design Review (SDR) through Formal Qualification Test
    (FQT)).

-   The development mode (at a minimum, embedded versus non-embedded).
    Embedded Software is defined as software that is inside a physical
    object and controls its behavior. This is a more or less specialized
    term for software inside navigational devices, radar sets,
    oscilloscopes, and other instruments. Embedded software has its own
    characteristic productivity and quality profiles. (Source: Capers
    Jones). This is software that determines the functionality of
    microprocessors and other programmable devices that are used to
    control electronic, electrical and electro-mechanical equipment and
    sub-systems. The programmable devices are often \"invisible\" to the
    user. (Source: The TickIT Guide.))

-   Especially for historical costs, if known, the name of the
    contractor responsible for developing the program. NCCA contends
    that contractor-specific data holds the greatest possibility for
    increasing the accuracy and decreasing the variance associated with
    software estimating tools.

In addition to the above items, there are four other areas that can be
cost drivers for software that are input parameters for most software
parametric cost models:

1.  Project Application -- MIS, communications, radar, etc.

2.  Specification Level - MIL-STD-2167A, commercial, etc.

3.  Development Model - Waterfall, Spiral, etc.

4.  Project Scope - rehost, development, maintenance, etc.

Since all of the information requested affects the projected
productivity of the development effort, it is crucial that the
information gathered be as specific as possible. In addition to the
aforementioned information on the program being estimated, the analyst
must compile the same information for the analogous historical programs
that will be used to develop the price analysis comparison. Furthermore,
the actual effort, schedule, and cost (price) to develop the software,
by software development phase if possible, should be obtained. With this
information the most accurate productivity, schedule, and labor rate
metrics can be developed. If the **Screening Information Request (SIR)**
was developed correctly, the **Software Development Plan (SDP) Contract
Data Requirements List (CDRL)** item is an excellent source of
historical data. The SDP typically requires a list of previously
delivered programs developed by the contractor, with the associated
technical and programmatic data. If, however, the SDP is not available,
this type of information can and should still be obtained from the
contractor in whatever form is available. When collecting historical
data, the analyst must ensure that the information is for completed
programs because projections of on-going efforts are often mixed in with
the list of the contractor's programs. Since software development is
continuously evolving, the analyst should always try to obtain the most
recent data available.

It is important to know how the SLOC was counted so that any
productivity or effort estimating relationships developed will be valid.
There are two main categories of code counting conventions: physical and
logical. Counting physical SLOC is accomplished by tallying the number
of carriage returns in the source document. Logical SLOC are counted by
tallying logical units (for example, an IF-THEN-ELSE statement is
considered one logical unit).

The impact of the code counting convention is emphasized in the NCCA
Handbook, which referenced two studies. An Institute for Defense
Analyses (IDA) study found that on average, physical SLOC produce a code
size that is about 20 percent higher than counting the same code using a
logical SLOC definition. NASA\'s Software Engineering Laboratory (SEL)
also found wide differences between physical and logical code counts.
They found that a FORTRAN program's ratio of physical lines to logical
statements ranged from 2.5 to 5 due to variations in the number of
comments. Likewise, Ada programs exhibited a similar ratio of 2.5 to 6
physical lines per logical statement.

Not only is knowing the amount of source code necessary, but knowing the
"condition" of the code is also important. NCCA used the term
"condition" to describe the composition of the source code (i.e., %new,
%reused). Sections 19.2.4 (Technology Insertion) and 19.4.3 (Reused
Code) contain additional information concerning reused code. The amount
of HOL a program contains is also an important factor to consider. All
programming languages, except Assembly language, are defined as HOLs.
Analysts should ask for the new and reused SLOC by language so as to
avoid having to derive these values.

When using historical software effort data, it is important to consider
the level of requirements under which the software was developed. A
major program may have several software development efforts spanning
different acquisition phases. For example, typical acquisition
strategies require development of prototypes and associated software
during a Prototype Phase. After a competitive selection process, one
contractor\'s design is chosen for further development. Final
development takes place for the deployable software by the winning
contractor. The contractor probably reused code from the Prototype Phase
that may not have undergone the same level of documentation, testing, or
review as software developed for deployment. As a result, using
historical prototype data points to estimate effort prior to deployment
may not be appropriate without some adjustment.

Therefore, it is recommended that the data shown in Table 19-11 be
requested from the offerors (if not already in the SIR) and from the
sources of other data/ICE to allow a valid comparison.

Table 19-9. Software Estimate Comparison Parameters

  ----------------------- -----------------------------------------------
  **Data Area**           **Description**

  Project Application     What is the software's functional purpose

  Project Size            Most commonly in SLOC but could be in Function
                          Points or Object Points

  Size Counting           Defines what is included in the SLOC/FP/OP
  Convention              count

  Project SW Language     If coded in more than one language, try to get
                          percentages. As a minimum, know Assembly versus
                          HOL

  Project Phasing         What phases of the SW development schedule were
                          included in the contract price

  Code Condition          \% new, % modified (need definition e.g., less
                          than 20% recoded), % unmodified, % COTS

  Development Mode        Embedded versus non-embedded as a minimum

  Specification Type      MIL-STD-2167A, commercial, etc.

  Development Model       Waterfall, Spiral, etc.

  Project Scope           Rehost, development, maintenance, etc.
  ----------------------- -----------------------------------------------

If as a minimum the information in Table 19-11 is obtained for the
proposed project and the item(s) to be used as the basis of comparison
(ICE, market survey, similar items, etc.), then there should be good
substantiation for the comparison.

### Price Analysis Techniques

The FAA AMS Toolbox Guidance T.3.2.3, Section A-3, par. a-f provides the
following pricing techniques to use when performing price analysis. The
expected contract type that would be appropriate for the tool to be used
with is indicated in Parenthesis.

-   Comparison of proposed prices with Independent Cost Estimates. (Any
    type)

-   Comparison of proposed prices of competitors received in response to
    the solicitation. (Competition)

-   Comparison of proposed prices received in response to the Screening
    Information Request (SIR) or Market Survey. (Competition)

-   Comparison of prior proposed prices and contract prices with current
    proposed prices for the same or similar end items and services in
    comparable quantities. (Sole Source or Cost Plus)

-   Comparison with competitive published catalogs or lists, published
    market prices or commodities, similar indexes, and discount or
    rebate arrangements. (Sole Source or Cost Plus)

-   Application of Software Cost Estimating Model parameters or Rules of
    Thumb (such as person-months per SLOC, or other units; see section
    19.7.5) (AMS used the term "rough yardsticks" for this technique) to
    highlight significant inconsistencies that warrant additional
    pricing inquiry. (Sole Source or Cost Plus)

-   Ascertaining that the price is set by law or regulation.

### Price Analysis Tools

[Independent Cost Estimates (Ices)]{.underline}

ICEs are the key estimates required by and for cost and pricing
evaluations. This estimate is an assessment of the total cost to be
incurred by the offeror if the contract is awarded. ICEs should be
developed early in a procurement cycle and be used to determine the
reasonableness and completeness of the proposals and to detect buy-ins,
unbalanced pricing, and other gaming techniques. Additionally,
independent estimates can be used for future budget estimates and
procurement planning and decision-making. ICEs may also be complemented
by economic analysis and/or cost studies. Without these key elements,
structuring of Sections B, L, and M can be difficult. The IPT will
determine when an ICE is appropriate. The Contracting Officer may
request an ICE prior to issuance of a **Screening Information Request
(SIR).**

The evolution of these estimates progresses as the procurement concept
grows. Estimates must be conducive to the pricing evaluation. Too often
estimates are done only to support budget or technical trade-offs and
cannot be translated to **contract line item number (CLIN)** and **work
breakdown structure (WBS)** break-outs. In order for the analyst to
develop a realistic pricing evaluation, initial estimates must support
the development of Section B CLINS and provide evidence of establishing
marketable prices that relate to these CLINs.

#### Comparison with a Similar Item's Proposal/Price/Cost Estimate

This section is based on a U.S. Air Force Material Command (AFMC) white
paper "Methods for Evaluating Similar Items", dated January 1996. The
ability to compare a proposal with a similar item for price analysis
presumes the price for that similar item is reasonable and acceptable.
The Government might have purchased the item previously on the basis of
adequate price competition, catalog or market pricing, commercial item
pricing, or negotiations using cost or pricing data. If so,
documentation that demonstrates price reasonableness for the item would
already be possessed by the Government. Any of these should generate the
confidence necessary to support the contention that price analysis will
produce a reasonable price. However, it is also possible that the
Government may never have purchased the similar item. In that situation,
it would be necessary to establish the reasonableness of a similar
item's price before any further price analysis on the offered item could
be conducted. It is imperative to determine a suitable basis, else the
price analysis would be without merit.

Once comfortable with the reasonableness of the similar item's price, an
understanding of the technical similarities and differences between the
offered item and the similar item is necessary. The pertinent
characteristics (e.g., size, language, application type, etc. from Table
19-11) of each item must be identified to facilitate comparison.

If a direct comparison is not possible, break down the offered and
similar items until a common baseline is reached. It may be as simple as
segregating options and upgrades to the same basic, lower-level unit,
along with two lists of adders that would complete the items. The
automobile industry is the clearest example where this method can be
applied. However, the same thing can be done with software, isolating
differences such as functionality to leave the same basic operating
system and application type. Then, proceed with establishing prices for
the baseline unit and each adder.

The most complex, and perhaps most frequent, situation to be
encountered, especially for a new development, involves the inability to
reach a common, identifiable baseline unit. In this case compare the
characteristics of the two items and determine some relationship between
them. Examples could be found where similar items might be compared in
areas such as size, language, development phases, and specification
level. The price of the similar item, having been determined reasonable,
is used as the baseline, and the differences between the two items are
considered as pluses and minuses to that baseline.

When the procured and similar items are broken down for evaluation, any
suitable price analysis tools and techniques may be used to substantiate
prices of the segregated pieces. Utilize purchase history or catalog,
market, or commercial price assessment as applicable. By closely
aligning characteristics into comparable categories, some parametric
relationship might be disclosed to explain the impact of a
characteristic. Of course, if reasonableness of a portion cannot be
established using price analysis techniques, the negotiator should
request cost information (cost or pricing data if the threshold is met)
on that portion and perform a cost analysis.

[The Cost/Price Model]{.underline}

The analyst should plan for the development of a cost/price model. This
is not a Cost Estimation Model as discussed in Appendix 19D, but a
spreadsheet type model used specifically for cost/price analysis and/or
proposal evaluation. When preparing a negotiation position, a cost/price
model should consist of spreadsheets for the basic items being
negotiated (usually CLINs or WBS elements), summary sheets, and sheets
containing backup data (other direct costs, rates etc.). Essentially,
all of the elements that are to be analyzed should be represented in the
model.

In a competitive procurement it is often useful to provide an automated
model with the SIR for direct input by the contractors. This reduces
evaluation time and minimizes errors. The construction of the model will
depend heavily on the type(s) of contract, CLIN structure, WBS
structure, and quantity of data required. The analyst must work closely
with the CO and IPT, to assure that mutual goals are met. A technical
description and an in-depth explanation of cost/price models are
provided in Chapter 3, "Automated Cost Models" of this handbook.

For software procurements, the model can be especially useful since it
should contain the input parameter requirements (Table 19-11 as a
minimum) needed to compare proposals to the ICE and/or similar items,
plus enter data into a Software Cost Estimation Model.

### Software Cost Analysis

Cost analysis is the review and evaluation of the separate cost elements
and proposed profit/fee of:

-   An offeror's or contractor's cost or pricing data or information
    other than cost or pricing data and

-   The judgmental factors applied in projecting from the data to the
    estimated costs.

The purpose of the evaluation is to form an opinion on the degree to
which the proposed costs represent what the cost of the contract should
be, assuming reasonable economy and efficiency. However, cost analysis
does not necessarily provide a picture of what the market is willing to
pay for the product involved. For that price analysis is needed.

The cost areas analyzed in a Cost Analysis, each discussed in detail in
a separate chapter of the FAA PH, are as follows:

-   Direct Labor

-   Material

-   Other Direct Costs (ODCs)

-   Indirect Costs

-   Facilities Capital Cost of Money

-   Profit/Fee

Common terms used in cost analysis that are applicable to software are
contained in Table 19-12 below.

Table 19-10. Cost Analysis Terms and Definitions

  -----------------------------------------------------------------------
  **Terms**         **Definition**
  ----------------- -----------------------------------------------------
  Basis of Estimate A statement of the rationale used by a contractor to
                    generate a cost estimate for a specific task or item
                    to be produced.

  Direct Labor      Work performed by individuals which is directly
                    related to a specific cost objective. This work is
                    readily identifiable with a particular product or
                    service.

  Indirect Labor    Work performed by individuals which is not
                    identifiable with a single final cost objective but
                    is identified with two or more final cost objectives
                    or an intermediate cost objective. One example of
                    indirect labor is the work expended by the controller
                    of a company. The controller's work is not directly
                    identifiable in the production of a specific product
                    or service, since his or her work is spread across
                    several projects or tasks.

  Labor Hour        The unit of time by which direct labor activity is
                    measured.

  Labor Rate        The dollar amount paid to an individual per a given
                    amount of time in consideration of work accomplished.

  Labor Cost        The product (i.e., result) of multiplying labor hours
                    by appropriate labor rates.

  Labor Category    A grouping of workers with similar skills or
                    expertise.

  Labor Mix         The combination of functional skills and levels of
                    worker experience required to accomplish a given
                    task.

  Other Direct      Other costs charged directly to the Government that
  Costs (ODC)       have not been included in proposed material, direct
                    labor, indirect costs, or any other category of cost.
                    ODCs can include but are not limited to: special
                    tooling, travel expenses, relocation expenses,
                    pre-production and start-up costs, packaging and
                    transportation costs, royalties, spoilage and rework,
                    computer expenses, federal excise taxes, and
                    reproduction costs.
  -----------------------------------------------------------------------

### Considerations for Software Cost Analysis

In addition to general cost analysis considerations, there are three key
considerations that apply to most proposals for a software program.
These are the software development productivity, code condition (percent
new and reused code) and the software defect rate by program phase.

[Productivity]{.underline}

Productivity is a primary indicator of how efficient the contractor is
in developing software. Productivity relates software development effort
to the organizational capabilities, experience, and individual talents
of the team that will perform the software development. This rate should
be based upon historical data. The productivity rate tends to remain
constant for a given organization, so previous proposals by the same
contractor are especially valuable. It is much more difficult to compare
rates between different contractors because it is so dependent upon
organization and personnel. Productivity is also affected by programming
language, processes, specification level and software tools.

Referring to NCCA's Handbook, two calculations should be made for
software development productivity:

1.  Productivity expressed as hours per SLOC based on the following
    formula:

$$\text{Hours/SLOC} = \frac{\text{Total PM*152 Hours/PM}}{\text{Total SLOC}}$$

2.  Productivity expressed as hours per new SLOC. This metric is
    important because new SLOC tends to drive the effort.

$$\text{Hours/New SLOC} = \frac{\text{Total PM * 152 Hours/PM}}{\text{Total SLOC * \% New SLOC}}$$

> *PM = person-months of effort, 152 is assumed to be the average
> work-hours in a month.*

This rate along with the code size usually forms the basis for the
software development effort. Therefore, it is important to compare
productivity against a Rule of Thumb estimate, other programs and/or the
contractors past performance to determine if it reasonable. Then, over
the life of the contract, compare the rate from proposal to proposal.
The rate does tend to increase (become less efficient) gradually over
the life of the contract as additional requirements are made to the
program (new code has to be integrated with more and more old code).

[Code Condition]{.underline}

Code condition, although mentioned in section 19.7.1 in regards to price
analysis, is also important to cost analysis. The proposal should
separate the total software development effort by percent of new code,
modified code, COTS and unmodified code (see Sections 19.1.2, 19.2.4
(Technology Insertion), and 19.4.3 for more details on the definitions
of these terms and Appendix 19E for COTS considerations).

Reused code (modified, unmodified and COTS fall into this category) is
included in a program to reduce effort, cost and schedule. According to
NASA's Software Engineering Lab (SEL), "Cost and Schedule Estimation
Study Report", dated Nov 1993, for projects with moderate to low code
reuse (less than 70 percent), the post-CDR growth in SLOC due to
requirement changes and TBDs is commensurate with past SEL experience:
40 percent. For projects with high code reuse (70 percent or more), the
post-CDR growth in SLOC is only about half as much (20 percent). For
projects with moderate to low code reuse, the post-CDR growth in
schedule is 35 percent. For projects with high reuse, the post-CDR
growth in schedule is 5 percent.

As mentioned in Section 19.1.2, once the percentage of modifications in
a block of code exceed 20%, it is usually less expensive to write new
code. Often contractors are optimistic in the amount of code that can be
reused. Monitoring metrics of software program growth (increasing cost
and schedule), there are typically two sources of new code increase,
requirements growth from the user and less modified/unmodified code than
projected by the contractor. As the contractor gets into the process of
actually developing the software, typically the task of reusing software
becomes more difficult than planned. Therefore it is important to keep
track of these percentages over the life of a contract as both
indicators of future problems and for cost analysis. Table 19-4 of
Section 19.3.1 includes code size as a process effectiveness metric.

Figure 19-10 below is an example of new code growth at the expense of
reused code taken from an actual FAA program. Although total SLOC grew
only 5% during the 17 month period shown, the amount of new code that
had to be developed grew from 59% of the total code to 89%. There was
also about 25% schedule growth on this program.

![](media/image1.wmf)

> Figure 19-10. Comparison of New Code versus Reused Code

[Software Defects]{.underline}

The contractor should indicate in the proposal the number of defects
expected to be discovered and fixed for each phase of the program. As
discussed in Section 19.6.2, the earlier in the development cycle that a
software bug is discovered/detected, the less expensive it will be to
fix. The quantity of software defects should be expressed in or
converted to a rate since this is how defects are compared across the
industry.

$$\text{SW Defect Rate} = \frac{\text{Defects}}{\text{SLOC}}$$

There should be separate rates calculated by phase using the SW Defect
Rate equation for new code and modified code, plus a rate for defects
remaining in the existing, unmodified code. These calculations would
then become the basis for the effort required for testing/engineering to
find and to fix the defects in each phase. Defect metrics can be useful
during software cost analysis to aid the analyst in determining if the
contractor's estimated effort for identifying and fixing defects is
consistent with the contractor's/industry's historical averages.

Figure 19-11 shows a typical trend for detecting software defects based
on studies of completed software programs by Musa, Iannino and Okumoto
in their book *Software Reliablity-Measurement, Prediction,
Application*. The defects discovered during coding and unit test are
usually not formally reported/tracked by the contractor so expect that
only the estimates for the phases after Unit Test will be provided in a
proposal. Musa et. al., per their study, organized the data to predict
the number of defects remaining in the software per KSLOC at the start
of each phase. Each defect quantity includes the defects for the
remaining phases (6.01 is included in the 19.70). The book indicated
that program size (SLOC) was the most important defect prediction
factor, with specification change activity, programmer skill level and
design documentation thoroughness being the other most significant
predictive factors. Defects continue to exist in the software even after
site installations are completed that are gradually discovered as
changes in software use or hardware occur.

> ![](media/image2.wmf)Figure 19-11. Mean Software Defect Rate by Phase

Figure 19-12 below shows a defect prediction curve from a previous FAA
software system. The contractor updated the defect prediction curve (top
line) at proposal time (based upon 6 defects/KSLOC of new code at the
start of system testing for that program) and then tracked the defect
detection curve as the program progressed. The points where the top
curve stepped up occurred because of increases in program requirements
(new SLOC being added). Even though the amount of code increased by
steps, notice the discovery rate remained at basically a constant rate
because it is proportional to the amount of testing being conducted.
Although there will be more defects to be discovered as the amount of
code increases, the testers are able to identify defects only so fast.
Because this program spanned many years with the same contractor,
personnel experience and processes improved overtime so that the defect
rate decreased (distance between middle and top lines grew larger).

> ![](media/image3.wmf)Figure 19-12. Software Defect Prediction versus
> Actual Curves

As will be discussed in Section 19.10.3 there are Product Quality
metrics that track the defect discovery rate and defect solution (i.e.,
fixed or closed). These are indicators of backlog and rework effort, as
well as the contractor's process capability in finding and fixing
defects, and the potential quality of the software product. The bottom
curve of Figure 19-12 shows the number of defects that were fixed. Since
the delta of unfixed defects increased, this would normally be an area
of concern. For this program, a decision was made near the end of the
program to not spend the funds to fix low-priority defects.

The defect rates in the proposal should approximate the trends of
Figures 19-11 and 19-12, or this could be an area for further
investigation. If the system has significantly more defects than
predicted by the industry averages, then the contractor's software
process and quality assurance activities should be looked at. Defect
detection and correction tend to drive schedule (and cost) during the
system test phase and maintenance costs during operations. According to
NRaD's Manual, the cost of corrective maintenance will depend on the
quality of the software. Software that is fielded with a great number of
defects will result in a greater number of user generated software
trouble reports which in turn could potentially result in high
maintenance costs.

### Cost Analysis of a Parametric Estimate

Since most software cost estimates and proposal estimates today are
based at least in part on parametric models, this section summarizes
some points to consider when analyzing a parametric estimate. This
section, along with the REVIC (a software cost estimating model based on
COCOMO, see appendix 19D for details) example are extended from NASA's
Handbook.

With the proliferation of parametric cost estimating models and tools,
both commercial models and "home grown" versions, it is impossible to
describe what to look for in every model and cost parameter. However,
some generalizations can be made. An analyst confronted with a
parametric cost estimate, should take a few steps to ensure a fair
review. These are:

-   Understand the cost model used. Do not hesitate to ask questions and
    to consult with the experts. A user's understanding of the model is
    not necessary, just a general knowledge about how it works. Appendix
    19D and 19E can be of assistance and most commercial model builders
    welcome calls from users, analysts and auditors.

-   Review the program inputs to the model. Is the schedule correct?
    Does the WBS adequately describe the product being estimated? Is
    anything missing? Are there duplications? Does the WBS follow the
    statement of work?

-   Review the technical inputs to the model with government engineers
    or IPT. Check them for reasonableness and benchmark them using the
    experience of the resident experts.

-   Understand the model's cost drivers. Generally, there are a few
    select parameters that are the predominant cost estimating factors
    that drive cost. Many of the others are just "tweaks" to the model.
    Concentrate on the cost drivers when performing the analysis.

-   Be aware of the assumptions the cost estimator made when the model
    was built. Are they still reasonable for this procurement?

-   Be knowledgeable of the historical cost basis for the model, if any.
    Be sure to review the source documentation. Be wary of any model
    used that has no basis in historical cost. How was the data "tuned"
    or normalized? Were data outliers disregarded? If so, why? Was the
    model calibrated? How was the calibration performed? Were any
    universal or "standard" cost factors used? Would they be applicable
    in this case?

-   Question how the future environment might be different from the
    historical case. Have these differences been accounted for?

-   Review the track record of the estimator or the estimating
    organization. What is their past performance? Have their estimates
    been reliable?

-   Understand the economics involved with the model. What are the
    effective costing rates used? Are they reasonable? Do they reflect
    the organization and skill levels being estimated? Are the skill
    levels appropriate for the type of work being performed?

-   Identify what cost factors have been "tweaked" and why. Focus on the
    "big ticket" items. Using expert opinion and "rules of thumb," are
    any significant cost factors outside the range of reasonableness?
    For instance, it is very easy to calibrate a software cost model's
    cost (hours or dollars) per line of software code. Is the CER
    reasonable for the estimate? Since, models may define a software
    line of code differently, it is important to understand the
    definitions used in the model being evaluated.

### Rules Of Thumb Use in Cost Analysis

Continuing to refer to NASA's Handbook, rules of thumb are general
rules, CER's or other cost factors that are widely used, but do not
apply precisely to a specific case. Each estimating organization will
have its own peculiar set. Most long time estimators can develop a rapid
estimate (a ROM) based on their own personal rules of thumb. These rules
come from their own experience. Examples of rules of thumb are the
"default" factors that populate many parametric models including the
commercial ones. If no factor input is available, the model uses a
universally derived factor - one taken from a universal database or
expert opinion.

Although there is nothing wrong with the use of these rules per se, the
danger is evident. The rules are only norms or benchmarks, and they will
never apply to a specific estimate. Their value lies in the comparison
of a factor in a model to the universal case. Too much deviation should
be investigated. Why is the actual case so different from the rule or,
why doesn't the rule apply here?

For instance, if the rule of thumb is that a line of software code (for
a particular type of software) should take about one-half hour to write
(on average), and the parametric model indicates two hours, then some
investigation is in order. Although either CER could be the correct one,
there is far too much deviation from the norm, and suspicions are
aroused. However, if the model indicates six tenths of an hour compared
to one-half hour, the rule has performed the "sanity" check. The use of
the rule does not preclude a thorough cost analysis, however, if a cost
driver has been identified.

Rules of thumb are important, but they must be used wisely. Each analyst
should develop his/her own list. This can be accomplished through
experience, or by consulting with experts in the field. Many rules exist
embedded within commercial parametric models, and are available to the
users and modelers. In any event, it is important to have a list of
benchmark factors. There will probably be a different list for each
organization. Organizational and product differences require it. There
are no shortcuts or magic formulas, so use the rules as a quality check,
or for quick, rough order of magnitude estimates.

### Parametric Cost Analysis Example

What follows is an example of some of the above discussion, based upon
an example in NASA's Handbook. The example here uses the software cost
estimating model, REVIC.

The fundamental REVIC equation is: Man Months = A\*(KDSI)^B^\*ΠFi,
where,

-   A and B are coefficients of several Software Development Modes (for
    instance, embedded, organic, Ada);

-   KDSI is Thousand Delivered Source Instructions; essentially SLOC,

-   Fi is the product of various environmental factors the model uses to
    adjust the cost estimates. These environmental factors include:
    analyst and programmer capabilities, applications experience,
    programming language, storage constraints, requirements volatility,
    reliability requirements, database size, product complexity, the use
    of modern programming practices and software tools, platform
    (airborne, space, etc.), schedule compression, etc. Each factor is
    given a factor value from very low through nominal to very high.

> Suppose the following cost estimate were included in the proposal:
>
> REVIC = IMS EP 1 = 3.312(DSI/1000)^1.2^ \* ΠFi
>
> IMS EP 1 = (3.312)(1337/1000)^1.2^ \*.874 = 4.1PM

The **basis of estimate (BOE)** says this: "The Integrated Management
System Evaluation Package for Release 1 (IMS EP 1) requires 1337 new
lines of embedded source code of new algorithms and uses complex
interfaces. Our superior programming staff using state-of-the-art
software tools, modern programming practices, and possessing significant
application experience allows the product of the environmental factors
to be less (13%) than nominal." Some observations can be noted about
this estimate. First, this particular estimate is relatively small (4.1
person-months). Although there may be more important cost issues, this
one is analyzed as a "spot check". Additional information about REVIC is
available in Appendix 19D and E on the Internet. REVIC is available free
to government agencies/contractors, so a lot of information is
available, including a copy of the model and its user manual. The
resident expert can confirm that the technical description in the BOE is
correct. That and a comparison with Table 19D-2 indicates that the
model's values for A and B are the correct ones for an embedded system.
A quick calculation can compare the estimate with a rule of thumb
supplied by the resident expert.

4.1PM \* 155hrs./PM ÷ 1337DSI = .475hrs/DSI

Lets assume that the resident expert believes this factor to be quite
low, possibly two times. He also indicates that the DSI number needs to
be verified for accuracy, since it is a major cost driver for this
estimate. The **environmental factors (EFs)** for the model do not
appear to have been calibrated except for the vague references
"\...superior programming staff\...." in the BOE. Since the product of
all the EFs nominal values equals 1.4 (referring to a copy of REVIC 9.2
program), it would appear that about 75% of the twenty EFs were set to
the nominal value. He is also suspicious of the "\...less (13%) than
nominal" environmental factors product. (Nominal here he believes should
equal at least 1.5 due to the type platform involved.) Therefore 13%
less appears to be a mistake of some kind (1.4-.874 = .526, or 36%), but
it will have to be investigated. He also argues that since this is a
competitive proposal that is almost 80% software, all software estimates
may be "aggressive," and should be carefully reviewed.

All these questions could be investigated as part of the analysis.
Although this is a relatively easy example, it demonstrates the general
idea of what to look for in a parametric estimate.

### COTS Special Pricing Considerations

See Appendix 19E for a detailed discussion of the areas that are special
and should be considered in pricing a proposal that is for or includes
COTS items.

### Software Pricing Summary

Pricing is used by procuring officials to establish a "fair and
reasonable price". Price analysis and cost analysis are the two basic
techniques used to accomplish this purpose. Price analysis should always
be performed. Under certain circumstances, cost analysis also needs to
be performed. Price analysis plays the lead role in determining price
reasonableness and fairness, and it becomes the responsibility of the
analyst to research and gather pricing data from sources other than the
contractor. As a result, the analyst must understand the fundamentals of
performing price analysis along with the peculiar aspects associated
with software cost estimation. In order to conduct software price
analysis, the terms for comparing the data must be properly defined.
There are three common Pricing Tools normally used for Price Analysis:
ICEs, comparison with a similar program and an automated model
(spreadsheet). In addition to the general cost analysis considerations,
there are three key considerations that apply to most proposals for a
software program: software development productivity, code condition
(percent new, reused code) and the software defect rate by program
phase. The software cost estimation techniques/methods contained in this
chapter should provide the analyst with the background required to
understand a software program cost estimate and this section provides
specific areas on which to focus the software pricing effort.

## Model Selection

The purpose of this section is to outline a process that can aid the
estimator in selecting the best model for his/her particular needs. With
the multitude of software cost and sizing models available, the model
user faces the dilemma: "Which model should I use?" In his article "A
Common Sense Approach to Software Cost Estimation, " Daniel Ferens
outlines a four step approach to help the user determine the answer to
this often perplexing question. The four steps are:

-   Determine needs,

-   Select candidate models,

-   Choose the most appropriate model or models, and

-   Reevaluate the choice.

### Step 1: Determine Needs

This first step is the most crucial. Several studies have concluded that
different models are best for different applications. Therefore, a user
must understand his or her unique situation before selecting a model.
The user should first write a general statement of the organization's
needs, then attempt to describe the situation in more detail. A
"weighted factors approach" as illustrated in **Table 19-11** can help a
user better define his or her unique situation. Also, accuracy
requirements should be defined.

The list of factors and weightings in Table 19-11 reflect the importance
of these factors to the user's organization. They are only presented as
an example; the factors and weightings for other organizations will
often be quite different.

Also, the listing of factors and assignment of weightings can be
subjective. Still, such a listing can provide the user or organization a
framework for considering qualitative evaluation factors. When deciding
which factors to include, consideration should be given to not only to
the use of the model for cost estimating, but also how useful it will be
supporting the software pricing process outlined in section 19.7. The
data requirements from section are especially important.

Table 19-11. The Weighted Factors Approach

  --------------------- ---------------- ----------- ------- ------- ------------ ------- -------
                        **Importance**   **Model                     **Factor             
                                         Ratings**                   Products**           

  **Factor**            **Rating**       **A**       **B**   **C**   **A**        **B**   **C**

  Input Data            10               10          9       7       100          90      70
  Availability                                                                            

  Model Functionality   9                10          6       7       90           54      63

  Ease of Use           8                8           9       6       64           72      48

  Ease of Calibration   6                2           5       5       12           30      30

  Database Validity     5                7           7       4       35           35      20

  Currency              5                3           5       5       15           25      25

  Accessibility         4                6           9       4       24           36      16

  Range of              2                1           7       10      2            14      20
  Applicability                                                                           

  Ease of Modification  1                3           4       2       3            4       2

                                         Weighted                    345          360     294
                                         Totals                                           
  --------------------- ---------------- ----------- ------- ------- ------------ ------- -------

According to the NASA Handbook, the accuracy of the estimates the model
provides is dependent upon the input data provided by the user,
therefore, input data availability is critical. For government
estimators, being able to assess a contractor's software development
personnel capabilities is usually an unknown, especially before contract
award. The other cost drivers in the model should be carefully reviewed
to determine what data are/will be available to the government.

Major functional capabilities that should be considered when selecting a
software estimating tool are listed below. Depending on the
organization\'s needs, the level of significance of these capabilities
may differ, and should be considered accordingly. In addition, the
organization should analyze their own needs and identify additional
desired capabilities specific to them. The organization should then
match available tools with overall needs.

In general, the tool should:

-   **Allow easy adaptation to an organization\'s development
    environment.** This means the tool needs to be capable of being
    customized to fit the using organization\'s development environment.
    Customization includes allowing the developer to define applicable
    inputs, as well as to modify coefficients and exponents of the
    equations used by the tool. This feature will allow continuous
    improvement to the estimation capability of the tool since the
    organization\'s historical data and current project data will be
    included in the software estimate generated.

-   **Be relatively easy to learn and use.** The tool should be well
    documented including methodologies and equations used. Documentation
    should be at a level that is understandable. The tool should include
    help menus and examples sufficient to assist the support staff in
    answering questions and providing training. The amount of formal
    training required to use the tool should be relatively minimal,
    required inputs should be well defined, and visibility into internal
    equations and theories should be provided.

-   **Provide early estimates.** The tool should be capable of
    generating estimates early and quickly in the life cycle process
    when requirements and development environments are not fully
    defined. The tool should also allow task detail to be added
    incrementally as functions, activities, and other information
    becomes more completely defined. Since there are many unknowns early
    in the estimating process, the tool should reflect degrees of
    uncertainty based on the level of detail input (risk analysis). In
    general, the tool should provide sufficient information to allow
    initial project resource planning as well as reasonably early
    \"go/no go\" decisions.

-   **Be based on software life cycle phases and activities.** The tool
    should be capable of providing estimates for all phases and
    activities of the most commonly used software life cycle models. It
    should allow the organization to decompose and map software
    development tasks into those phases and activities, as well as
    support a program WBS. In addition, it should allow for \"what if\"
    situations and include factors for design trade-off studies.

-   **Applicability allows for variations in both application languages
    and functions.** It is very important that the tool provide
    estimates specific to the application of the software project since
    the associated estimating equations, cost drivers, and life cycle
    phases should be unique to each application. General application
    categories include **Information Systems (IS)**, simulation and
    modeling systems, real-time systems, accounting systems, and systems
    based on higher-order languages.

-   **Provide accurate size estimates.** The size of a software
    development project is a major cost driver in most estimating tools,
    yet size is one of the most difficult input parameters to estimate
    accurately. The tool should include the capability to help estimate
    the size of the software development project, or at least help
    define a method for estimating the size.

-   **Provide accurate schedule estimates.** As previously mentioned,
    schedule overruns are common and can be extremely costly. The
    software estimating tool should be able to provide schedule
    estimates accurately. The purpose of scheduling is not only to
    predict task completion given task sequence and available resources,
    but also to establish starting and ending dates for the associated
    work packages and life cycle phases.

-   **Provide maintenance estimates separately.** The software
    estimating tool should be able to provide software maintenance
    estimates as a separate item.

### Step 2: Select Candidate Model

The second step is to select a set of candidate models that meet the
needs determined in Step 1. An examination of needs can point out which
types of models are most suitable. For size estimation, the user can
choose from various categories of models (e.g. analogy, grass roots,
expert judgment, or parametric); however, for cost models, the choices
will probably be restricted to parametric models as explained in section
19.5.5. Once the category or categories have been identified, candidate
models should be selected. For example, the parametric cost models
discussed in Appendix 19D should probably be considered, along with any
others the user determines may be appropriate.

### Step 3: Choose Appropriate Model

The user should perform both qualitative and quantitative (accuracy)
assessments of the candidate models selected in Step 2, and choose the
best model or models for his or her organization. It is recommended that
two models be selected for routine use; one as the primary model and one
for cross-checking the results of the primary model. A study by George
A. Coggins and Roy C. Russell entitled *Software Cost Estimating Models:
A Comparative Study of What the Models Estimate* showed that cost
models, even given "equivalent" inputs, produce significantly different
cost and schedule estimates. Their conclusion was that a user should
learn one or two models well instead of trying to use several different
models. Nevertheless, other models can still be used occasionally, if
only for consideration for future use as discussed in Step 4.

To effectively perform this step, the user must become familiar with
each of the candidate models. This often involves attending a training
course and using the model for several months, if time permits. (A well
planned, on-going training program can benefit a cost estimating
organization tremendously in terms of knowledge of models.) Once the
user becomes sufficiently familiar with the models, the selection
process can begin. It is highly desirable that the user does his or her
own studies for this process, and not rely solely on outside
information. Validation studies performed by outside agencies certainly
can help the user in the model selection process. An excellent example
is a study by Elizabeth K. Bailey, et al, *A Descriptive Evaluation of
Automated Software Cost-Estimation Models*, which compared and evaluated
features of most of the cost models described in Appendix 19D. While
outside studies such as this can provide valuable information, the user
should employ these studies as supplementary material and not as a
substitute for his or her own efforts since they do not reflect the
unique facets of the user's environment.

For quantitative assessments of candidate models, the weighted factors
approach shown in Table 19-11 can help. The user assigns a weight to
each factor (in Step 1), assigns a rating between "1" and "10" to each
model on how well it addresses each factor, multiplies the model and
importance ratings, and sums the results. The highest total can indicate
the best model (e.g., Model B in Table 19-11); however, other models
that are close (e.g., Model A in Table 19-13 ) should be scrutinized
further. Since there is some subjectivity in this process, small
differences may be negligible. Again, while the weighted factors
approach is somewhat subjective, it can help a user consider what is
important in model selection and quantify the rating process.

For qualitative assessments, or determining whether the models meet
accuracy requirements, the user should calibrate the models, then run
the models against projects for which the user has historical data and
which were not used in calibration. An example is Ourada's study
performed on military ground programs with the REVIC model for an early
version of the Air Force Space and Missiles Center database. For the 28
military ground programs, Ourada calibrated REVIC using 14 programs
(selected at random), then ran the calibrated model against the other 14
programs to assess probable accuracy for other military ground programs
outside the database. This approach is often arduous, but is essential
if a user truly wishes to determine which model is most suitable from an
accuracy standpoint.

### Step 4: Reevaluate Model Choice

User needs and models change over time. Many commercial models such as
PRICE-S and SEER-SEM are updated every year, and major refinements occur
every few years. New models occasionally appear and may merely fade
away, or they may become widely used, or may be more suitable than
current models. Therefore, a user should reevaluate his or her selection
every few years. There is no reason to be "married" to a particular
model or models for life unless they continue to be the best available.

### Model Selection Summary

In summary, the four-step approach presented here can be an invaluable
aid to a user in model selection, especially when the choice is not
dictated by those in higher authority. The most crucial step is the
first, needs determination; all other steps hinge on the success of the
first step. The four-step approach is sometimes laborious, but the
benefits of improved estimating can make it worthwhile.

## Intelligent Use of Models

As discussed earlier, there is a plethora of software cost and size
models available, many of which are very sophisticated. This may lull a
manager into an over dependence on models for estimation. According to
Robert E. Park, in his article "An Open Letter to Cost Model
Evaluators," some managers apparently believe that "models, not
estimators, are responsible for estimates". Software models, however,
are not magic boxes; they are only as good as the input data used. Also,
models inherently have limitations. Furthermore, models are not always
useful in analyzing non-cost factors in decision making. The manager,
therefore, must recognize the capabilities and limitations of models and
use them intelligently. Some of these limitations are now discussed.

### Input Data

The problem of the instability of parametric models, as described by
Boehm, is exacerbated by the sensitivity of effort and schedule to
changes in input parameters. For example, in most cost models, changes
in program size will result in at least an equivalent percentage change
in cost or effort. Some inputs have even more dramatic effects. For
example, changing personnel capability (analyst and programmer) in
COCOMO or REVIC from "very high" to "very low" will result in a 300%
increase in effort required. In SEER-SEM, changing security requirements
from the lowest to the highest rating will have a similar effect. In
PRICE-S, changing the Productivity Factor by 0.1 can result in a 20%
change in effort. All models have one or more inputs for which small
changes will result in large changes in effort and, perhaps, schedule.

The input data problem is further compounded in that some inputs are
difficult to obtain, especially early in a program. As discussed in
section 19.4, size must be estimated early in a program using one or
more sizing models. These models usually have not been validated for a
wide range of projects. Some sensitive inputs, such as analyst and
programmer capability, are subjective and often difficult to determine.
Studies like one performed by Brent L. Barber, *Investigative* *Search
of Quality Historical Software Support Cost Data and Software Support
Cost-Related Data*, show that personnel parameter data are difficult to
collect. Figure 19-13, extended from the *SEER-SEM User's Manual* shows
the relative impact on cost/effort of the different input parameters for
that model. Even "objective" inputs like Security Requirements in
SEER-SEM may be difficult to confirm early in a program, and later
changes may result in substantially different cost and schedule
estimates. Some sensitive inputs such as the PRICE-S Productivity Factor
and the SLIM PI should be calibrated from past data. If data are not
available, or if consistent values of these parameters cannot be
calibrated, the model's usefulness may be questionable.

> Figure 19-13. Example of Relative Cost Driver Impacts

**Error! Not a valid link.**

A special problem confronts Government managers who must obtain input
data from contractors. First of all the Government manager should
include a data collection form in the contractual requirements. The
recommended minimum input data requirements are shown in Table 19-11 in
the software pricing section. However, an analyst may be tempted to run
a model using the data provided on the form. This will merely result in
a duplication of the contractor's inputs and proposed cost. A better use
of this data would be as a "starting point" to conduct further
discussions with the person or group who filled out the form. The form
should be as generic as possible so as to collect the type of data
needed for a variety of models, not just a specific model. Of course, if
the model is known, then a specific input form makes sense. An example
of a generic form is the data form included in the *Space and Missiles
Center Software Database User\'s Manual*. A potential drawback to such
generic forms is that they tend to be lengthy and cumbersome to complete
and use.

### Model Accuracy

Even if "perfect" input data were available, models may not produce
accurate results. Studies have compared model estimates with \"known\"
input data and actual cost and schedule information, and have not found
model accuracy to be scintillating. For example, a 1989 study performed
by ITT Research Institute for eight Ada programs, *Test Case Study:
Evaluating the Cost of Ada Software Development*, showed that the most
accurate models for that type of application (SoftCost-Ada and SASET)
were accurate to within 30% of actual costs only 57% of the time. The
Ourada study showed even worse results for SEER-SEM, SASET, and REVIC
for the 28 military ground programs in an early edition of the Space and
Missiles Center database. One limitation of these results is that the
models were not calibrated. One of the SEI software cost estimating
requisites is that cost models be calibrated/tuned to reflect
demonstrated accomplishments on past projects.

Calibration can and does improve model accuracy. A 1981 study by Robert
Thibodeau entitled *An Evaluation of Software Cost Estimating Models*
showed that calibration could improve model accuracy by up to 400%.
However, the average accuracy was still only 30% for an early version of
SLIM and 25% for an early version of PRICE-S. Furthermore, this accuracy
was only demonstrated for certain types of programs. Likewise, the ITT
Research Institute study showed that the accuracy of PRICE-S and
System-3 (a forerunner of SEER-SEM) improved with calibration, but only
to within 30%, 62% of the time. For development cost or effort
estimation, therefore, a manager probably cannot expect the models to be
accurate for budgetary purposes (i.e., within 10%), at least for new
programs.

Software maintenance cost accuracy is in an even worse state. An Air
Force study performed by Ferens in 1983 and published in the ISPA
Journal of Parametrics, concluded that no software support cost models
could be shown to be accurate. Unfortunately, the results of that study
have never been refuted. Maintenance cost estimating is further
complicated in that traditional estimating practices do not always
apply. For example, most models estimate cost (or effort) and schedule
given some measure of the work to be done (such as program size). In the
block change process used by the Air Force in supporting weapons
software, however, the time and number of personnel are fixed, and the
amount of work that can be done within those constraints must be
estimated. Most cost models cannot address this situation (although the
latest editions of SoftCost-OO and SEER-SEM may be useful). The software
support estimation problem is further convoluted by lack of quality
software support cost data for model development, calibration, and
validation.

Even if models can be shown to be accurate, another effect must be
considered. According to Tarek Abdel-Hamid and Stuart E. Madnick in
their book *Software Project Dynamics: An Integrated Approach*, "a
different estimate creates a different project." What a cost model
predicts may dictate how a project is managed. For example, if a model
predicts an unrealistically high effort and schedule for a program, and
management uses the model results to plan and manage the program, then
it may actually take longer and cost more than necessary. Abdel-Hamid
shows that the use of a "safety factor" (analogous to management
reserve) of 25% to 50% in software estimates will result in 15% to 35%
increases in actual person-months.

### Model Calibration

The general requirements and procedures for calibrating and validating a
cost model are the same for software cost models. This section provides
additional details regarding the software cost model calibration
process.

NASA's Handbook defines calibration as the process of determining the
deviation from a standard in order to compute the correction factors.
For cost estimating models, the standard is considered historical actual
costs. The calibration procedure is theoretically very simple. It is
simply running the model with normal inputs (known parameters such as
software lines of code) against items for which the actual cost is
known. These estimates are then compared with the actual costs and the
average deviation becomes a correction factor for the model. In essence,
the calibration factor obtained is really good only for the type of
inputs that were used in the calibration runs. For a general total model
calibration, a wide range of components with actual costs need to be
used. Better yet, numerous calibrations should be performed with
different types of components in order to obtain a set of calibration
factors for the various possible expected estimating situations. For
instance, the PRICE Software Model addresses this situation using
internal productivity factors. These can be modified by the calibration
process.

According to SEI, organizations that acquire software intensive systems
should have a corporate memory or historical database and a process for
calibrating their cost models to reflect demonstrated performance of
similar past projects.

The act of calibration standardizes a model. Many models are developed
for specific situations and are, by definition, calibrated to that
situation. Such models usually are not useful outside of their
particular environment. However, general cost estimating models
including commercially available ones such as the PRICE and SEER models
(described in Appendix 19D) are designed to be useful as estimating
tools for a wide range of situations. The act of calibration is needed
to increase the accuracy of one of these general models by making it
temporarily a specific model for whatever product it has been
calibrated. Calibration is in a sense customizing a generic model.

Items which can be calibrated in a model are: product types, operating
environments, labor rates and factors, various relationships between
functional cost items, and even the method of accounting used by a
contractor. All general models should be standardized (i.e. calibrated).

Using historical data as a basis for customizing or calibrating a tool
is essential. Insure that information for current project development
efforts are saved for future reference. At a very minimum, use software
life cycle model phases and activities as a basis for collecting and
maintaining project envelopment information for future tool use. Once
calibration of the model is completed, model validation in accordance
with FAA CEH, chapter 12, section 12.4.2 is necessary to establish the
model's accuracy.

##  Software Project Tracking And Measurement 

Once an estimate has been completed and a project started, it is
imperative that a reporting and measurement system be established to
compare estimated and actual cost, schedule, and performance. Such a
system will not only be invaluable for project management, but also can
provide the data necessary for model calibration. SEI suggests that an
organization have data collection and feedback processes that foster
capturing and correctly interpreting data from work performed and
entering them into an historical database in order to develop reliable
cost estimates.

Unfortunately, past endeavors in this area have not generally been
successful; either the data was not collected or the data collected was
inadequate or misleading. To effectively track and manage software
programs, suitable metrics must be selected, and adequate data
collection procedures must be established. This section discusses which
metrics may be suitable for software management and the requirements for
an effective data collection procedure. The purpose of this section is
to provide a short overview of the subject to highlight the importance
of this area. There are many handbooks and manuals that cover cost,
performance and quality tracking metrics in depth. One excellent
reference is DoD's Joint Logistics Commanders handbook *Practical
Software Measurement: A Guide to Objective Program Insight* (available
for download at [www.psmsc.com)]().

### Cost and Schedule Tracking

A basic system for tracking costs is necessary for effective software
management. One such system which is used widely in Government agencies
(DoD, IRS, NASA) is an **earned value management system (EVMS)**. Figure
19-14 illustrates the basic terms used in an EVMS.

> Figure 19-14. Basic Earned Value Terminology

**Error! Not a valid link.**

In the beginning of a program, a manager determines the total program
dollars available (BAC) and a time phased expenditure profile (BCWS),
which considers both the amount to be spent during each period (e.g.,
monthly) and the amount of work to be accomplished during that period.
The time phased plan is expressed in the common denominator of dollars,
but represents work scheduled to be accomplished by those dollars. This
can be determined with the aid of one of the cost models discussed in
Appendix 19D. Most of the software cost estimating models generate
monthly schedule and resource usage profiles. The BAC is not usually
equal to the TAB; the manager usually sets aside some money for
management reserve that is used to cover risk and uncertainty.

As the program progresses, the manager also tracks ACWP, which is the
actual amount spent on the program, and BCWP, which is the value of the
work actually performed or "earned value". To determine how well the
program is progressing compared to what was planned, the manager
computes a cost and schedule

variance at the end of each reporting period using the following
formulas:

> \% Cost Variance = 100 \* (BCWP - ACWP) / BCWP
>
> \% Schedule Variance = 100 \* (BCWP - BCWS) / BCWS

Negative variances are an indication of problems with the program,
especially if they are large. In the DoD environment, it is frequently
mandated that the contractor explain variances greater than 10% to the
managing DoD agency, including what action is planned to alleviate the
cost and schedule problems.

There are two challenges to using earned value for software. One is
that, to be effective, data must be reported down to at least the CSCI
level! If a CSCI is an entity that is managed separately, then it must
be tracked separately. Unfortunately, many government projects in the
past did not track costs to the CSCI level; therefore, the actual cost
of these CSCIs is still unknown. Furthermore, cost and schedule overruns
have been the hallmark of government software programs. Perhaps this
situation would have been alleviated if an earned value management
system had been used at the CSCI level to demonstrate problems early in
a program where they are easier and much less costly to correct. The
second challenge is in measuring work performed: How can a manager
assess earned value for software programs? Most of the remainder of this
section discusses metrics that can be useful with performance
measurement systems to help assess earned value.

### Performance Measurement Indicators

Performance measurement indicators are metrics which help measure
performance, or work performed, in contrast to quality metrics which are
discussed later. For all metrics, there is currently a plethora of books
and articles; however, there is not an abundance of standardization or
agreement among writings. Some of these articles are listed in the
chapter list of references. The Software Engineering Institute also has
published a *Software Measurement Guidebook* \[CMU/SEI-97-SR-019\] that
provides tracking and monitoring methods to evaluate status and earned
value for projects. It is up to the manager responsible to independently
determine a set of appropriate metrics for his or her program.

In selecting metrics, a manager should first consider what constitutes a
\"good\" metric. Several writers have proposed general criteria for
"good" metrics. For example, Tom DeMarco, in his book *Controlling
Software Projects*, states that metrics should be measurable, or
quantifiable; independent from influence by project personnel;
accountable, in that data can be collected; and precise, in that the
degree of exactness can be specified. Watts S. Humphrey, in *Managing
the Software Process*, states that metrics should be objective (versus
subjective), explicit (versus derived), absolute (versus relative), and
dynamic (versus static). Capers Jones categorizes metrics as hard, soft,
and normalized. Hard metrics can be quantified with little subjectivity;
soft metrics require human judgment; and normalized metrics are hard
metrics which should only be used for comparison and not for absolute
measures. Hard metrics, of course, are most desirable, but programs may
require some soft or normalized metrics. In their article "Using Earned
Value for Performance Measurement on Software Development Projects,"
David S. Christensen and Daniel V. Ferens have proposed a list of
metrics criteria, partly based on surveys of program managers at the Air
Force's Aeronautical Systems Center. Their research was based on the
thesis of Bradley J. Ayres and William M. Rock entitled *Development of
a Standard Set of Software Indicators for Aeronautical Systems Center*.
The criteria for metrics include explicit, absolute, and objective, as
discussed above. Two other criteria are that the metric be timely, or
available early in a program; and relevant, or appropriate for the
specific program being measured.

The surveys by Ayres and Rock identified relevance as the key criterion
for a metric. The next challenge for a manager, therefore, is to
determine which metrics are most relevant for his or her program. Again,
the 1992 study by Ayres and Rock identified a set of metrics which were
most relevant for Aeronautical Systems Center programs. Christensen and
Ferens refined the list of metrics and determined their suitability for
use in earned value programs. The seven metrics and their roles in
earned value management are explained in the text that follows. Table
19-12 summarizes how well they meet the criteria as a qualitative
metric.

Table 19-12. Summary of Metrics and Qualities

  --------------- -------------- -------------- --------------- -------------- ------------ -----------------
                  **Quality**                                                               

  **Metric**      **Relevant**   **Explicit**   **Objective**   **Absolute**   **Timely**   **Independent**

  Requirements    Yes            Yes            Somewhat        Yes            Yes          Somewhat
  and Design                                                                                
  Progress                                                                                  

  Code and Test   Yes            Yes            Somewhat        Yes            No           Somewhat
  Progress                                                                                  

  Person-Months   Yes            Somewhat       Yes             Yes            Yes          Somewhat
  of Effort                                                                                 

  Program Size    Yes            Somewhat       Yes             Yes            Yes          Somewhat

  Computer        Yes            No             Somewhat        Yes            Some-what    Yes
  Resource                                                                                  
  Utilization                                                                               

  Requirements    Yes            Yes            Yes             Somewhat       Yes          Yes
  Stability                                                                                 

  Design          Yes            Yes            Somewhat        Somewhat       No           Yes
  Stability                                                                                 
  --------------- -------------- -------------- --------------- -------------- ------------ -----------------

##### Requirements and Design Progress 

This metric illustrates the number of CSCI requirements delineated
during requirements analysis and preliminary design. The requirements
are described in two documents: the Software Requirements Specification
(SRS) and the Software Design Document (SDD). The planned requirements
can be used for laying out a software development plan. As the CSCIs are
written, they can be used as metrics to show progress in the development
effort. For instance, if there are 100 CSCIs of approximately the same
size planned to accomplish a certain requirement, the budget can be
divided among those CSCIs. Progress would be measured by earning the
value of that CSCI as it is written. This metric can be complicated
because the total requirements may change during a program, and counting
"completed" requirements can be difficult. If these limitations can be
overcome, this metric is very valuable in assessing progress and
demonstrating any problems early in a program.

##### Code and Test Progress

This metric illustrates the number of computer software units (CSUs)
that have successfully been designed, coded and tested. This information
is available from software development files or similar documentation.
As in the requirements and design progress metric, planned and actual
CSUs can be used to plan and measure progress. This metric is usually
easier to assess than the requirements and design progress metric, but
is not available until later in the program.

##### Person-months of Effort

This metric compares planned and actual person-months expended for a
project and is useful in computing actual costs. It can also be useful
in assessing overall program health, since using too many people for a
project can result in cost overruns, while using too few can result in
schedule slippage.

##### Program Size 

This metric tracks the total size of the program (in either SLOC or
function points), and the percentages of code that are new, reused, or
modified. Since there is a direct relationship between size and effort,
this metric may be helpful in assessing cost variances. For instance, if
the SLOC are higher at a given point than were initially projected, this
could help explain a cost overrun. However, this metric has drawbacks as
a progress measurement tool since it lends itself to misuse because
programmers can write additional code to feign progress; thus the false
impression that progress is occurring is given. Another problem with
using program size is that, early in a program, size must be estimated
with one of the techniques discussed in section 19.4, which may not be
very accurate. Therefore, using it to measure progress or even plan work
is a questionable practice. For this purpose, it is best to use program
size as a technical parameter to investigate cost variances based on
other metrics. On the other hand, for pricing purposes, tracking program
size is important to insure that current and projected size are
consistent with the proposed level of effort.

##### Computer Resources Utilization

This metric is a measure of computer memory, timing, and input/output
(I/O) channel capacity consumed by the software. It is closely related
to the software size metric in that increases in size will result in
increases in capacities used. Like size, this metric can be useful
throughout the program to analyze cost and schedule variances.
Additionally, it can help a manager perform hardware-software trade-off
studies early in a program.

##### Requirements Stability

This metric is related to requirements and design progress. It tracks
total requirements and the number of changes (additions, deletions, and
modifications) to requirements. Numerous or frequent changes will result
in additional effort consumed with no indication of progress; therefore,
this metric can explain unfavorable cost and schedule variances.

##### Design Stability

This metric is similar to requirements stability and is related to code
and test progress. It tracks total CSUs and the number of changes.
Frequent changes result in additional effort consumed; therefore, this
metric, like requirements stability, can help explain unfavorable
variances. Like code and test progress, a limitation of this metric is
that it is not useful until later in a program.

These seven metrics are not the only metrics available for performance
measurement. According to Lloyd K. Mosemann in his paper *Guidelines for
Successful Acquisition and Management of Software Intensive Systems*,
document completion can be used in determining whether milestones have
been met. According to Ferens and Mark T. Hunter in their article "Use
of Cost plus Award Fee Contracts for Software Development Efforts", some
performance metrics can also be useful in assessing award fees for
cost-plus-award fee contracts. Among them are the ones discussed above,
along with compliance with development plans and number of change
reports successfully resolved. As stated earlier, a manager must
determine the best metrics for his or her program; however, the set of
seven discussed above is a useful starting point.

### Software Quality Metrics

In addition to measuring performance, a manager frequently wants to
measure the quality of a program. There are many metrics that can help a
manager in these endeavors. Quality metrics can also be used with
performance indicators to assess causes of cost and schedule variances.
Several general categories of quality metrics and examples in each
category are now discussed.

##### Defect Metrics

Several cost models, including PRICE-S, SEER-SEM, SLIM, and CHECKPOINT,
estimate defects in addition to software cost and schedule. SLIM and
CHECKPOINT are especially sophisticated in this area; they provide
defect estimates for the entire life cycle of a software program. If the
predictions of these models can be trusted, a manager can use defect
metrics to compare estimated and actual defects. Also, some
organizations such as the Air Force require that defect information be
collected. Although often associated with coded products, defect metrics
can also be used during requirements and design to measure errors in
documentation or other products generated during these phases.

One method of collecting defect information, as described by Mosemann,
is to track not only the total number of defects, but also the number of
defects open and closed, in order to assess how well an organization is
correcting problems. According to Jones, it is also useful to track the
"bad fix rate", the portion of fixes that either do not correct the
problem or result in a problem somewhere else in the program or system.
Other useful defect measures, according to Jones and Putnam, include
defect severity levels and causes, defect distribution among modules
(CSCs and CSUs), and defect removal efficiency, which may be assessed
from the number of total and closed defects. As mentioned in section
19.7.5, defect metrics can be useful during software cost analysis to
help the analyst determine whether the contractor's estimated effort for
identifying and fixing defects is consistent with the
contractor's/industry's historical averages.

##### Complexity Metrics

These metrics measure the relative complexity of a program and can be
useful in discovering problem areas. An example of complexity metrics,
developed by Thomas J. McCabe and G. Gordon Schulmeyer is discussed in
their paper "System Testing Aided By Structural Analysis." Such metrics
measure the number of independent paths in a program or module, with
"essential complexity" adjusted for use of structured coding techniques
such as case or do-while statements. Lower McCabe's values indicate
simpler design and, hence, less effort. Other complexity metrics,
according to Jones, include Haistead's metrics, logical complexity, and
design entropy, which is the tendency of the design to deteriorate when
changes are made. One advantage of complexity metrics is that they are
sometimes available relatively early in a program.

##### Module Metrics

These metrics measure properties of individual modules, such as CSCs or
CSUs, and are useful during the design phases of a program. According to
Roger S. Pressman in his book *Software Engineering: A Practitioner\'s
Approach*, two useful module metrics are degree of cohesion and degree
of coupling. Cohesion measures the internal strength of a module; and
higher ratings (such as functional cohesion) are better. Coupling
measures the degree of dependence on other modules, and lower ratings
(such as no coupling or simple data coupling) are better. Another useful
measure is span of control, or the number of modules directly under the
control of one module. Values ranging from five to nine are usually
desirable.

##### Testing Metrics

In addition to defect measures collected during testing, some other
testing measures can be useful. Some testing metrics are also useful in
performance measurement. Breadth of testing, or percentage of
requirements demonstrated, and depth of testing can help a manager
determine the confidence level he or she can place in a product. Test
sufficiency, or an estimate of the number of errors remaining in a
program, can be useful, but is often difficult to estimate. Seeded error
discovery is a technique where an engineer independently inserts errors
into a program that are supposedly representative of the types of errors
expected. A measure is taken of the number of unseeded and seeded errors
discovered; this measure can be used to determine total errors and
testing efficiency. It can be problematic, however, in that seeded
errors do not always match the types of errors that actually do occur.
Also, the developer must insure the final product is not released with
seeded errors still present!

##### Product and Operational Metrics

Although they are not available until the end of program development,
product metrics are useful to a customer to assess the quality of the
developed product. One example is adequacy of documentation, especially
for final product documents such as the Software Product Specification
or the User's Manual. Another is the number of known problems remaining;
software is often delivered with known problems (although this is highly
undesirable). A related metric is product reliability, either measured
in error count (e.g., number of defects per thousand SLOC) or
**mean-time-to-defect (MTTD)**, a software equivalent of the hardware
mean-time-between-failure measure.

When the software is operational, defects and MTTD should be continually
tracked for quality assessment. Also, time to correct errors and
availability can be useful measures. Finally, according to David P.
Youll in his book *Making Software Development Visible*, the number of
user problem reports can be a beneficial indicator of software quality.
Youll recommends tracking the number and frequency of problem reports as
well as the cause, severity, and time to correct problems.

### Data Collection

Selection of metrics for a program is of little use if there is no way
to collect the data for them. Therefore, an effective data collection
system must be established. According to Humphrey, an effective data
collection system, like software development, must begin with a plan and
have specific objectives. The objectives of data collection can include
understanding of a process, evaluation of a program, control of a
program, or prediction of future trends and events. The data collection
plan must have management support and the data collection procedures
developed must consider the impact on the entire organization. Since
data collection is an added expense in dollars and personnel time, the
support of management is essential. Meanwhile, support by people
actually collecting the data is also necessary, or they will perform a
minimal effort and the data will probably be of questionable validity.

According to Putnam, effective data collection requires that data be
collected at regular intervals, be used to measure "actuals", and
compare estimates and actuals to assess deviations (in other words,
employ a performance measurement system). Also, a commitment to
investigate deviations and act upon results is essential. The data
collection itself can be augmented by some of the myriad of automated
tools currently available. Taking action, however, is still a personnel
issue for which there is no automation.

Data collection requirements can be placed in the contract. Before this
is accomplished, the agency must determine what metrics are to be used.
For the Air Force, there is currently a policy that requires that
certain \"core\" metrics be collected, and recommends several others be
collected. Sometimes these can be negotiated with the developing
contractor. Again, it is tantamount that software data be collected to
at least the CSCI level!

One additional benefit of data collection is that information for model
calibration can be obtained. Currently, the Air Force's Space and
Missiles Center is managing a large database of over 2,500 programs with
historical size, effort, and other data that can be used in model
calibration and validation. Since all model developers recommend
calibration of their cost models to an organization's peculiar
environment, efforts to collect calibration data should be actively
pursued.

### Software Tracking and Measurement Summary

To effectively manage a software effort, a manager must collect relevant
data to at least the CSCI level. Fortunately, there is a profusion of
metrics available for software tracking and measurement. Unfortunately,
there are few standardized lists of metrics available that cover a broad
range of applications. The manager must select from the many available
metrics that are most suitable for his or her program. Additionally, a
manager must establish a viable data collection system that is supported
by management and by the personnel responsible for collecting the data.
Data for model calibration should be considered in software data
collection efforts.

## Summary

This chapter was written to provide the FAA analyst with an introduction
to the basic concepts of software cost estimating/pricing. In other
words, this chapter is a starting point in the very specialized and
involved field of software cost estimating/pricing. Before an analyst
attempts to review a FAA software cost estimate by applying the
techniques or models discussed herein, two words of caution: GET HELP!
As was shown in the chapter, even experts in this field, using the most
sophisticated techniques and metrics, along with a model calibrated to
the organization developing the software, can accomplish an estimate
[that does not accurately]{.underline} predict the actual cost of a
software development program.

Figure 19-15, extended from Boehm's book *Software Engineering
Economics* indicates that the accuracy of the software cost/size
estimate creates a great degree of uncertainty in the early stages of a
program. This is because one may not know the specific nature of the
product to be developed to better than a factor of 4. As the life cycle
proceeds, and product decisions are made, the estimate becomes
increasingly more accurate because the nature of the products and its
consequent size are better known, and the nature of the process and its
consequent cost drivers are better known.

Figure 19-15. Accuracy of Cost Estimate versus Development Phase

**Error! Not a valid link.**

This chapter has discussed the software estimating process and the
various methodologies used in software estimation. The basic software
estimating functional capabilities were also discussed. A review of
product literature and user manuals indicates that many tools will
perform most of the functional capabilities outlined in this chapter.
The user must be able to customize the software estimating tool to their
own software development environment. This requires collecting
historical data from past completed projects to calibrate the software
tool.

The use of two or more software estimating tools using different
methodologies is recommended. The user should select a primary tool for
software estimating and an alternate tool for comparison and validation.
These tools should be used throughout the software development process.
Parametric tools are considered to be the best for software estimating
for the following reasons:

-   Equations are based on previous historical development projects.

-   Outputs are repeatable and formulas can be analyzed.

-   They can be customized to fit the user\'s environment.

-   They require minimal time and effort to use.

-   They are particularly useful in earlier phases of software
    development.

Assistance can be obtained in several ways. First and foremost,
soliciting the guidance of cost estimators and engineers who have
previous experience with software cost estimating/pricing and software
cost estimating tools is critical. Their expertise can help the analyst
in reviewing the estimate and often may be able to provide data from
similar historical programs. Second, once a commercially available
software model has been selected, enlist the services of their technical
staff to ensure the model is used properly. Also, software companies
generally offer classes on how best to use their model. The classes can
vary in duration from a day long seminar to a couple of weeks at their
facility. And finally, the analyst can do some reading on his or her
own. This chapter referenced many good sources to allow the analyst to
develop a working knowledge of the software cost estimating/pricing
discipline.

Price analysis now plays the lead role in determining price
reasonableness and fairness, and the responsibilities of the analyst
include researching and gathering pricing data from sources other than
the contractor. As a result, the analyst must understand the
fundamentals of performing price analysis along with the peculiar
aspects associated with software cost estimation. In order to conduct
software price analysis, the terms for comparing the data must be
properly defined. It is also important to understand the special
considerations for integrating COTS components into a software system
especially as they relate to software cost estimation and pricing.

Finally, according to SEI, there are six institutional process
requisites that organizations in the business of building or acquiring
software-intensive systems must possess if they are to consistently
produce reliable cost estimates. These requisites are:

-   A corporate memory, or historical database(s), for cataloging cost
    estimates, revisions, reasons for revisions, actuals, and other
    descriptive information, such as any constraints or trends that
    affect the project;

-   Structured processes for estimating software size and the amount and
    complexity of existing software that can be reused;

-   Cost models calibrated/tuned to reflect demonstrated accomplishments
    on similar past projects;

-   Audit trails that record and explain the values used as cost model
    inputs;

-   Processes for dealing with externally imposed cost or schedule
    constraints in order to ensure the integrity of the estimating
    process; and

-   Data collection and feedback processes that foster capturing and
    correctly interpreting data from work performed.

**19A. Acronyms**

**4GL:** Fourth Generation Language

**ACEIT:** Automated Cost Estimating Integrated Tools

**ACWP:** Actual Cost of Work Performed

**AMS:** Acquisition Management System

**BCWP:** Budgeted Cost of Work Performed

**BCWS :** Budgeted Cost of Work Scheduled

**BFP:** Basic Feature Point

**CDRL:** Contract Data Requirements List

**CDR:** Critical Design Review

**CER:** Cost Estimating Relationship

**CMM:** Capability Maturity Model

**CO:** Contracting Officer

**COCOMO:** COnstruction COst Model

**COCOTS:** COnstruction COTS

**COTS:** Commercial-off-the Shelf

**CPM:** Critical Path Method

**C/SCSC:** Costs/Schedule Control System Criteria

**CSC:** Computer Software Component

**CSCI:** Computer Software Configuration Item

**CSU:** Computer Software Unit

**DDE:** Dynamic Data Exchange

**DoD:** Department of Defense

**EA:** Evolutionary Acquisition

**EI:** External Inputs

**EIF:** External Interfaces

**EO:** External Outputs

**EQ:** External Inquiries

**EVMS:** Earned Value Management System

**FAA CEH:** FAA Cost Estimating Handbook

**FAA PH:** FAA Pricing Handbook

**FAQ:** Frequently Asked Questions

**FCA:** Functional Configuration Audit

**FPA:** Function Point Analysis

**FPC:** Function Point Count

**FPH:** FAA Pricing Handbook

**GAO:** U.S. General Accounting Office

**GUI:** Graphical User Interface

**HWCI:** Hardware Configuration item

**HOL:** Higher Order Language

**ICE:** Independent Cost Estimate

**IEEE:** Institute of Electrical and Electronics Engineers

**IFPUG:** International Function Point User\'s Group

**ILF:** Internal Files

**IRS:** Interface Requirement Specification

**IS:** Information System

**KDSI:** Thousands Delivered Source Instructions

**LAN:** Local Area Network

**LCC:** Life Cycle Cost

**MTTD:** Mean-Time-To-Detect

**NASA:** National Aeronautics and Space Administration

**NCCA:** Naval Center for Cost Analysis

**NRaD:** United States Navy's Naval Command, Control, Surveillance
Center, RDT&E Division, Software Engineering Process Office

**OLE:** Object Linking and Embedding

**OO:** Object Oriented

**OOA:** Object Oriented Analysis

**OOD:** Object Oriented Design

**O&M:** Operations and Maintenance

**PC:** Personal Computer

**PCA:** Physical Configuration Audit

**PERT:** Program Evaluation and Review Technique

**RAM:** Random Access Memory

**REVIC:** Revised Enhanced Version of Intermediate COCOMO

**SASET:** Software Architecture, Sizing, and Estimating Tool

**SDD:** Software Design Document

**SDP:** Software Development Plan

**SDR:** Software Design Review

**SEER-SEM:** System Evaluation and Estimation of Resources Software
Estimating Model

**SEF:** Software Estimation File

**SEI:** Software Engineering Institute

**SER:** Size Estimating Relationship

**SIR:** Screening Information Request

**SLIM:** Software Life Cycle Model

**SLOC:** Source Lines of Code

**SRR:** Systems Requirements Review

**SRS:** Software Requirements Specification

**SSCAG:** Space Systems Cost Analysis Group

**SSR:** Software Specification Review

**SSS:** System Segment Specification

**STSC:** Software Technology Support Center

**VHOL:** Very High Order Language

**WBS:** Work Breakdown Structure

**19B. Software Cost Estimation Terminology**

**Actual Cost of Work Performed (ACWP):** The actual direct costs
incurred on a project at any given time.

**Algorithmic Models (also known as parametric models):** produce a cost
estimate using one or more mathematical algorithms using a number of
variables considered to be the major cost drivers. These models estimate
effort or cost based primarily on the hardware/software size, and other
productivity factors known as cost driver attributes.

**Analogy:** A method of estimating developed on the basis of
similarities between two or more programs, systems, items etc.

**Analogy (or Comparative) Models:** Models that use a method of
estimating that compares a proposed project with one or more similar and
completed projects where costs and schedules are known. Then,
extrapolating from the actual costs of completed projects, the model(s)
estimates the cost of a proposed project.

**Annual Change Traffic (ACT):** The fraction of a software product\'s
source instructions which undergoes change during a year, either through
addition or modification. The ACT is the quantity used to determine the
product size for software maintenance effort estimation.

**Baseline:** An established, fixed version of the project plan against
which actual implementation of the project is measured.

**Bottoms-Up (Engineering Estimate or Grass Roots) Models:** A method of
estimation that estimates each component of the project separately, and
the results are combined (\"rolled up\") to produce an estimate of the
entire project.

**Budgeted Cost of Work Performed (BCWP)** **(Earned Value):** The total
value of work performed at any given time.

**Budgeted Cost of Work Scheduled (BCWS) (Budgeted Cost To-Date):** The
total budgeted cost for work scheduled to be completed at any given
time.

**Calibration:** A technique used to allow application of a general cost
model to a specific set of data. This is accomplished by calculating
adjustment factor(s) to compensate for differences between the
referenced historical costs and the costs predicted by the cost model
using default values.

**Computer-Aided Software Engineering (CASE):** Identifies a sector of
the computer software industry concerned with producing software
development environments and tools. The main components of a CASE
product are individual tools that aid the software developer or project
manager during one or more phases of software development (or
maintenance). Other features are a common user interface;
interoperability of tools; and a repository or encyclopedia to provide a
common tool base and central project database. CASE may also provide for
code generation.

**Computer Software Component (CSC):** Since CSCIs may contain over
100,000 lines of code, they are further partitioned into computer
software components (CSCs) and computer software units (CSUs).

**Computer Software Configuration Item (CSCI):** An aggregation of
computer software that satisfies an end-use function and is designated
for configuration management. A CSCI may be broken down into CSCs and/or
CSUs.

**Computer Software Unit (CSU):** The lowest level in a breakdown of a
software product.

**COnstructive COst MOdel (COCOMO):** A software cost estimation model
developed by Barry Boehm and is described in his book, Software
Engineering Economics.

**Cost Analysis:** The review and evaluation of the separate cost
elements and proposed profit of (a) an offeror's or contractor's cost or
pricing data and (b) the judgmental factors applied in projecting from
the data to the estimated costs in order to form an opinion on the
degree to which the proposed costs represent what the cost of the
contract should be, assuming reasonable economy and efficiency.

**Cost Driver Attributes:** Productivity factors in the software product
development process that include software product attributes, computer
attributes, personnel attributes, and project attributes.

**Cost Drivers:** The controllable system design or planning
characteristics that have a predominant effect on the system\'s costs.
Those few items, using Pareto\'s law, that have the most significant
cost impact.

**Cost Estimating Relationship:** An algorithm relating the cost of an
element to physical or functional characteristics of that cost element
or a separate cost element; or relating the cost of one cost element to
the cost of another element.

**Cost Estimating Relationships (CER):** A mathematical expression that
describes, for predicative purposes, the cost of an item or activity as
a function of one or more independent variables.

**Cost Model:** An estimating tool consisting of one or more cost
estimating relationships, estimating methodologies, or estimating
techniques used to predict the cost of a system or one of its lower
level elements.

**Cost or Pricing Data:** All facts that, at the time of the price
agreement, the seller and buyer would reasonably expect to affect price
negotiations. Cost or pricing data requires certification. Cost or
pricing data are factual, not judgmental data, and are therefore
verifiable. While these data do not indicate the accuracy of the
prospective contractor's judgment about estimated future costs or
projections, they do include the data utilized to form the basis for
that judgment. Cost or pricing data are more than historical accounting
data; they are all the facts that can be reasonably expected to
contribute to the soundness of estimates of future costs and to the
validity of determinations of costs already incurred.

**Cost/Schedule Control System Criteria (C/SCSC):** A set of criteria
specified by the Federal Government for reporting project schedule and
financial information.

**Delivered Source Instructions (DSI):** The number of source lines of
code developed by the project. The number of DSIs is the primary input
to many software cost estimating tools. The term [delivered]{.underline}
is generally meant to exclude non-delivered support software such as
test drivers. The term [source instructions]{.underline} includes all
program instructions created by project personnel and processed into
machine code by some combination of preprocessors, compilers, and
assemblers. It excludes comments and unmodified utility software. It
includes job control language, format statements, and data declarations.

**Delphi Technique:** A group forecasting technique, generally used for
future events such as technological developments, that uses estimates
from experts and feedback summaries of these estimates for additional
estimates by these experts until a reasonable consensus occurs. It has
been used in various software cost-estimating activities, including
estimation of factors influencing software costs.

**Detail Estimating: Grass Roots, Bottoms-Up:** The logical buildup of
estimated hours and material by use of blue-prints, production planning
tickets, or other data whereby each operation is assigned a time value.

**DoD-STD-2167A/498:** A US Department of Defense standard that
specifies the overall process for the development and documentation of
mission-critical software systems. DoD-STD-2167A/498 has direct
connections to five other standards that are not software specific.
These five standards are as follows:

+-----------------------------------------------------------------------+
| -   **MIL-STD-1521:** Technical Reviews and Audit for Systems,        |
|     Equipment, and Computer Software                                  |
+-----------------------------------------------------------------------+
| -   **MIL-STD-480:** Configuration Control**:** Engineering Changes,  |
|     Deviations, and Waivers                                           |
+-----------------------------------------------------------------------+
| -   **MIL-STD-481:** Configuration Control**:** Engineering Changes,  |
|     Deviations, and Waivers (Short Form)                              |
+-----------------------------------------------------------------------+
| -   **MIL-STD-490:** Specification Practices                          |
+-----------------------------------------------------------------------+
| -   **MIL-STD-499:** Engineering Management                           |
+-----------------------------------------------------------------------+

MIL-STD-480 and 481 have been superseded by MIL-STD-973. MIL-STD-973
consolidates several earlier standards for configuration management and
resolves inconsistencies and ambiguities that existed between them. When
DoD-STD-2167A/498 is applied to contract, appropriate tailoring
instructions should be included to indicate that the contract is to
comply with the more recent MIL-STD-973.

**Domain:** A specific phase or area of the software life cycle in which
a developer works. Domains define developers and users areas of
responsibility and the scope of possible relationships between products.
The work can be organized by domains such as Software Engineering
Environments, Documentation, Project Management etc.

**Evolutionary Acquisition (EA) Model:** The EA Model is pretty similar
in content to the Waterfall Model except it encourages prototyping. The
underlying factor in EA is to field a well-defined core capability
quickly in response to a validated requirement, while using a phased
upgrade program to eventually enhance the system to provide the full
system capability. This process is also referred to as evolutionary
prototyping.

**Expert Judgment Models:** use a method of software estimation that is
based on consultation with one or more experts that have experience with
similar projects. An expert-consensus mechanism such as the [Delphi
technique]{.underline} may be used to produce the estimate.

**Fair Price (See Also Reasonable Price):** From the perspective of a
buyer, a fair price is a price that is in line with (or below) the fair
market value of the contract deliverable (to the extent that fair market
value can be approximated through price analysis). \"Fair market value\"
is the price you should expect to pay, given the prices of bona fide
sales between informed buyers and informed sellers under like market
conditions in competitive markets for deliverables of like type,
quality, and quantity. When data on probable performance costs are
available, a separate test of \"fairness\" is whether the proposed price
is in line with (or below) the total allowable cost of providing the
contract deliverable. This cost would be the cost incurred by a well
managed, responsible firm using reasonably efficient and economical
methods of performance, plus a reasonable profit. From the perspective
of a seller, a fair price is a price that is realistic in terms of the
seller\'s ability to satisfy the terms and conditions of the contract.

**Fourth Generation Language (4GL):** Fourth generation languages are
programming languages closer to human languages than typical high level
programming languages. Most 4GLs are used to access databases. For
example a typical 4GL command is "FIND ALL RECORDS WHERE NAME IS
"SMITH"."

**Function Points:** Function Points are those pieces of code that
perform some specific activity related to inputs, inquiries, outputs,
master files, and external system interfaces.

**Historical Data:** A term used to describe a set of data reflecting
actual cost or past experience of a product or process.

**Incremental Development:** The incremental development approach is a
top down implementation of distinct functional elements of the product.
The development of each increment is accomplished as a separate
waterfall type of development. The incremental development methodology
differs from the evolutionary approach in that under the incremental
strategy the end product is well-defined.

**Knowledge Base:** The repository of knowledge in a computer system or
organization. The collection of data, rules, and processes that are used
to control a system, especially one using artificial intelligence or
expert system methods.

**Life Cycle:** The stages and process through which hardware or
software passes during its development and operational use. The useful
life of a system. Its length depends on the nature and volatility of the
business, as well as the software development tools used to generate the
databases and applications.

**Management Information Systems:** A computer-based system of
processing and organizing information that provides different levels of
management within an organization with accurate and timely information
needed for supervising activities, tracking progress, making decisions,
and isolating and solving problems.

**Metric:** Quantitative analysis values calculated according to a
precise definition and used to establish comparative aspects of
development progress, quality assessment or choice of options.

**New Line of Code:** A source line of code that will be developed
completely, i.e., designed, coded and tested.

**Paradigm:** A model, example, or pattern. A generally accepted way of
thinking.

**Parametric Cost Model:** A mathematical representation of parametric
cost estimating relationships that provides a logical and predictable
correlation between the physical or functional characteristics of a
system, and the resultant cost of the system. A parametric cost model is
an estimating system comprising cost estimating relationships (CERs) and
other parametric estimating functions, e.g., cost quantity
relationships, inflation factors, staff skills, schedules etc.
Parametric cost models yield product or service costs at designated
levels and may provide departmentalized breakdown of generic cost
elements. A parametric cost model provides a logical and repeatable
relationship between input variables and resultant costs.

**Platform:** Hardware or software architecture of a particular model or
family of computers. The term sometimes refers to the hardware and its
operating system.

**Price Analysis:** The process of examining and evaluating a proposed
price without evaluating its separate cost elements and proposed profit.

**Procedures:** Manual procedures are human tasks. Machine procedures
are lists of routines or programs to be executed, such as described by
the Job Control Language (JCL) in a mini or mainframe, or the batch
processing language in a personal computer.

**Process:** The sequence of activities (in software development)
described in terms of the user roles, user tasks, rules, events, work
products, resource use, and the relationships between them. It may
include the specific design methodology, language, documentation
standards etc.

**Rapid Prototyping:** The creation of a working model of a software
module to demonstrate the feasibility of the function. The prototype is
later refined for inclusion in a final product.

**Rayleigh Distribution:** A curve that yields a good approximation to
the actual labor curves on software projects.

**Real-Time:** 1) Immediate response. The term may refer to fast
transaction processing systems in business; however, it is normally used
to refer to process control applications. For example, in avionics and
space flight, real-time computers must respond instantly to signals sent
to them. 2) Any electronic operation that is performed in the same time
frame as its real-world counterpart. For example, it takes a fast
computer to simulate complex, solid models moving on screen at the same
rate they move in the real world. Real-time video transmission produces
a live broadcast.

**Reasonable Price (See Also Fair Price):** A price that a prudent and
competent buyer would be willing to pay for the contract deliverable,
given adequate data on (1) market conditions, (2) alternatives for
meeting the requirement, (3) the evaluated price of each alternative,
and (4) non-price evaluation factors (in \"best value\" competitions).

**Re-engineering:** Process of restructuring and redesigning an
operational (or coded) hardware or software system or process in order
to make it meet certain style, structure, or performance standards.

**Reusability:** Ability to use all or the greater part of the same
programming code or system design in another application.

**Reuse:** Software development technique that allows the design and
construction of reusable modules, objects, or units, that are stored in
a library or database for future use in new applications. Reuse can be
applied to any methodology in the construction phase, but is most
effective when object oriented design methodologies are used.

**Security:** The protection from accidental or malicious access, use,
modification, destruction, or disclosure. There are two aspects to
security, confidentiality and integrity.

**Software Development Life Cycle:** The stages and process through
which software passes during its development. This includes requirements
definition, analysis, design, coding, testing, and maintenance.

**Software Development Life Cycle Methodology:** Application of methods,
rules, and postulates to the software development process to establish
completeness criteria, assure an efficient process, and develop a high
quality product.

**Software Engineering Institute (SEI):** SEI is a federally funded
research and development center established in 1984 by the DoD with a
broad charter to address the transition of software engineering
technology. The SEI is an integral component of Carnegie Mellon
University and is sponsored by the Office of the Under Secretary of
Defense for Acquisition and Technology. SEI developed the *Software
Acquisition Capability Maturity Model* (CMM) and the *Checklist and
Criteria for Evaluating the Cost and Schedule Estimating Capabilities of
Software Organizations*.

**Software Method (or Software Methodology):** Focuses on how to
navigate through each phase of the software process model (determining
data, control, or uses hierarchies; partitioning functions; and
allocating requirements) and how to represent phase products (structure
charts; stimulus-response threads; and state transition diagrams).

**Software Tool:** Program that aids in the development of other
software programs. It may assist the programmer in the design, code,
compile, link, edit, or debug phases.

**Source Lines of Code (SLOC):** All executable source code statements
including deliverable Job Control Language (JCL) Statements, Data
declarations, Data Typing statements, Equivalence statements, and
Input/Output format statements. SLOC does not include any statement that
upon its removal, the program will still compile, e.g., comments, blank
lines, and non-delivered programmer debug statements.

**Space Systems Cost Analysis Group:** The SSCAG is an organization
co-chaired by the cost directorates of the Air Force Space and Missile
Systems Center (SMC) and NASA at Johnson Space Center. The Software
Subgroup of the SSCAG, under a multi-year task, developed the Software
Methodology Handbook and a software database of over 2600 records used
to calibrate five software estimating models (PRICE-S, SEER-SEM, SLIM,
SASET AND REVIC).

**Spiral Development:** The spiral model encompasses features of the
phased life cycle as well as the prototype life cycle. However, unlike
those life cycles, the spiral model uses risk analysis as one of its
elements. It also uses the waterfall model for each step so as to avoid
any risks.

**Top-Down Models:** Use a method of estimation that estimates the
overall cost and effort of the proposed project derived from global
properties of the project. The total cost and schedule is partitioned
into components for planning purposes.

**Update:** To update an estimate or CER means to utilize the most
recent data to make it current, accurate and complete.

**Validation:** In terms of a cost model, a process used to determine
whether the model selected for a particular estimate is a reliable
predictor of costs for the type of system being estimated.

**Waterfall Model:** An eight-phase process used in developing software
for most Department of Defense (DoD) weapon systems, as described in
DoD-Standard 2167A. This process, when done sequentially, is based on
the waterfall model of software development, as described by Barry
Boehm. Each phase requires the delivery of particular documentation
products.

**Work Breakdown Structure:** A work breakdown structure is a
product-oriented family tree, composed of hardware, software, services,
data and facilities which results from system engineering efforts during
the development and production of a defense material item, and which
completely defines the program. A work breakdown structure displays and
defines the product(s) to be developed or produced and relates the
elements of work to be accomplished to each other and to the end
product. MIL-HDBK 881 is the modern guide for developing a WBS. (See
Appendix 19C contains a sample WBS.)

**Workstation:** High-performance, single user microcomputer or
minicomputer that has been specialized for graphics, CAD, CAE, or
scientific application.

**SAMPLE SOFTWARE WORK BREAKDOWN STRUCTURE**

The following sample Work Breakdown Structure is based on the sample
Software WBS contained in NRaD's *Software Size,* *Cost and Schedule
Estimation Process,* modified for this handbook*.* It is oriented to the
development of a MIL-STD-498 or DoD-STD-2167A software product. The
sample WBS is very detailed and is meant to be tailored to each
project\'s specific tracking needs and requirements.

I.  PROJECT MANAGEMENT

    A.  Project Management Plan

        1.  Define task execution plans

        2.  Define project schedule/milestones

        3.  Define prelim SW functional requirements

        4.  Develop project WBS

        5.  Define SW management metrics process

        6.  Define/allocate project functions

            a.  Hardware engineering

            b.  Software engineering

            c.  Configuration Management

            d.  Software Quality Assurance

            e.  Inspect, Validate & Verify (IV&V)

            f.  Project support

            g.  Software metrics

            h.  Determine staffing

        7.  Determine SWE tool requirements

        8.  Determine training requirements

        9.  Plan activities/reviews

    B.  Management Reporting

II. SOFTWARE ESTIMATION

    A.  Preliminary project estimate

        1.  Establish cost estimation methodology/tools

        2.  ID similar functions/historical projects

        3.  Identify project objectives and requirements

        4.  Plan the activities/initial WBS

        5.  Estimate product size

        6.  Estimate cost and effort

        7.  Estimate schedule

        8.  Identify potential risk areas

        9.  Set up cost estimate tracking metrics

        10. Review/refine with project personnel

    B.  System Requirements Phase Estimate

        1.  Establish software estimate file

        2.  Develop top-level SW requirements WBS

        3.  Develop baseline software estimates

        4.  Formal inspection/approval of estimate

        5.  Refine/record/track estimates

    C.  Software Requirements Phase Estimate Update

        1.  Update top-level design WBS

        2.  Update baseline requirements phase estimates

        3.  Conduct formal inspection of estimate

        4.  Refine/record/track estimates periodically

    D.  Design Phase Estimate Update

        1.  Develop detailed CUT WBS

        2.  Update baseline CUT estimates

        3.  Formal inspection of estimate

        4.  Refine/record/track estimates periodically

    E.  Code and Unit Test Phase Estimate Update

        1.  Update detailed WBS

        2.  Update/revise estimates

        3.  Update/revise risk assessment

        4.  Update preliminary maintenance estimate

        5.  Refine/record/track estimates periodically

        6.  Formal inspection of estimate

    F.  Integration Phase Estimate Update

        1.  Update detailed WBS

        2.  Update/revise estimates

        3.  Update/revise risk assessment

        4.  Update/revise maintenance estimate

        5.  Refine/record/track estimates periodically

        6.  Formal inspection of estimate

        7.  Post-Deployment Phase Estimate

        8.  Close out/report final cost/schedule/size

III. RISK MANAGEMENT

     A.  Preliminary Risk Management Plan

         1.  Identify potential risk areas

         2.  Analyze risk areas

         3.  Prioritize risks

         4.  Identify risk tracking metrics

         5.  Write Plan

     B.  Risk Management Plan review/inspection

         1.  Update Risk Management Plan

         2.  Baseline Risk Management Plan

         3.  Update/revise risk assessment

IV. SOFTWARE METRICS

    A.  Software Metrics plan

        1.  Process Effectiveness Metrics

        2.  Process Cost Metrics

        3.  Cost/Schedule Tracking Metrics

        4.  Performance Measurement Indicators

        5.  Software Quality Metrics

    B.  Update/revise Metrics Plan

    C.  Track/analyze cost/schedule/size estimate variances

    D.  Track/analyze cost estimating effort and tool costs

    E.  Track/analyze cost/schedule variance

    F.  Track/analyze requirements and design progress

    G.  Track/analyze code and test progress

    H.  Track/analyze person-months effort

    I.  Track/analyze program size

    J.  Track/analyze computer resources utilization

    K.  Track/analyze requirements stability

    L.  Track/analyze design stability

    M.  Track/analyze complexity metrics

    N.  Track/analyze modules metrics

    O.  Track/analyze testing metrics

    P.  Track/analyze defects metrics

    Q.  Track/analyze product and operational metrics

    R.  Formal reports

    S.  Report final project analysis

V.  SYSTEM ENGINEERING

    A.  Analyze operational requirements

    B.  Hardware/software trade-off studies

    C.  Sizing/timing studies

    D.  Define computer resource constraints

        1.  Memory

        2.  Throughput

        3.  I/O channel utilization

        4.  I/O throughput

        5.  Storage

        6.  Database overhead

    E.  Identify risks

    F.  Preliminary System Segment Specification (SSS)

        1.  Define system states

        2.  Define modes of operation

        3.  Define system capability

        4.  Define external inter. requirements

        5.  Write external interface descriptions

        6.  Define physical characteristics

        7.  Define system quality requirements

            a.  Reliability

            b.  Maintainability

            c.  Availability

            d.  Environmental conditions

            e.  Transportability

            f.  Flexibility and expansion

        8.  Specify design/construct requirements

        9.  Define system workmanship requirements

        10. Define system interchange requirements

        11. Define system safety requirements

        12. Define system security requirements

        13. Determine Government Furnished Equipment

        14. Define computer resource reserve capacity

        15. Define system logistics

        16. Specify personnel requirements

        17. Define training requirements

        18. Determine for each System/Segment

            a.  Purpose

            b.  Description

            c.  Identify capabilities

            d.  Identify requirements precedence

            e.  Validate and Verify (V&V) requirements

            f.  QA provisions

            g.  Inspection responsibility

            h.  Special Tests and exams

            i.  Requirements cross reference

    G.  Preliminary Interface Requirement Specification (IRS)

        1.  Identify HWCIs/CSCIs interfaces

        2.  Draw interface diagram(s)

        3.  Identify for each interface

            a.  Timing

            b.  Protocol

            c.  Priority

            d.  Unit of measure

            e.  Limits/ranges

            f.  Data elements

            g.  Precision

    H.  SSS/IRS inspection(s)/review(s)

    I.  System Requirements Review (SRR)

    J.  Update SSS

    K.  Baseline SSS

    L.  Update Preliminary IRS

VI. CONFIGURATION MANAGEMENT (CM)

    A.  Preliminary CM Plan

    B.  CM Plan review/inspection

    C.  Update CM Plan

    D.  Baseline CM Plan

    E.  Set Up Configuration Control Board (CCB)

    F.  Conduct/maintain Configuration Status (CS) accounting

        1.  Prepare Configuration Status Report (CSR)

    G.  Conduct Functional Configuration Audit (FCA)

    H.  Conduct Physical Configuration Audit (PCA)

    I.  Process Engineering Change Proposals (ECPs)

    J.  Process System Change Notices (SCNs)

    K.  Generate inputs to the Software Version Document (SVD)

    L.  Prepare Version Control Reports

VII. SOFTWARE QUALITY ASSURANCE (SQA)

     A.  Preliminary SQA Plan

     B.  Plan SW quality metrics process

     C.  SQA Plan inspection(s)/review(s)

     D.  Update SQA Plan

     E.  Baseline SQA Plan

VIII. SOFTWARE DEVELOPMENT PLAN (SDP)

      A.  Preliminary SDP

          1.  Define formal review procedures/criteria

          2.  Define Software Development Library procedures

          3.  Define corrective action process

          4.  Define Problem/Change Report Format

          5.  Define design standards

          6.  Define coding standards

          7.  Define testing approach

          8.  Define requirements traceability process

      B.  SDP inspection(s)/review(s)

      C.  Update SDP

      D.  Baseline SDP

IX. INTERFACE REQUIREMENTS

    A.  Analyze Preliminary IRS

    B.  Interface Design Document

        1.  Interface 1 to N define

        2.  Data elements

        3.  Message descriptions

        4.  Priority

        5.  Communications protocol

        6.  Interface inspection(s)/walk-through(s)

        7.  Interface rework

    C.  Interface Design Document (IDD) inspection(s)

    D.  IDD rework

    E.  Participate in Preliminary Design Review (PDR)

    F.  Update Interface Design Document

    G.  Baseline Interface Design Document

X.  DATABASE REQUIREMENTS

    A.  Preliminary Database Design Document (DDD)

        1.  Analyze database requirements

        2.  Identify data requirements

        3.  Perform database studies

    B.  Database Design Document inspection(s)

    C.  Participate in PDR

    D.  Update Database Design Document

    E.  Baseline Database Design Document

XI. DEVELOPMENTAL SOFTWARE

    A.  Analyze system requirements

    B.  Identify software requirements

    C.  Determine derived software requirements

    D.  Identify candidate COTS software

    E.  Identify candidate reusable software

    F.  Perform feasibility studies

    G.  Select computer language(s)

    H.  Allocate functions/identify CSCIs

    I.  Determine software requirement testability

    J.  CSCI 1 to N

        1.  FUNCTIONAL REQUIREMENTS

            a.  Analyze CSCI requirements

            b.  Preliminary Software Requirements Specification

                1.  Identify internal interfaces

                2.  Identify functional/derived requirements

                    a.  engineering

                    b.  data elements

                    c.  safety

                    d.  security

                    e.  human engineering

                3.  Identify software quality factors

                4.  Identify design constraints

                5.  Identify qualification methods

                6.  Trace requirements to SSS

            c.  Software Requirements Specification (SRS)
                inspection(s)/review(s)

            d.  Software Specification Review (SSR)

            e.  Update SRS

            f.  Baseline SRS

        2.  PRELIMINARY DESIGN

            a.  Preliminary Design analysis

            b.  Identify CSUs

            c.  Identify internal interfaces

            d.  Identify external interfaces

            e.  Preliminary Software Design Documents (SDDs)

                1.  Overview

                2.  Architecture

                3.  Memory/processing time allocation

                4.  CSCI design description

                    a.  CSU 1 - N

                        i.  Identify allocated requirements

                        ii. Identify CSUs

                        iii. Identify relationships between CSUs

                        iv. Data flow and execution control

                        v.  Identify derived requirements

                        vi. Trace requirements to SRS

                        vii. CSU inspection(s)/walk-through(s)

                        viii. CSU design rework

                5.  Preliminary SDD inspection(s)/review(s)

                6.  Preliminary Design Review (PDR)

                7.  Update Preliminary SDD

        3.  DETAILED DESIGN

            a.  Detailed SDD

                1.  CSU 1 - N

                    a.  Describe constraints

                    b.  Describe input/output data elements

                    c.  Describe local data elements

                    d.  Describe interrupts and signals

                    e.  Describe algorithms

                    f.  Describe data structures

                    g.  Describe local datafiles/database

                    h.  Describe limitations

                    i.  Trace requirements to Preliminary SDD

                    j.  CSU inspection(s)/walk-through(s)

                    k.  CSU design rework

                2.  SDD inspection(s)/review(s)

                3.  SDD rework

                4.  Critical Design Review (CDR)

                5.  Update SDD

                6.  Baseline SDD

        4.  CODE and UNIT TEST

            a.  CSU 1 - N

                1.  Design/document unit test

                2.  Code and compile

                3.  Write comments/header

                4.  Code inspection(s)/walk-through(s)

                5.  Rework

                6.  Testing and analysis

                7.  Rework

                8.  Maintain Software Development Folder (SDF)

                9.  Turn over accepted CSU to CM

        5.  TEST READINESS REVIEW

        6.  CSU INTEGRATION and TESTING

            a.  Analyze Software Test Report

            b.  Perform necessary rework

            c.  Perform CSU regression testing

            d.  Update SDFs

        7.  TEST READINESS REVIEW

        8.  CSCI INTEGRATION and TESTING

            a.  Analyze Software Test Report

            b.  Perform necessary rework

            c.  Perform CSU regression testing

            d.  Update SDFs

XII. SOFTWARE INTEGRATION & TESTING

     A.  Software Test Plan

         1.  Determine general test requirements

         2.  Determine test classes

             a.  stress

             b.  timing

             c.  erroneous input

             d.  maximum capacity

         3.  Determine test levels

             a.  CSCI

             b.  CSCI to CSCI integration

         4.  CSCI to HWCI integration

         5.  System

         6.  Determine test definitions

             a.  Test 1 - N

                 1.  determine objective

                 2.  determine special requirements

                 3.  identify test type/class

                 4.  determine qualification method

                 5.  cross reference to SRS requirements

                 6.  determine type of data to record

                 7.  identify assumptions/constraints

                 8.  determine test schedule

                 9.  identify data analysis techniques

     B.  Perform Integration & Testing

         1.  System

             a.  Integrate CSCIs

             b.  Write System Test Description

             c.  Conduct Test Readiness Review

             d.  Perform testing and analysis

             e.  Write System Test Report

             f.  Rework

             g.  Regression testing

             h.  CSCI

                 1.  Integrate CSUs

                 2.  Write Software Test Description

                 3.  Conduct Test Readiness Review

                 4.  Perform testing and analysis

                 5.  Write Software Test Report

                 6.  Rework

                 7.  Regression testing

                 8.  CSU

                     a.  Integrate CSUs

                     b.  Write Software Test Description

                     c.  Conduct Test Readiness Review

                     d.  Perform testing and analysis

                     e.  Write Software Test Report

                     f.  Testing and analysis

                     g.  Rework

                     h.  Regression testing

**19D. Popular Parametric Software Cost Models**

###### 19D.1 [Introduction]{.underline}

There are several sophisticated parametric software cost models that
consider multiple parameters in computing cost or effort required. A
list of some common models is provided below, listed in alphabetical
order. Subsequent sections are a basic discussion of each model. For
each of the models, general information, principal inputs, processing,
principal outputs, calibration, and life cycle (or support)
considerations will be discussed. When more detailed information is
needed, cost estimators are encouraged to consult the referenced
documents.

-   Automated Cost Estimating Integrated Tools (ACEIT)

-   COnstructive COst MOdel (COCOMO.II)

-   Cost Xpert

-   ForeSight

-   KnowledgePLAN Model

-   PRICE-S Model

-   Revised Enhanced Version of Intermediate COCOMO (REVIC)

-   SoftCost-Object-Oriented (OO) Model

-   Software Architecture, Sizing, and Estimating Tool (SASET)

-   Software ESTimator (SoftEST)

-   Software Life Cycle Model (SLIM)

-   System Evaluation and Estimation of Resources Software Estimating
    Model (SEER-SEM)

###### 19D.2 [Automated Cost Estimating Integrated Tools (ACEIT)]{.underline}

ACEIT is an estimating system consisting of a suite of tools designed to
assist cost analysts in arriving at cost estimates, conducting
\"what-if?\" studies, developing cost proposals and evaluations,
conducting risk and uncertainty analysis, and developing **Cost
Estimating Relationships (CERs)**. Its primary purpose is Financial
Management. ACEIT is a Joint Service system, sponsored by the **Air
Force Materiel Command (AFMC)** Electronic Systems Center and the US
Army Cost and Economic Analysis Center **(CEAC)**. The result of
government-sponsored efforts, the ACEIT suite of applications is
available to U.S. Government organizations with no charge for use (but
there is an annual maintenance & support fee). It is used by Acquisition
Program Offices and at various other levels across the Armed Services,
throughout DoD, and by other Government agencies.

The ACEIT system is comprised of five analysis/estimating tools:

-   Automated Cost Estimator (ACE), (the spreadsheet)

-   Automated Cost Database (ACDB), (library of commercial and
    non-commercial cost models)

-   Cost Analysis Statistics Package (CO\$TAT), (full-feature statistics
    package)

-   Cost Risk Assessment (RI\$K), (model which quantifies risk
    associated with a cost estimate)

-   ACE Executive.

ACE is the heart of ACEIT. It automates all of the steps of the
estimating process, including, building a **Work Breakdown Structure
(WBS)**, specifying estimating methods, performing learning, time
phasing and inflation, and documentation. ACE also provides access to
on-line databases and a knowledge base of over 1000 cost estimating
relationships, models, and source documents from which users can
identify appropriate estimating methods and incorporate them into their
estimate.

ACDB is an automated cost database with the capability to enter, search,
and retrieve cost, schedule, technical, and programmatic data and to
automatically load retrieved data into the **Cost Analysis Statistics
Package (CO\$TAT)**. CO\$TAT is a cost analysis statistical package
built specifically for the cost estimator to perform statistical
analyses commonly used in cost estimation. RI\$K is a cost risk
application that performs risk and uncertainty analysis. ACE Executive
allows estimates and models hosted in ACE to act as cost model servers
to other applications.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **ACEIT** |   | Company:  | ACEIT Gov't Prog     |
|                 |           |   |           | Director             |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 3.1       |   | Address:  | ESC/FMC              |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1997      |   |           | Hanscom AFB, MA      |
|                 |           |   |           | 01731                |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  |           |   | 800 Phone |                      |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 781-377-5120         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | **as of   |   | Fax       |                      |
| Information:**  | 1Q1998**  |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | Free w/   |   | Internet: | [www.aceit.com](ht   |
| Gov't:          | ACEIT     |   |           | tp://www.aceit.com/) |
|                 | maint     |   |           |                      |
|                 | contract  |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \         |   | Marketing | Telecote Research,   |
| Commercial:     | $2,500/yr |   | Contact:  | Inc.                 |
|                 |           |   |           |                      |
|                 |           |   |           | Jenny Nigra          |
+-----------------+-----------+---+-----------+----------------------+
| Annual          | \$1.5K    |   |           |                      |
| Maintenance     | \$2.5K/yr |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     | 805-964-6963         |
| Gov't:          |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    | \         |   | Fax       | 805-964-7329         |
| Commercial:     | $9,900/yr |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   | E-Mail:   | jn                   |
|                 |           |   |           | igra@sb.telecote.com |
+-----------------+-----------+---+-----------+----------------------+
| Evaluation Copy | Free for  |   |           |                      |
|                 | 90 days   |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                Win 95/NT         OS:            Win 3.1

  RAM:               16 MB             RAM:           8 MB

  HD Storage:        60 MB             HD Storage:    40 MB
  ------------------ ------------ ---- -------------- -----------------------

  -----------------------------------------------------------------------
  The ACEIT system is comprised of five analysis/estimating tools:
  Automated Cost Estimator (ACE), (the spreadsheet); Automated Cost
  Database (ACDB), (relational data base); Cost Analysis Statistics
  Package (CO\$TAT), (full-feature statistics package); Cost Risk
  Assessment (RI\$K), (quantifies risk associated with a cost estimate)
  and ACE Executive. ACE, the Automated Cost Estimator, is a special
  purpose program, specifically developed for cost analysis. It has been
  organized and structured to follow the steps or phases used in
  developing a cost estimate, ranging from defining what is being
  estimated through time-phasing and final documentation. ACDB is the
  cost database module that contains cost, schedule, technical, and
  programmatic data on system acquisition contracts. It is a PC-based
  program designed to assist the user in building, loading, maintaining,
  and querying a relational data base of cost and technical data, and in
  analyzing data subsets retrieved from it. The ACEIT Executive is a
  three-part application: the ACE Calculation Server, the RI\$K
  Calculation Server, and the Microsoft Excel Client. The Server
  applications let you run ACE and RI\$K sessions from any client
  application. Using the Excel client, you can link applications that
  would naturally interface with Excel to ACE and RI\$K. The user can
  either enter own CERs/equations or import the results of other models
  such as PRICE and SEER. Outputs look like spreadsheets or can link to
  Excel.

  -----------------------------------------------------------------------

####### 19D2.1 ACEIT Inputs 

ACEs provides a workscreen or spreadsheet to perform each step required
to make an estimate. The complete list of steps is as follows:

1.  Define the WBS/CES

2.  Enter the estimating methodologies

3.  Adjust G&A, fee, overhead, escalation, units

4.  Apply learning curves

5.  Time phase the estimate

6.  Estimate/perform what-ifs

7.  Document the estimate

8.  Review and Refine

The input screens resemble spreadsheets and require the equations and
variables to be defined. Within each workscreen, functionality is
provided in ACE by giving each column a dedicated purpose. For example,
the Learning Curve Slope column only lets you enter a learning curve
slope or a variable name that represents the slope. Each column has its
own syntax. By making entries in several columns for a row, you can
specify that rows estimating methodology. Typically, the rows in ACE
represent WBS/CES line items or input variable items.

ACE can import from PRICE Software Models, SEER Cost Models, MS Excel
and MS Word.

####### 19D2.2 ACEIT Processing

Since the user enters the equations and variables, the user basically
defines the processing. The ACE structure allows the user to
progressively move from one workscreen to another. The user creates the
model structure of the estimate, and calculates the result. This model
building can be free-form. The user does not need to move through the
workscreens in a pre-defined order. Unique IDs or variable names can be
specified for each row and then used in the ACE columns of other rows.
This lets rows of the model link together into a system of equations.

####### 19D2.3 ACEIT Outputs

Reports are basically spreadsheets containing the results. Tools that
ACE can export data to include MS Excel and MS Word.

####### 19D2.4 ACEIT Calibration

Since the user controls the equations, calibration is accomplished by
creating historical databases, experience and refining the equations.

####### 19D2.5 ACEIT Life Cycle Considerations

Include the WBS Lifecycle elements into the model.

###### 19D.3 [COnstructive COst MOdel (COCOMO.II)]{.underline}

COCOMO.II is a screen-oriented, interactive-software package that
assists in budgetary planning and schedule estimation of a software
development project prior to any work beginning. Through the flexibility
of COCOMO.II, a software project manager can develop a model (or
multiple models) of projects in order to identify potential problems in
resources, personnel, budgets, and schedules both before and after the
potential software package has been completed.

The COCOMO.II software package is based upon a recently revised version
of the original **COnstructive COst MOdel (COCOMO)** first published by
Barry Boehm in his book *Software Engineering Economics*, Prentice-Hall
(1981), and the Ada COCOMO (1987) predecessors. Table 19D-1 summarizes
the different versions of COCOMO.

Table 19D-1. COCOMO Model Versions

  ------------------------ ----------------------------------------------
  **Version**              **Description/Release Date**

  COCOMO 1.0               original version released 1981 for waterfall
                           model.

  Basic Level              Consists of 3 simple models with single
                           parameter effort & schedule equations.

  Intermediate Level       Contains nominal equations with 15 multipliers
                           that adjust the results to reflect the unique
                           attributes of the specific program.

  Detailed Level           Adjusts the Intermediate Level multipliers for
                           each phase of the SW development life cycle.

  Ada COCOMO               1987 update for Ada SW language development

  COCOMO.II.1997.0 (also   Released on March 7, 1997 to better estimate
  referred to as COCOMO    costs for new SW development processes.
  2.0)                     
  ------------------------ ----------------------------------------------

The primary objectives of the COCOMO.II.1997 effort were to:

-   To develop a software cost and schedule estimation model attuned to
    the life cycle practices of the 1990\'s and 2000\'s.

-   To develop software cost database and tool support capabilities for
    continuous model improvement.

-   To provide a quantitative analytic framework, and set of tools and
    techniques for evaluating the effects of software technology
    improvements on software life cycle costs and schedules.

The full COCOMO.II model includes three stages. Stage 1 supports
estimation of prototyping or applications composition efforts. Stage 2
supports estimation in the Early Design stage of a project, when less is
known about the project's cost drivers. Stage 3 supports estimation in
the Post-Architecture stage of a project. The current version of
COCOMO.II implements stage 3 formulas to estimate the effort, schedule,
and cost required to develop a software product. It also provides the
breakdown of effort and schedule into software life cycle phases and
activities from the original COCOMO manual. These are still reasonably
valid for waterfall model software projects, but need to be interpreted
for non-waterfall projects.

Equation 19D-1 below is the equation used by COCOMO.II to calculate the
estimated effort.

> Equation 19D-1. Effort Estimation Equation

$$\text{PM} = \prod_{i = 1}^{\text{17}}\left( \text{EM}_{i} \right) \cdot A \cdot \left\lbrack \left( 1 + \frac{\text{BRAK}}{\text{100}} \right) \cdot \text{Size} \right\rbrack^{\left( 1\text{.}\text{01} + 0\text{.}\text{01}\sum_{j = 1}^{5}\text{SF}_{j} \right)} + \left( \frac{\text{ASLOC} \cdot \left( \frac{\text{AT}}{\text{100}} \right)}{\text{ATPROD}} \right)$$

Where:

$$$\text{Size} = \text{KNSLOC} + \left\lbrack \text{KASLOC} \cdot \left( \frac{\text{100} - \text{AT}}{\text{100}} \right) \cdot \frac{\text{AA} + \text{SU} + 0\text{.}4 \cdot \text{DM} + 0\text{.}3 \cdot \text{CM} + 0\text{.}3 \cdot \text{IM}}{\text{100}} \right\rbrack$

$$\left( B = 1\text{.}\text{01} + 0\text{.}\text{01}\sum_{j = 1}^{5}\text{SF} \right)$$

  --------------------------------------------------------------------------
  **Symbol**   **Description**
  ------------ -------------------------------------------------------------
  A            Constant, provisionally set to 2.5

  AA           Assessment and assimilation

  ADAPT        Percentage of components adapted (represents the effort
               required in under-standing software)

  AT           Percentage of components that are automatically translated

  ATPROD       Automatic translation productivity

  BRAK         Breakage: Percentage of code thrown away due to requirements
               volatility

  CM           Percentage of code modified

  DM           Percentage of design modified

  EM           Effort multipliers: RELY, DATA, CPLX, RUSE, DOCU, TIME, STOR,
               PVOL, ACAP, PCAP, PCON, AEXP, PEXP, LTEX, TOOL, SITE

  IM           Percentage of integration and test modified

  KASLOC       Size of the adapted component expressed in thousands of
               adapted source lines of code

  KNSLOC       Size of component expressed in thousands of new source lines
               of code

  PM           Person Months of estimated effort

  SF           Scale Factors: PREC, FLEX, RESL, TEAM, PMAT

  SU           Software understanding (zero if DM = 0 and CM = 0)
  --------------------------------------------------------------------------

+-----------------+-----------+---+-----------+----------------------+
| **Product       |           |   | **Vendor  |                      |
| Information:**  |           |   | P         |                      |
|                 |           |   | rofile:** |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **CO      |   | Company:  | Developed by USC-CSE |
|                 | COMO.II** |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 2.0       |   | Address:  | Univ of So.          |
|                 |           |   |           | California           |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1997      |   |           | Irvine, CA           |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  | 1981      |   | 800 Phone |                      |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 213-740-6470         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | **as of   |   | Fax       |                      |
| Information:**  | 1Q1998**  |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | Free      |   | Internet: | Http:                |
| Gov't:          | download  |   |           | //sunset.usc.edu/COC |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | from      |   |           | OMOII/cocomo.html    |
| Commercial:     | COCOMO.II |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          | home page |   | Marketing |                      |
| Maintenance     |           |   | Contact:  |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     |                      |
| Gov't:          |           |   | Number    |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Fax       |                      |
| Commercial:     |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   | E-Mail:   | Cocomo-info@         |
|                 |           |   |           |                      |
|                 |           |   |           | Sunset.usc.edu       |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                Solaris 2.4,      OS:            Solaris 2.4, SunOS 4.x
                     SunOS 4.x                        w/Motif or Win 3.1
                     w/Motif or                       
                     Win 95/NT                        

  RAM:               4 MB              RAM:           4 MB

  HD Storage:        5 MB              HD Storage:    5 MB
  ------------------ ------------ ---- -------------- -----------------------

+-----------------------------------------------------------------------+
| **Product Description:**                                              |
|                                                                       |
| COCOMO.II model is an updated version of the 1981 COCOMO and 1987 Ada |
| COCOMO models tailored to the new software development life cycle     |
| processes and capabilities. Major new modeling capabilities of        |
| COCOMO.II include: a tailorable family of software sizing models,     |
| involving Object Points, Function Points, and Source Lines of Code    |
| (SLOC); non-linear models for software reuse and reengineering; an    |
| exponent-driver approach for modeling relative software diseconomies  |
| of scale; and several changes to previous COCOMO effort-multiplier    |
| cost drivers.                                                         |
+-----------------------------------------------------------------------+

####### 19D.3.1 COCOMO.II Inputs

The primary COCOMO.II input are the program size, in KDSI, Function
Points or Object Points. However, ratings for sixteen additional
attributes must be assessed. These attributes are included in four
categories as follows:

-   **Product attributes:** These attributes describe the environment in
    which the program operates. The attributes in this category are:
    reliability requirements, database size, documentation matched to
    life cycle needs, required reusability and program complexity.

-   **Platform attributes:** These attributes refer to the limitations
    placed upon the development effort by the hardware and operating
    system being used to run the project. The attributes in this
    category are execution time constraints, main storage constraints,
    and platform volatility.

-   **Personnel attributes:** These attributes describe the skill levels
    of personnel assigned to the program. The attributes in this
    category include: analyst capability, applications experience,
    programmer capability, programming language experience, platform
    experience and personnel continuity.

-   **Project attributes:** These attributes refer to the constraints
    and conditions under which project development takes place. The
    attributes in this category are use of software development tools
    and multi-site development.

These 16 factors (or **effort multipliers (EM)**) are incorporated into
the schedule and effort estimation formulas by multiplying them
together. The numerical value of the i^th^ adjustment factor is called
EM~i~ and their product is called the adjustment factor or EAF. The
actual effort, PM~total~, is the product of the nominal effort times the
EAF.

####### 19D.3.2 COCOMO.II Processing

Using COCOMO.II, a nominal assessment of man-months based on size alone
is assessed for the program being considered. Next, the ratings for all
attributes are multiplied to compute the required man-months of effort
for the project. The primary challenges in using COCOMO.II are
determining size and assigning proper ratings to the sixteen attributes.

####### 19D.3.3 COCOMO.II Outputs

The output of the COCOMO.II model is simply the level of effort in
man-months for the project being estimated and a schedule in months. The
effort output can easily be converted to a monetary value if the cost
per man-month is known. The Phase Distribution is one of the outputs.
Its function is to display a breakdown of the software effort and
schedule into the phases of the development cycle. These phases are
plans & requirements, design, programming and integration & test. The
outputs of the model are very basic and not very flexible, so
performance metrics will need to be created outside of this model.

####### 19D.3.4 COCOMO.II Calibration

Calibration is essential to the proper use of software cost models. The
user of the COCOMO.II may calibrate EAFs and the effort/schedule
equations of the current project. (Detailed procedures for COCOMO
calibration are discussed in the COCOMO Reference Manual.)

####### 19D.3.5 COCOMO.II Life Cycle Considerations

There are no life cycle considerations included in the COCOMO.II model.
The COCOMO Maintenance model (COCOMO-M), described in Chapter 30 of
Boehm\'s book, can be used to estimate annual man-months required to
support a software program. No mention of COCOMO-M is contained in the
COCOMO.II literature. For the intermediate level COCOMO-M, all inputs
from COCOMO 1.0 are used, except that different numerical values are
assigned to two attributes: reliability and use of modern design
practices. The annual support costs can be computed by multiplying the
number of man-months by the average cost of a man-month.

###### 19D.4 [Cost Xpert]{.underline}

Marotz, Inc. developed a new model called Cost Xpert. The model is
designed to provide software project cost estimates, to determine
optimal delivery time, to break the project down into tasks by
allocating time/effort and to perform quantitative risk/sensitivity
analysis. It supports projects using object- oriented development, GUI
development and formalized software reuse.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **Cost    |   | Company:  | Marotz, Inc.         |
|                 | Xpert**   |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 1.0       |   | Address:  | 13518 Jamul Dr       |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1997      |   |           | Jamul, CA 91935      |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  | 1997      |   | 800 Phone | 800-477-6168         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 619-669-3100         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | **as of   |   | Fax       | 619-669-6914         |
| Information:**  | 1Q1998**  |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | .         |   | Internet: | [www.marotz.com](htt |
| Gov't:          |           |   |           | p://www.marotz.com/) |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \$995     |   |           |                      |
| Commercial:     |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          | 25%       |   | Marketing |                      |
| Maintenance     |           |   | Contact:  |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     |                      |
| Gov't:          |           |   | Number    |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    | 10 user,  |   | Fax       |                      |
| Commercial:     | \$7K      |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   | E-Mail:   | [                    |
|                 |           |   |           | Info@marotz.com](mai |
|                 |           |   |           | lto:info@marotz.com) |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                Win 95/NT         OS:            Win 3.1

  RAM:               4 Mb              RAM:           4 Mb

  HD Storage:        4 Mb              HD Storage:    4 Mb
  ------------------ ------------ ---- -------------- -----------------------

+-----------------------------------------------------------------------+
| **Product Description:**                                              |
|                                                                       |
| Cost Xpert is an automated tool that supports software costing,       |
| scheduling and risk assessment using SLOC, function points, GUI       |
| Metrics, top down and bottoms up methodologies. Cost Xpert consists   |
| of five main notebook tabs: Project, Volume, Environment, Constraints |
| and Results. Outputs include an input data summary, software          |
| development cycle task report, risk report, labor report by labor     |
| category, maintenance report, and documentation deliverables report.  |
+-----------------------------------------------------------------------+

####### 19D.4.1 Cost Xpert Inputs 

The most important input parameter is size, which can be input as SLOC
or function points, GUI Metrics, top down and bottoms up methodologies.
All size inputs are converted to equivalent SLOC. When more than one
size estimating methodology is used, the user can select which results
to be averaged together to create the cost and schedule estimate. Cost
Xpert has about thirty inputs which are included in the three other
"notebooks tabs". A help screen for each input assists the user in
selecting proper values. Each of the tabs is now summarized.

-   **Project Tab:** Inputs in this tab include primary and secondary
    programming language; project coefficients (commercial, military,
    embedded etc.); project standards (commercial, DoD-STD-2167A etc.)
    identifies documentation required; project type (commercial,
    embedded etc.) used to identify likely risk factors/defect rates;
    and project life cycle (client-server, standard-small etc.) used to
    relate activities to effort. These inputs can all be tailored.

-   **Environment Tab:** The user inputs the factors that influence the
    efficiency of the software development team such as analyst and
    programmer capability; applications, virtual machine and language
    experience; execution time and main storage constraints; virtual
    machine and requirements volatility; computer turn around time;
    database size; product complexity and reliability; required reuse
    and security; and use of modern programming practices and tools.
    Each of these can be in five levels rated from very low to very high
    with the definitions and numerical values included on the tab view.

-   **Constraints Tab:** The user can use the constraints tab to assign
    numerical values to eight constraint areas relevant to the project.
    These areas are time-cost trade-off, review time, requirements
    analysis, minimum review time, Beta testing, cushion, overlap and
    risk tolerance. The estimated percentages can be assigned by the
    user or left as a default value.

####### 19D.4.2 Cost Xpert Processing

The model performs all computations based on "equivalent SLOC". The core
costing equations involve a relatively simple equation of the form
"Effort = α \* volume β." α and β are organization specific coefficients
and can be adjusted by the user during calibration. The exact equations
used in the model are not defined in the *User's Guide*. The
environmental factors used in Cost Xpert are very similar to those used
in the REVIC model.

####### 19D.4.3 Cost Xpert Outputs

The model's primary outputs are development effort and schedule,
including the optimal effort and schedule for the program(s) being
analyzed. Cost Xpert provides staffing profiles by labor category/month.
Cost Xpert also has a risk assessment with which the user can perform
numerous size, effort, and schedule risk analyses, as well as
identifying likely risks to the project and a sensitivity analysis.
Maintenance and document deliverable reports summarize those areas. The
schedule report is exportable to MS Project. There is a risk assessment
tool available (Risk Xpert) that supports both risk assessment and
contingency planning/tracking to mitigate risks in a formal, optimized
manner.

####### 19D.4.4 Cost Xpert Calibration

Cost Xpert contains a minimum amount of detail for calibrating the
model. Basically, it is done by comparing actual historical data to
model predicted results, then adjusting the coefficients α and β.
Further calibration is made by tailoring the inputs in the Project and
Constraints tabs.

####### 19D.4.5 Cost Xpert Life Cycle Considerations

The model predicts both the maintenance effort for each year of the
project and the projected maintenance adjusted for inflation. The
quantity of software defect estimates and support calls are also
estimated based on the original project input data.

###### 19D.5 [ForeSight Model]{.underline}

ForeSight is a parametric tool for forecasting time, effort, and cost of
non-military software projects; the projects can be new developments,
modifications, integration of off-the-shelf applications, maintenance
programs, or software upgrades of any type. ForeSight is a tool for all
project management tasks related to effort and time
forecasting\--budgeting, staffing plans, estimate to complete,
performance measurement etc. This model has a one button OLE interface
to MS Project.

ForeSight is a contemporary adaptation of the PRICE S model to address
non-military software projects only. It was based upon the same database
used to construct PRICE S. ForeSight, however, was tuned to the
commercial and non-military government (AIS) records of the PRICE S
database and has some different equations. In addition, several
databases unique to ForeSight development were incorporated, among them:
the **National Software Data Information Repository (NSDIR)**, the
**ISBSG (International Software Benchmarking Standards Group)**
database, and a proprietary database of **Object Oriented (OO)**
applications.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **Fo      |   | Company:  | Lockheed Martin      |
|                 | reSight** |   |           | PRICE Systems        |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 1.0       |   | Address:  | 700 E. Gate Drive,   |
|                 |           |   |           | Suite 200            |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1997      |   |           | Mount Laurel, NJ     |
|                 |           |   |           | 08054                |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  | 1997      |   | 800 Phone | 800-437-7423         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 609-866-6789         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | **as of   |   | Fax       | 609-866-6591         |
| Information:**  | 1Q1998**  |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \$975     |   | Internet: | www.pricesystems.com |
| Gov't:          |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     |           |   |           |                      |
| Commercial:     |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          | 20%/yr    |   | Marketing | Jennifer Conale      |
| Maintenance     |           |   | Contact:  |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     | 609-866-6550         |
| Gov't:          |           |   | Number    |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Fax       | 609-866-6789         |
| Commercial:     |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   | E-Mail:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Evaluation/Demo | both      |   |           |                      |
| Avail           | avail     |   |           |                      |
|                 | free      |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                Win 95/NT         OS:            Win 3.1

  RAM:               8 MB              RAM:           8 MB

  HD Storage:        10 MB             HD Storage:    10 MB
  ------------------ ------------ ---- -------------- -----------------------

+------------------------------+---+-----------------------------------+
|                              |   |                                   |
+------------------------------+---+-----------------------------------+
| **Product Description:**     |   |                                   |
|                              |   |                                   |
| ForeSight is a software tool |   |                                   |
| for forecasting time,        |   |                                   |
| effort, and cost of          |   |                                   |
| non-military software        |   |                                   |
| projects; the projects can   |   |                                   |
| be new developments,         |   |                                   |
| modifications, integration   |   |                                   |
| of off-the-shelf             |   |                                   |
| applications, maintenance    |   |                                   |
| programs, or software        |   |                                   |
| upgrades. Project sizing     |   |                                   |
| metrics include: SLOC,       |   |                                   |
| function points and          |   |                                   |
| predictive object points     |   |                                   |
| (POPs). ForeSight produces   |   |                                   |
| estimates that include:      |   |                                   |
| size, schedules, staffing,   |   |                                   |
| labor effort, cost, and      |   |                                   |
| quality. This model has a    |   |                                   |
| one-button interface to MS   |   |                                   |
| Project for file creation,   |   |                                   |
| importing and updating. It   |   |                                   |
| tracks milestones and        |   |                                   |
| benchmarks the project       |   |                                   |
| against past history.        |   |                                   |
| ForeSight supports           |   |                                   |
| waterfall, spiral,           |   |                                   |
| evolutionary and incremental |   |                                   |
| development projects. It     |   |                                   |
| allows Risk Analysis and     |   |                                   |
| will generate an Estimate To |   |                                   |
| Complete (ETC).              |   |                                   |
+------------------------------+---+-----------------------------------+
|                              |   |                                   |
+------------------------------+---+-----------------------------------+

####### 19D.5.1 ForeSight Inputs

One of the primary inputs for the ForeSight model is program size. Size
is usually expressed in SLOC that can be input directly by the user or
computed using a sizing model included in the ForeSight package. The
Sizer model estimates size by SLOC, by using **predictive object points
(POPs)** or by converting a function point count to SLOC. Other key
inputs include the following:

-   **Application:** The seven basic functional categories of the
    inherent software instruction difficulty are: mathematical, string
    manipulation, data storage and retrieval, on-line, real-time,
    interactive, and operating system. The result of this classification
    procedure is summarized in a single value called Application.

-   **Productivity factor:** A parameter that can be calibrated which
    relates the software program to the organizational capabilities,
    experience, and individual talents of the team that will perform the
    software development. This factor should be based upon historical
    data.

-   **Complexity:** Complexity describes the effects of additional
    factors affecting the development environment that are directly
    related to schedule or time, such as a development tools, personnel
    skills, and potential requirements growth.

-   **Platform:** Platform relates the cost of software development to
    the requirements of the environment in which the software must
    operate. It is a measure of the transportability, reliability,
    testing, and documentation required by the contract.

-   **Utilization:** The proportion of the processor capability used,
    relative to its available speed and memory capacity, is represented
    by the variable called Utilization.

-   **Level of New Design and Code:** Percentage of the end product that
    will require new design and/or coding effort. This input considers
    the degree of modification or reuse for the software.

There are other inputs for internal and external integration effort,
schedule start and end dates, programming language used, and economic
factors.

####### 19D.5.2 ForeSight Processing 

ForeSight uses a common core equation that relates schedule and effort
to software product size and software production capability. The form of
the core equation is:

M = A \* Size^B^;

where M is either effort or time and A and B are functions of software
production capability and/or product type. ForeSight adjusts the size
measure (in units of lines of code, function points, or object points)
for software functionality and amount of reuse. Other areas around the
core equation that influence project performance are treated by the
forecasting engine. The exact equations are proprietary and are not
included in the user's manual.

####### 19D.5.2 ForeSight Outputs

Results are provided in a number of fix formatted forms, including text
and graphic reports, screens, Microsoft Project inputs, and a ForeSight
project file in Microsoft Access format. ForeSight produces estimates
that include: size, schedules, staffing, labor effort, cost, and
quality.

####### 19D.5.3 ForeSight Calibration

The [Help]{.smallcaps} menu provides no indication of how to calibrate
this model other than by changing the individual attribute ratings or
through updating "historical elements". Historical elements are software
systems, subsystems or elements that have been completed and for which
cost, schedule or effort information is available. A historical element
may be integrated into a new system/subsystem as COTS or as an analogy
for forecasting purposes. "In either case ForeSight will calibrate the
technical and cost information to establish a unique performance measure
for each historical element."

####### 19D.5.4 ForeSight Life Cycle Consideration

The ForeSight Life Cycle Model, included in the ForeSight package, is a
detailed model which computes software operation and support costs. The
Profile menu generates a graphical display that is used to control the
distribution of effort expended during the support period. The total
effort is a combination of four distributions: Defect Detection, Defect
Repair, Enhancement, and Adaptation. All four may be viewed on a single
Profile screen at once. Associated with every element of an ForeSight
Estimation Breakdown Structure is a table of data that enables tailoring
of the type listed above. Supplemental data controls the level of tasks
and resources estimated for elements.

###### 19D.6 [KnowledgePLAN Model (Replaces CHECKPOINT)]{.underline}

The KnowledgePLAN model algorithms are derived from over 6,700 projects.
The model is applicable to all types of programs and has its knowledge
base updated annually.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **Knowle  |   | Company:  | Software             |
|                 | dgePLAN** |   |           | Productivity         |
| (replaces       |           |   |           | Research. Inc.       |
|                 | CH        |   |           |                      |
|                 | ECKPOINT) |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 2.0       |   | Address:  | 1 New England        |
|                 |           |   |           | Executive Pk         |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1998      |   |           | Burlington, MA 01803 |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  | 1997      |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Free Demo       | slide     |   | 800 Phone | 781-273-0140         |
| Available:      | show & 30 |   | Number:   |                      |
|                 | day loan  |   |           |                      |
|                 | of SW     |   | Phone     |                      |
|                 |           |   | Number    |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Fax       | 781-273-5176         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing**     | as of     |   | Internet: | [www.spr.com](       |
|                 | 1Q1998    |   |           | http://www.spr.com/) |
| *               |           |   |           |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     |           |   |           |                      |
| Gov't:          |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \$5,900   |   | Marketing | William Walsh        |
| Commercial:     |           |   | Contact:  |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          | \         |   | Phone     | 301-657-6266         |
| Maintenance     | $500/copy |   | Number    |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Fax       | 301-942-4361         |
| Gov't:          |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    | 25us      |   | E-Mail:   | <billw@spr.com>      |
| Commercial:     | ers/\$99K |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                Win 95/NT         OS:            Win 95/NT

  RAM:                                 RAM:           

  HD Storage:                          HD Storage:    
  ------------------ ------------ ---- -------------- -----------------------

+-----------------------------------------------------------------------+
| **Product Description:**                                              |
|                                                                       |
| KnowledgePLAN is a tool that guides the user through the development  |
| of a software project estimate and plan using a large knowledge base  |
| that is updated annually. KnowledgePLAN combines knowledge-based      |
| estimation, "what if" analysis and scheduling functionality within    |
| one tool. Work Breakdown Structures can be integrated into a Project  |
| (MPX compatible) management tool and allows the PM tool to vary       |
| schedules, task assignments and resource allocations. Allows project  |
| sizing by SLOC, Function Point and Analogy. KnowledgePLAN creates and |
| refines detailed project plans for a bi-directional interface with    |
| Microsoft Project or other enterprise project management system. The  |
| tool will track milestones, schedules, resources, actual work effort  |
| and defects found.                                                    |
+-----------------------------------------------------------------------+

####### 19D.6.1 KnowledgePLAN Inputs

Like the other models discussed, the primary input for KnowledgePLAN is
size. However, KnowledgePLAN is different than most other models in that
it works primarily in sizing by analogy or with function points instead
of SLOC. The model will accept SLOC, but converts SLOC to function
points using conversion factors in the model. In addition to size, the
model requires certain inputs for a quick (or basic) estimate, and
additional parameters for a detailed estimate. The inputs for each
option are now summarized.

1.  **Quick Estimate Inputs:** The following inputs are required for
    both quick and detailed estimates:

    1.  **Project description information.**

    2.  **Project nature:** Whether it is a new program, an enhancement,
        a conversion, reengineering, maintenance etc.

    3.  **Project scope:** Whether the program is a stand-alone program,
        program within a system, a disposable prototype etc.

    4.  **Project Topology:** Whether the project is being implemented
        as a stand-alone platform, local or wide area network, a client
        server, or a distributed network.

    5.  **Project class:** Who is the customer/end-user of the software
        project such as contract, commercial, government, IT/MIS and is
        it a single site, multi-site or network.

    6.  **Project type:** What kind of software is the deliverable;
        non-procedural application or system with subtypes such as
        off-line processing, interactive GUI etc.

    7.  **Software Products:** For sizing by analogy, which
        system/application is similar in size (small, medium, large)
        with similar applications such as business, generic, billing
        etc.

2.  **Detailed Estimate Inputs:** KnowledgePLAN has over 100 factors, or
    project attributes, that can be used to fine-tune the model's
    estimates. These are categorized into personnel, technology,
    process, environmental, product and maintenance which are briefly
    described. Each factor is rated essentially from "very low" to "very
    high" in five possible steps with "3" being average or typical.
    Lower ratings generally result in increased effort and schedule. If
    the user has no information for a factor, he or she can leave it as
    "not answered". The model keeps track of how many attributes have
    been answered (i.e., 25%) and this is included in the tool's risk
    assessment.

    1.  **Personnel attributes:** These attributes are divided into four
        subsets: project management, development experience, user
        personnel experience, and quality personnel experience. Examples
        of project management attributes are organization structure,
        team morale, and project management experience.

    2.  **Technology attributes:** These attributes address the impact
        of Life Cycle software tools and hardware platforms on the
        development environment. Examples of attributes are design
        automation environment, project documentation library,
        development hardware stability, and terminal response time.

    3.  **Process attributes:** These attributes address the life cycle
        methodology used on the project that affects development and
        quality assurance. Examples are analysis, life cycle, quality,
        testing, and documentation..

    4.  **Environment attributes:** These attributes address the
        importance of organization and office factors.

    5.  **Product factors:** The Products Attributes dialog box allows
        the user to characterize any constraints that may affect the
        ability to fulfill user requirements such as security or
        performance. The Products Attributes dialog box has two tabs:
        Requirements and Architecture.

####### 19D.6.2 KnowledgePLAN Processing

The specific algorithms used in this model are not defined in the
KnowledgePLAN User's Guide. The Guide does state that the project
estimates are regulated by: algorithms from the project's assigned
knowledge base, by task category inclusion rules, by adjustment rules
from the estimation engine, by task/category properties and by task
estimation flags. The starting point is the knowledge base assigned to
the project by the model based primarily on the answers to the quick
estimate inputs. Knowledge bases contain the rules and estimates for the
base estimate. The algorithms are tuned to the default settings of the
project classification variables and can be viewed/modified by the user.

####### 19D.6.3 KnowledgePLAN Output 

The model provides the user with several different outputs. Like most
other models, KnowledgePLAN provides an estimate of schedule, staffing,
and effort in dollars or person-months in table or Gantt chart views.
KnowledgePLAN provides a risk analysis of various input options
selected; along with estimates for size, defects, test and reliability,
maintenance, documentation, and various productivity parameters.

####### 19D.6.4 KnowledgePLAN Calibration 

The user can calibrate KnowledgePLAN by overriding the model's default
values for inputs or quality and productivity rates. This can either be
done directly or, more commonly, by creating templates based on
historical data. Templates include any values for which the user wants
to override the model's defaults, and are requested as part of a basic
estimate. Templates pass along the domain and knowledge base that the
new project will use. The *KnowledgePLAN User\'s Guide* provides
information about template creation.

####### 19D.6.5 KnowledgePLAN Life Cycle Considerations 

KnowledgePLAN includes more than fifteen "maintenance" inputs in the
"detailed estimate inputs". Examples are maintenance personnel
experience, amount of replacement and restructure planning, number of
sites, and customer support. The user can also specify the amount of
code to be added or deleted annually. The model estimates the annual
cost of maintenance and enhancements for a time period specified by the
user, up to 20 years.

###### 19D.7 [PRICE-S Model]{.underline}

Martin Marietta Price Systems (initially RCA Price, then GE Price)
originally developed this model as one of a family of models for
hardware and software cost estimation. Developed in 1977, PRICE-S was
the first commercially available detailed parametric software cost model
to be extensively marketed and used. The PRICE-S model is proprietary;
not all equations are published, although the *PRICE-S Reference Manual*
describes the basic parametric relationships. The model is applicable to
all types of software projects. It considers all eight phases of the
software development cycle, plus system concept and operational testing
phases.

  ------------------ ------------- ---- ------------ ------------------------------------------------------
  **Product                             **Vendor     
  Information:**                        Profile:**   

  Product:           **PRICE-S**        Company:     Lockheed Martin PRICE Syst.

  Latest Version:    3.0                Address:     700 E. Gate Drive, Suite 200

  Last Release:      1997                            Mount Laurel, NJ 08054

  First Release:     1977               800 Phone    800-437-7423
                                        Number:      

                                        Phone        609-866-6789
                                        Number:      

  **Pricing          **As of            Fax Number:  609-866-6591
  Information:**     1Q1998**                        

  Single User Gov't: \$2,625            Internet:    [www.pricesystems.com](http://www.pricesystems.com/)

  Single User        \$15,000                        
  Commercial:                                        

  Annual Maintenance Same, a lease      Marketing    Jennifer Conale
                                        Contact:     

  Site License       \$11,813           Phone Number 609-866-6550
  Gov't:                                             

  Site License       \$25,000           Fax Number:  609-866-6789
  Commercial:                                        

  GSA:                                  E-Mail:      

  Evaluation/Demo    None                            
  Avail                                              
  ------------------ ------------- ---- ------------ ------------------------------------------------------

+-----------------+-----------+---+-----------+----------------------+
| **System        |           |   |           |                      |
| Requirements:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| *               |           |   | **M       |                      |
| *Recommended:** |           |   | inimum:** |                      |
+-----------------+-----------+---+-----------+----------------------+
| OS:             | U         |   | OS:       | Unix/Motif or Win    |
|                 | nix/Motif |   |           | 95/NT                |
|                 | or Win    |   |           |                      |
|                 | 95/NT     |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| RAM:            |           |   | RAM:      |                      |
+-----------------+-----------+---+-----------+----------------------+
| HD Storage:     |           |   | HD        |                      |
|                 |           |   | Storage:  |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Product       |           |   |           |                      |
| Description:**  |           |   |           |                      |
|                 |           |   |           |                      |
| The PRICE       |           |   |           |                      |
| Software Model  |           |   |           |                      |
| Suite (PRICE-S) |           |   |           |                      |
| is a parametric |           |   |           |                      |
| cost and        |           |   |           |                      |
| scheduling      |           |   |           |                      |
| model that      |           |   |           |                      |
| consists of     |           |   |           |                      |
| three models to |           |   |           |                      |
| estimate costs  |           |   |           |                      |
| and schedules   |           |   |           |                      |
| for the         |           |   |           |                      |
| development,    |           |   |           |                      |
| operation and   |           |   |           |                      |
| support of      |           |   |           |                      |
| computer        |           |   |           |                      |
| software. The   |           |   |           |                      |
| PRICE-S         |           |   |           |                      |
| Acquisition     |           |   |           |                      |
| Model is an     |           |   |           |                      |
| application of  |           |   |           |                      |
| PRICE empirical |           |   |           |                      |
| modeling        |           |   |           |                      |
| methods to the  |           |   |           |                      |
| problem of      |           |   |           |                      |
| forecasting     |           |   |           |                      |
| software cost   |           |   |           |                      |
| and schedule.   |           |   |           |                      |
| The PRICE-S     |           |   |           |                      |
| Sizing Model    |           |   |           |                      |
| utilizes three  |           |   |           |                      |
| analytical      |           |   |           |                      |
| estimating      |           |   |           |                      |
| tools to        |           |   |           |                      |
| estimate the    |           |   |           |                      |
| size of the     |           |   |           |                      |
| software being  |           |   |           |                      |
| developed. The  |           |   |           |                      |
| first tool is   |           |   |           |                      |
| used to         |           |   |           |                      |
| estimate SLOC   |           |   |           |                      |
| based on inputs |           |   |           |                      |
| for qualitative |           |   |           |                      |
| descriptors,    |           |   |           |                      |
| quantitative    |           |   |           |                      |
| descriptors and |           |   |           |                      |
| sizing factors. |           |   |           |                      |
| The second tool |           |   |           |                      |
| is a Function   |           |   |           |                      |
| Point sizer     |           |   |           |                      |
| that converts   |           |   |           |                      |
| the function    |           |   |           |                      |
| point count to  |           |   |           |                      |
| SLOC. The third |           |   |           |                      |
| tool is an      |           |   |           |                      |
| Object Metric,  |           |   |           |                      |
| Predictive      |           |   |           |                      |
| Object Points   |           |   |           |                      |
| (POPs) for      |           |   |           |                      |
| sizing object   |           |   |           |                      |
| oriented        |           |   |           |                      |
| development     |           |   |           |                      |
| projects. The   |           |   |           |                      |
| PRICE-S Life    |           |   |           |                      |
| Cycle Cost      |           |   |           |                      |
| Model is used   |           |   |           |                      |
| to develop      |           |   |           |                      |
| early costing   |           |   |           |                      |
| of the          |           |   |           |                      |
| maintenance and |           |   |           |                      |
| support phase   |           |   |           |                      |
| for a software  |           |   |           |                      |
| project. The    |           |   |           |                      |
| Acquisition     |           |   |           |                      |
| Model provides  |           |   |           |                      |
| the development |           |   |           |                      |
| cost and design |           |   |           |                      |
| parameters to   |           |   |           |                      |
| the Life Cycle  |           |   |           |                      |
| Cost Model,     |           |   |           |                      |
| with the user   |           |   |           |                      |
| inputting the   |           |   |           |                      |
| support         |           |   |           |                      |
| activity and    |           |   |           |                      |
| support period. |           |   |           |                      |
| Support cost    |           |   |           |                      |
| estimates       |           |   |           |                      |
| include         |           |   |           |                      |
| Corrective,     |           |   |           |                      |
| Enhancement and |           |   |           |                      |
| Growth          |           |   |           |                      |
| categories.     |           |   |           |                      |
| ForeSight is a  |           |   |           |                      |
| new, less       |           |   |           |                      |
| expensive       |           |   |           |                      |
| software cost   |           |   |           |                      |
| model that is   |           |   |           |                      |
| compatible with |           |   |           |                      |
| PRICE-S except  |           |   |           |                      |
| for             |           |   |           |                      |
| m               |           |   |           |                      |
| ilitary/mission |           |   |           |                      |
| critical        |           |   |           |                      |
| software        |           |   |           |                      |
| development     |           |   |           |                      |
| projects.       |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

####### 19D.7.1 PRICE-S Inputs

One of the primary inputs for the PRICE-S model is program size. Size is
usually expressed in SLOC that can be input directly by the user or
computed using a sizing model included in the PRICE-S package. The Sizer
model estimates size by SLOC, by using **predictive object points
(POPs)** or by converting a function point count to SLOC. Other key
inputs include the following:

-   **Application.** The seven basic functional categories of the
    inherent software instruction difficulty are: mathematical, string
    manipulation, data storage and retrieval, on-line, real-time,
    interactive, and operating system. The result of this classification
    procedure is summarized in a single value called Application.

-   **Productivity factor.** A calibratible parameter that relates the
    software program to the organizational capabilities, experience, and
    individual talents of the team that will perform the software
    development. This factor should be based upon historical data.

-   **Complexity.** Complexity describes the effects of additional
    factors affecting the development environment that are directly
    related to schedule or time, such as a development tools, personnel
    skills, and potential requirements growth.

-   **Platform.** Platform relates the cost of software development to
    the requirements of the environment in which the software must
    operate. It is a measure of the transportability, reliability,
    testing, and documentation required by the contract.

-   **Utilization.** The proportion of the processor capability used,
    relative to its available speed and memory capacity, is represented
    by the variable called Utilization.

-   **Level of New Design and Code.** Percentage of the end product that
    will require new design and/or coding effort. This input considers
    the degree of modification or reuse for the software.

There are other inputs for internal and external integration effort,
schedule start and end dates, programming language used, and economic
factors.

####### 19D.7.2 PRICE-S Processing 

Parametric relationships which combine management perception and
historical results are used to relate new software projects to costs and
schedules that are typical of the work to be accomplished. Although much
material concerning the PRICE-S algorithms has been published, some
details concerning the algorithms are proprietary and are not available
to the user. It is known that PRICE-S computes a "weight" of software
based on the product of instructions and application inputs, which are
comparable to hardware volume and density respectively. The productivity
factor and complexity inputs are very sensitive parameters that affect
effort and schedule respectively. Platform is known to be an exponential
input; hence, it can be very sensitive. Other input parameters are used
to adjust the "weight" of software for a specific program. A 1988 paper
published by PRICE Systems, entitled *The Central Equations of the PRICE
Software Cost Model*, describes many internal algorithms in detail,
although some algorithms may have been modified since that time.

####### 19D.7.3 PRICE-S Outputs

PRICE-S computes an estimate in person-months which may be converted to
cost in dollars or other currency units. The model estimates schedule by
milestones, with a staffing profile. In addition to cost and schedule
estimates, PRICE-S provides automatic sensitivity and schedule effect
analyses, together with monthly cost and progress summaries to support
budgeting, risk analysis, and project tracking.

####### 19D.7.4 PRICE-S Calibration

Organizational performance history serves as input to a calibration mode
that fits the model to user specific environments that characterize
productivity within a line of business. The PRICE-S model can be run in
the ECIRP (PRICE backwards) mode to calibrate selected parameters. The
most common calibration is that of the productivity factor, which,
according to the *PRICE-S Reference Manual*, tends to remain constant
for a given organization. It is also possible to calibrate platform,
application, and selected internal parameters.

####### 19D.7.5 PRICE-S Life Cycle Consideration

The PRICE-S Life Cycle Model, included in the PRICE-S package, is a
detailed model which computes software operation and support costs. The
Life Cycle Model is designed to be used in conjunction with the
Acquisition Model that provides the development costs and design
parameters to the Life Cycle Model. The primary inputs are support
descriptors such as number of installations, expected growth, and
quality and enhancement levels; three calibratible support productivity
factors; and separate size and expected growth. The Life Cycle Model
outputs cost in three support categories: maintenance, enhancements,
modifications and growth. It also outputs a predicted number of
delivered defects in the program to be supported.

###### 19D.8 RESERVED

####### **19D.9 SoftCost-Object-Oriented (OO) Model**

The SoftCost Object-Oriented (SoftCost-OO) model, developed and marketed
by Resources Calculations, Inc. (RCI), evolved from the SoftCost-Ada
model developed by Don Reifer which was, in turn, based on the published
work of Dr. Robert Tausworthe of the Jet Propulsion Laboratory. In
addition to Ada, SoftCost-OO has calibration files for the C++ language
and generic object-oriented paradigms. The model is markedly different
from the SoftCost-R model, which more closely paralleled COCOMO. The
SoftCost-OO model has a database of over 240 Ada, 50 C++, and 120
object-oriented projects. The model is one of a family of models
marketed by RCI, which includes SoftCost-R (a software cost estimation
tool for data processing, scientific and real-time applications),
Asset-R (a function point tool to estimate software size) and SSM (an
analogy software sizing model).

SoftCost-OO includes a **work breakdown structure (WBS)** file editor
which allows use of preloaded WBS files, modification of pre-loaded
files, or development of a new WBS from scratch. The WBS file editor
also checks for correct allocation of duration and effort percentages
for each WBS item, and for an allowable set of predecessor activities in
a network.

SoftCost-OO is applicable to all types of object-oriented programs and
considers all phases of the software development cycle. The model's
equations are published in the *SoftCost-OO User\'s Guide*; however, the
computer program used to solve these equations and related analyses is
proprietary to RCI.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **Soft    |   | Company:  | Resource             |
|                 | Cost-OO** |   |           | Calculations, Inc.   |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 3.2       |   | Address:  | 7853 E. Arapahoe Ct, |
|                 |           |   |           |                      |
|                 |           |   |           | Ste. 2500            |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1996      |   |           | Englewood, CO        |
|                 |           |   |           | 80112-1361           |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  | 1989      |   | 800 Phone |                      |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 303-267-0379         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | **as of   |   | Fax       | 303-674-4869         |
| Information:**  | 1Q1998**  |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     |           |   | Internet: | [www.rcinc.com](ht   |
| Gov't:          |           |   |           | tp://www.rcinc.com/) |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \$8,000   |   |           |                      |
| Commercial:     |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          | 25% of    |   | Marketing | Tony Collins         |
| Maintenance     | 1^st^     |   | Contact:  |                      |
|                 | year      |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     | 011-440-175-366-2572 |
| Gov't:          |           |   | Number    |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    | \$11,000  |   | Fax       | 011-440-175-366-3398 |
| Commercial:     |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   | E-Mail:   | tco                  |
|                 |           |   |           | llins@dial.pipex.com |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                DOS               OS:            DOS

  RAM:                                 RAM:           

  HD Storage:                          HD Storage:    
  ------------------ ------------ ---- -------------- -----------------------

+-----------------------------------------------------------------------+
| **Product Description:**                                              |
|                                                                       |
| SoftCost-OO is an object oriented and reuse software estimation tool. |
| This tool is a PC-based parametric cost model consisting of a screen  |
| editor, estimation model, and outputs. Phases covered include         |
| software requirements through software testing, including hardware    |
| and software integration testing. There are three databases, Ada,     |
| C++, and Object-Oriented. SoftCost-OO contains five submodels,        |
| sizing, estimating, risk, allocation (effort/labor categories), and   |
| life cycle (maintenance). The sizing submodel calculates equivalent   |
| source lines of code for projects developed in Ada, C++, or a mix of  |
| languages. Function points can also be used. The estimating submodel  |
| develops an effort and schedule estimate for a project using          |
| parametric and size information provided by the user. The risk        |
| submodel allows the user to play \"what-if\" gaming scenarios by      |
| varying effort, schedule, and size. The allocation submodel takes the |
| effort and schedule from the risk submodel and allocates it to the    |
| tasks that comprise the Work Breakdown Structure (WBS) for the        |
| project. This submodel also allocates effort by labor category across |
| life cycle phases. The life cycle submodel calculates maintenance     |
| effort and cost for the system based on the effort selected in the    |
| risk submodel. This submodel contains a load-balancing feature that   |
| allows a user to assess the impact of fixed workforce on maintenance. |
| Three other models marketed by RCI are required for non-object        |
| oriented and non-reuse software development estimates: SoftCost-R is  |
| used for estimating real time software in the traditional development |
| mode, SSM is an analogy software sizing tool and Asset-R is a         |
| function point sizing tool..                                          |
+-----------------------------------------------------------------------+

####### 19D.9.1 SoftCost-OO Inputs 

A key input of SoftCost-OO is size, which can be input either as SLOC or
function points. If SLOC is used, the minimum, most likely, and maximum
SLOC are input for new, reused, and modified components. In addition to
size, SOFTCOST-OO has twenty-eight other inputs, or attributes, in four
categories. Some of the categories and inputs are similar to those used
in COCOMO and REVIC. Most of these inputs require a rating ranging from
"1" to "6" with "3" being "nominal" or "no-effect"; however, a few
inputs such as number of organizations allow other numerical values. A
help screen for each input assists the user in selecting proper values.
The inputs in each category are as follows:

-   **Product attributes:** The attributes in this category are
    application type (avionics, command and control, simulation etc.),
    number of organizations, system architecture (centralized,
    distributed, multiple processors etc.), organizational interface
    complexity, staff and computer resources availability, and security
    requirements.

-   **Process attributes:** The attributes in this category include use
    of modern development methods, use of tools, tool and environment
    stability, degree of standardization, scope of support, and use of
    peer reviews.

-   **Product attributes:** The attributes in this category include a
    technology usage factor, product complexity, requirements
    volatility, degrees of real-time and optimization, reuse costs and
    benefits, and database size.

-   **Personnel attributes:** The attributes in this category include
    analyst capability; application, language, methodology, and
    environment experience; team capability, and number of OO projects
    completed.

SoftCost-OO also has a "Quick Run" capability where, if a similar
project has previously been estimated, it can be recalled and only size
is required as an input. Once size has been entered, the project can be
analyzed like any other project.

####### 19D.9.2 SoftCost-OO Processing

SoftCost-OO is one of the few models for which the mathematical
algorithms are completely described in the Reference Manual. The
SoftCost-OO equations are:

-   PM = A~0~ \* A~l~ \* A~2~ \* A~3~ \* A~4~ \* (SLOC)^C^

-   M = B~0~ \* B~1~ \* B~2~ \* (PM)^D^

PM is the effort in person-months; M is schedule in months; A~0~ and
B~0~ are calibration constants which depend on the application type
input; A~l~ is an architectural constant which depends on the system
architecture input; A~4~ and B~2~ are scaling factors which vary with
size, degree of reuse, and number of OO projects completed; A~3~ is the
product of all inputs not used for other equation factors; and C and D
are exponents which vary with number of OO projects completed. The
*SoftCost-Ada User\'s Manual* illustrates values assigned to ratings for
all model inputs to help the user understand the effect of each on
effort and schedule. The model also has several other equations, as
described in the SoftCost-OO Reference Manual.

####### 19D.9.3 SoftCost-OO Outputs

SoftCost-OO computes a nominal estimate in person-months of effort and
schedule for each project, along with a productivity value and an
estimate of average number of personnel required. The model then allows
for investigation of the sensitivity of each of the input variables and
exploration of alternatives to the nominal estimate or changes in
schedule, effort, or confidence level. The model also provides schedule
outputs for Gantt and PERT charts.

####### 19D.9.4 SoftCost-OO Calibration

The model contains three calibration files; an Ada file, a C++ file, and
a generic OO file. The user must select one of these as an input for a
model estimate. The user can change these files to reflect his or her
environment, but this must be done carefully. RCI plans to develop an
on-line calibration capability to work with SoftCost-OO as a separate
product.

####### 19D.9.5 SoftCost-OO Life Cycle Considerations

SoftCost-OO contains a separate life cycle model for support costs. In
addition to SoftCost-OO developmental inputs, life cycle inputs include
annual change traffic, length of the support period, a sustaining
engineering factor, and economic factors. There is also a provision for
entering and reporting on various staff levels, fixed support costs, and
fixed work force levels.

**19D.10 RESERVED**

####### **19D.11 RESERVED**

###### 19D.12 [Software Life Cycle Model (SLIM)]{.underline}

This model was developed by Quantitative Software Management (QSM)
Corporation, and is based on the work of Lawrence Putnam. SLIM is
proprietary; however, much of the theory behind the model is published
in previous works by Putnam, and his book *Measures of Excellence*. A
key feature of SLIM is the use of the Rayleigh-Norden curve, illustrated
in Figure 19D-1, to allocate resources during a project. The time
integral of the Rayleigh-Norden curve results in the "software
equation", which is as follows: (Note: T~D~ is development time.)

Size = (Productivity Factor) \* (Effort)^1/3^ \* T~D~^4/3^

This software equation is fundamental to SLIM and the entire QSM
approach.

SLIM is applicable to all types of projects, although it was originally
developed for large projects. It computes costs for all software
development phases. The "main build" phase initially computed by SLIM
includes the detailed design through system test phases, but the model
has the option to include the "requirements and design" phase, including
software requirements and preliminary design, and a "feasibility study"
phase to encompass system requirements and design.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **SLIM**  |   | Company:  | Quantitative         |
|                 |           |   |           | Software Management, |
|                 |           |   |           | Inc.                 |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 4.0       |   | Address:  | 2000 Corporate       |
|                 |           |   |           | Ridge, Ste 900       |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1997      |   |           | McLean, VA 22102     |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  | 1978      |   | 800 Phone | 800-424-6755         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 703-790-0055         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | as of     |   | Fax       | 703-749-3795         |
| Information:**  | 1Q1998    |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     |           |   | Internet: | [www.qsm.com](       |
| Gov't:          |           |   |           | http://www.qsm.com/) |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \$16,500  |   |           |                      |
| Commercial:     |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          |           |   | Marketing | Joe Quattrone        |
| Maintenance     |           |   | Contact:  |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     | 703-790-0055         |
| Gov't:          |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    | \$35,000  |   | Fax       | 703-749-3795         |
| Commercial:     |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            | none      |   | E-Mail:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Eval/Demo       | Demo free |   |           |                      |
| Avail:          |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

+-----------------+-----------+---+-----------+----------------------+
| **System        |           |   |           |                      |
| Requirements:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| *               |           |   | **M       |                      |
| *Recommended:** |           |   | inimum:** |                      |
+-----------------+-----------+---+-----------+----------------------+
| OS:             | Win 95/NT |   | OS:       | Win 3.1              |
+-----------------+-----------+---+-----------+----------------------+
| RAM:            |           |   | RAM:      |                      |
+-----------------+-----------+---+-----------+----------------------+
| HD Storage:     |           |   | HD        |                      |
|                 |           |   | Storage:  |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Product       |           |   |           |                      |
| Description:**  |           |   |           |                      |
|                 |           |   |           |                      |
| SLIM is a       |           |   |           |                      |
| software cost,  |           |   |           |                      |
| schedule, risk  |           |   |           |                      |
| and reliability |           |   |           |                      |
| estimation tool |           |   |           |                      |
| for planning,   |           |   |           |                      |
| control and     |           |   |           |                      |
| risk analysis   |           |   |           |                      |
| of developing   |           |   |           |                      |
| software        |           |   |           |                      |
| systems. The    |           |   |           |                      |
| SLIM model is a |           |   |           |                      |
| combination of  |           |   |           |                      |
| a program       |           |   |           |                      |
| evaluation and  |           |   |           |                      |
| review          |           |   |           |                      |
| technique       |           |   |           |                      |
| (PERT)          |           |   |           |                      |
| algorithm,      |           |   |           |                      |
| linear          |           |   |           |                      |
| programming,    |           |   |           |                      |
| Monte Carlo     |           |   |           |                      |
| simulation and  |           |   |           |                      |
| sensitivity     |           |   |           |                      |
| profiling       |           |   |           |                      |
| techniques. It  |           |   |           |                      |
| uses expert     |           |   |           |                      |
| system          |           |   |           |                      |
| methodology and |           |   |           |                      |
| can be          |           |   |           |                      |
| customized to a |           |   |           |                      |
| specific        |           |   |           |                      |
| organization    |           |   |           |                      |
| through the use |           |   |           |                      |
| of historic     |           |   |           |                      |
| data.           |           |   |           |                      |
| Associated      |           |   |           |                      |
| software tools  |           |   |           |                      |
| that can be     |           |   |           |                      |
| procured from   |           |   |           |                      |
| QSM to augment  |           |   |           |                      |
| the             |           |   |           |                      |
| capabilities of |           |   |           |                      |
| SLIM are SLIM   |           |   |           |                      |
| Control         |           |   |           |                      |
| (controls       |           |   |           |                      |
| Software        |           |   |           |                      |
| projects by     |           |   |           |                      |
| employing       |           |   |           |                      |
| advanced        |           |   |           |                      |
| statistical     |           |   |           |                      |
| process control |           |   |           |                      |
| techniques for  |           |   |           |                      |
| monthly "health |           |   |           |                      |
| checks" using   |           |   |           |                      |
| the SEI         |           |   |           |                      |
| recommended     |           |   |           |                      |
| "core metrics") |           |   |           |                      |
| and size        |           |   |           |                      |
| planner.        |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

####### 19D.12.1 SLIM Input

The significant user inputs to the model are as follows:

-   **Size:** Size is one of the two key user inputs to SLIM. Size is
    usually input as SLOC. SLIM does allow the user to input function
    points which are converted to SLOC using a ratio, or "gearing
    factor", that may be specified by the user. The user may also input
    the programming language used and, as in REVIC, inputs the least,
    most likely, and greatest size for each program to be analyzed.

> Figure 19D-1. The Rayleigh-Norden Curve

**Error! Not a valid link.**

-   **Productivity Index (PI):** The other key input, PI, is a
    calibratible parameter that can either be input directly by the user
    or is computed by the model based on a multitude of additional
    inputs. The PI is a number that can vary from 1 to 40; higher values
    result in lower cost and schedule. It is also a very sensitive
    parameter; a change in one integer value can result in a twenty-five
    to thirty percent change in cost. SLIM uses fractions of PIs for
    fine-tuning. The inputs for computing or adjusting PI are as
    follows:

    1.  **Application type:** This is the key determinant of PI when the
        user asks the model to compute it. Some of the nine application
        types are micro-code, avionics, command and control,
        telecommunications, and business. If a program is of multiple
        application types, the user can specify percentages of each
        type.

    2.  **Tooling and methods:** This set of inputs includes factors
        such as hardware familiarity, use of various types of automated
        tools, and robustness and adherence to a development standard.
        For this set of additional inputs and for the next two sets, the
        user rates each factor from 1 to 10, with 5 usually being
        average.

    3.  **Technical difficulty:** This set of inputs includes the amount
        of new algorithms and logic, platform stability, and various
        complexity factors.

    4.  **Personnel profile:** This set of inputs includes management
        effectiveness, skill level, experience, degree of communication,
        and morale factors such as motivation and cohesiveness.

-   **Other Inputs:** The user can specify other factors that affect
    cost and schedule, including the following:

    1.  Phase adjustments: The user can specify the staffing profiles to
        be used during each phase of development, and can customize
        development phases.

    2.  Importance: The user can state the relative importance of cost,
        schedule, and quality for the program being estimated.

    3.  Constraints: The user can specify maximum cost or effort,
        maximum schedule, minimum and maximum staffing levels, and
        **minimum mean-time-to-defect (MTTD)** for the final product.
        The user can also specify the desired probability of meeting
        each constraint. The model will attempt to meet all constraints
        or determine an "optimal" solution that has the greatest chance
        of meeting all specified constraints.

####### 19D.12.2 SLIM Processing

Although the actual equations for SLIM are proprietary, it is known that
SLIM relies heavily on the Rayleigh-Norden curve and its inherent
assumptions. The original curve assumed that the maximum manpower was
allocated at T~D~, the development time, and that there was a 60:40
ratio of support costs to development costs. SLIM adjusts this curve as
required to meet project-peculiar inputs, especially for smaller
programs, and user-specified constraints.

The shape of the curve is determined by three key parameters within the
model: size, PI, and the **manpower buildup index (MBI)**. The MBI is a
number that varies between -3 and 10. It reflects the rate at which
personnel are added to a project. Higher ratings indicate faster buildup
rates, and result in shorter schedules but higher costs. The Rayleigh
curve, as shown in Figure 19D-1, is shifted upward and to the left as
MBI increases. Lower size or higher PI values result in both lower costs
and shorter schedules; the Rayleigh curve is shifted downward and to the
left. Although MBI is a significant parameter, the user cannot input MBI
directly. Instead, it is determined primarily by the user-specified
constraints.

####### 19D.12.3 SLIM Outputs 

The primary output of SLIM is an optimal solution to meet the
constraints specified by the user. In the absence of constraints, the
model will compute a "minimum time" solution for which the user can
reduce cost by relaxing the schedule. The "staffing view" of the model
shows a staffing profile along with the model's computed cost, effort,
schedule, peak staffing required, and MTTD. The staffing view also shows
the probabilities of meeting each of the specified constraints. The
model also has a "ballpark view" and a "consistency view" where time,
effort, MTTD, average staffing, and PI are compared with similar
projects in the SLIM database. SLIM has numerous report options for risk
analysis, defect profiles, and other areas of interest to the user. The
model has a very impressive feature that allows the user to vary a
certain parameter on a chart with the mouse and see the effect on the
other parameters, including risk.

####### 19D.12.4 SLIM Calibration 

The PI for SLIM can (and should) be calibrated using historical data.
The model has on-line calibration capability for the user to calibrate a
PI from historical projects. All that is required are program size,
development time in months, person man-months of effort, and, if
available, MTTD for each historical project. PI is determined to the
nearest tenth (e.g., 8.3) in calibration.

####### 19D.12.5 SLIM Life Cycle Considerations

SLIM has an optional "maintenance" phase output which uses SLIM
development inputs to compute man-months, schedule, and staffing
profiles. The user can not specify any support-peculiar inputs except
the shape of the curve (Rayleigh, stair step, exponential, straight
line, or level load). The model computes all support outputs based on
extrapolation of the staffing curve for the time after software
development is completed using the user-specified curve shape, and the
time to achieve either 99% or 99.9% reliability as specified by the
user.

###### 19D.13 [System Evaluation and Estimation of Resources Software Estimating Model (SEER-SEM)]{.underline}

One of a family of models marketed by Galorath Associates, SEER-SEM is
based on the work of Randall Jensen in his paper "An Improved
Macro-Level Software Development Resource Estimation Model." It uses the
Rayleigh-Norden curve (described in SLIM) to allocate resources during a
software project and to estimate cost and schedule. SEER-SEM is
applicable to all types of programs, and is applicable to all phases of
the software development cycle except system requirements and design.
The model is proprietary.

+-----------------+-----------+---+-----------+----------------------+
| **Product**     |           |   | **Vendor  |                      |
|                 |           |   | P         |                      |
| *               |           |   | rofile:** |                      |
| *Information:** |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Product:        | **S       |   | Company:  | Galorath Associates, |
|                 | EER-SEM** |   |           | Inc.                 |
+-----------------+-----------+---+-----------+----------------------+
| Latest Version: | 4.5       |   | Address:  | 100 N. Sepulveda     |
|                 |           |   |           | Blvd,                |
|                 |           |   |           |                      |
|                 |           |   |           | Ste. 1801            |
+-----------------+-----------+---+-----------+----------------------+
| Last Release:   | 1996      |   |           | El Segundo, CA 90245 |
+-----------------+-----------+---+-----------+----------------------+
| First Release:  |           |   | 800 Phone |                      |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
|                 |           |   | Phone     | 310-414-3222         |
|                 |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| **Pricing       | **as of   |   | Fax       | 310-414-3220         |
| Information:**  | 1Q1998**  |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Single User     |           |   | Internet: | [www.gaseer.com](htt |
| Gov't:          |           |   |           | p://www.gaseer.com/) |
+-----------------+-----------+---+-----------+----------------------+
| Single User     | \$15,000  |   |           |                      |
| Commercial:     |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+
| Annual          |           |   | Marketing | H. Kim Lett          |
| Maintenance     |           |   | Contact:  |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Phone     | 703-893-5733         |
| Gov't:          |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| Site License    |           |   | Fax       |                      |
| Commercial:     |           |   | Number:   |                      |
+-----------------+-----------+---+-----------+----------------------+
| GSA:            |           |   | E-Mail:   | klett@gaseer.com     |
+-----------------+-----------+---+-----------+----------------------+
| Evaluation/Demo | Demo free |   |           |                      |
| Avail:          |           |   |           |                      |
+-----------------+-----------+---+-----------+----------------------+

  ------------------ ------------ ---- -------------- -----------------------
  **System                                            
  Requirements:**                                     

  **Recommended:**                     **Minimum:**   

  OS:                Win 95/NT;        OS:            Win 3.1, Mac Sys 6
                     Mac Sys 7                        

  RAM:                                 RAM:           

  HD Storage:                          HD Storage:    
  ------------------ ------------ ---- -------------- -----------------------

+-----------------------------------------------------------------------+
| **Product Description:**                                              |
|                                                                       |
| SEER-SEM is a software development and maintenance analysis CASE tool |
| that estimates cost, effort, schedule, reliability and risk. The      |
| primary driver of the program is software size along with             |
| descriptions of rework required for pre-existing software and         |
| software designed for reuse. A separately purchased tool SEER-SSM can |
| be used to improve software sizing estimates by using historical      |
| information in concert with relative comparison methods. SEER-SSM     |
| produces size estimates in terms of SLOC or functions. SEER-SEM uses  |
| knowledge bases to describe development scenarios including staff     |
| loading, dynamic reallocation of resources and labor mix for          |
| methodologies.                                                        |
+-----------------------------------------------------------------------+

####### 19D.13.1 SEER-SEM INPUTS 

The inputs for this model can be divided into three categories: size,
knowledge base inputs, and input parameters. Each category of inputs is
now summarized.

-   **Size:** The user can input size using one of three measures: SLOC,
    traditional function points, or Galorath function points. SEER
    converts function points to SLOC before generating an estimate. All
    code is further categorized as "new", "preexists designed for
    reuse", or "preexists not designed for reuse". For pre-existing
    software, the user must specify the amount of code deleted, and the
    percentages of redesign, recoding, and retest required to modify or
    reuse the program for the current application. The model uses PERT;
    therefore, the user must input a "minimum", "most likely", and
    "maximum" value for all size inputs.

-   **Knowledge Base Inputs:** SEER-SEM contains a myriad of knowledge
    bases for different types of software. The knowledge bases assign
    default values to the input parameters described below based on the
    type of software selected. The user must specify the following six
    inputs to specify the knowledge base to be used by the model.

```{=html}
<!-- -->
```
-   **Platform:** The operating environment of the program, such as
    avionics, ground-based, or manned space.

-   **Application:** The overall function of the software, such as
    radar, command and control, mission planning, or testing.

-   **Acquisition method:** The method in which the software is to be
    acquired, such as development, modification, or re-engineering.

-   **Development method:** The method used for development, such as
    waterfall development, evolutionary development, object oriented,
    prototype or incremental development.

-   **Development Standard:** The Standard used in development and
    degree of tailoring to describe documentation, quality, and test
    standards such as ISO, 2167A etc.

-   **Class:** This input is primarily for user-defined knowledge bases.

```{=html}
<!-- -->
```
-   **Input Parameters:** SEER-SEM contains more than thirty input
    parameters with which the user can refine an estimate. As in COCOMO,
    the values normally range from "very low" to "very high". As in
    size, the user must specify a least, greatest, and most likely value
    for each input. A user can use the default values generated by the
    chosen knowledge base if no further information is available. The
    primary categories of input parameters and a brief description of
    each follows.

```{=html}
<!-- -->
```
-   **Complexity:** Assesses the difficulty of the software.

-   **Personnel capability and experience:** The parameters in this
    category, similar to the "personnel attributes" of COCOMO, measure
    the caliber of personnel used on the project.

-   **Development support environment:** Measures the usage of modern
    practices and tools, availability of resources, and frequency of
    changes to the environment.

-   **Product development requirements:** Measures the stringency of
    quality, documentation, and test requirements, as well as the
    frequency of changes in requirements.

-   **Reusability requirements:** Measures the degree of reuse needed
    for future programs and the percentage of software affected by
    reusability requirements.

-   **Development environment complexity:** Measures the complexity of
    the language, application, and host development system used.

-   **Target environment:** Measures special constraints for the target
    environment such as memory, special displays, and security (which is
    the most sensitive input parameter in the model).

-   **Other input parameters:** There are also special inputs for
    schedule constraints, labor rates, integration requirements,
    personnel costs, metrics, and software support.

####### 19D.13.2 SEER-SEM Processing 

Although the model is proprietary, some of the equations of the SEERSEM
model can be found in the *SEER-SEM User\'s Manual* and in articles
published by Randall Jensen. SEER-SEM computes an **effective technology
rating (ETR)** based on several input parameters. The model apparently
uses the Rayleigh-Norden curve to compute the required effort. SEER-SEM
also contains windows where the user can compare two projects, examine
several risk analysis graphs, and see what effect a changed input
parameter will have on the overall development cost and schedule.

####### 19D.13.3 SEER-SEM Outputs

SEER-SEM allows the user to select a variety of output reports and
charts. A "quick estimate" provides a snapshot of size effort, schedule,
and ETR anytime during the estimating process. Optional outputs include
a basic estimate, staffing by month, cost by month, cost by activity,
person-months by activity, and software metrics including delivered
defects.

####### 19D.13.4 SEER-SEM Calibration

The model may be calibrated by computing an ETR from past programs, by
computing cost and schedule multipliers from past programs, or both. The
cost and schedule multipliers are linear multipliers with a nominal
value of one (which would have no effect). The ETR, multipliers, or both
can be incorporated into a custom knowledge base for future programs of
the type calibrated. One limitation of the ETR is that the user cannot
input it directly as a model input; it must be adjusted by changing
other input parameters. The cost and schedule multipliers, however, can
be input directly. In addition to ETRs, knowledge bases and parameter
setting may be calibrated.

####### 19D.13.5 SEER-SEM Life Cycle Considerations

SEER-SEM contains an optional "maintenance" output report which provides
annual costs and person-months for each year of a user-specified
schedule in four categories: corrective, adaptive, perfective, and
enhancements. The user can specify the support time period desired along
with several other support-unique parameters. These include annual
change rate, number of support sites, expected program growth, and
differences between development and support personnel and environment,
and degree of rigor (level of support).

####### 19D.15 Other Models

The models discussed above do not encompass the entire arena of software
parametric cost models; there are numerous other models available. For
example, there are several variants of COCOMO available in addition to
REVIC. Several companies have software cost models that are used solely
within the company which developed the model. New models developed for
more general use do appear occasionally. The models contained in this
appendix were chosen because they were the ones included in most other
Parametric Software Cost Estimating references (therefore are assumed to
be the most widely used), they were free, or were new/inexpensive with a
new approach. A list of other potential estimating tools is included in
Appendix C of the **Software Technology Support Center's (STSC)**
*Report on Project Management and Software Cost Estimation
Technologies*, dated April 1995, available for Internet download at
www.stsc.hill.af.mil.

**19E. Commercial-Off-The-Shelf (COTS) Software**

***Table of Contents***

+--------+------------------------------------------------------+-----+
| 19E.1  | > [Introduction . . . . . . . . . . . . . . . . . .  | 19  |
|        | > . . . . . . . . . . . . . . . . . . . . . . . . .  | E-2 |
|        | > . . . . . . . . . . . . . . . . . . . . . . . . .  |     |
|        | > . . . .]{.smallcaps}                               |     |
+--------+------------------------------------------------------+-----+
| [      | > [COTS Software Five Panel Model . . . . . . . . .  | [   |
| 19E.2] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
| {.smal | > . . . . . .]{.smallcaps}                           | -3] |
| lcaps} |                                                      | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Market Phase . . . . . . . . . . . . . . . . | *   |
| E.2.1* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
|        | > . . . . . . . . . . . . .* . . . . . . .           | -4* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Qualify Phase . . . . . . . . . . . . . . .  | *   |
| E.2.2* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
|        | > . . . . . . . . . . . . . . .* . . . . . .         | -4* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Adapt Phase. . . . . . . . . . . . . . . . . | *   |
| E.2.3* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
|        | > . . . . . . . . . . . . . . . .* . . . . . .       | -6* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Assembly Phase . . . . . . . . . . . . . . . | *   |
| E.2.4* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
|        | > . . . . . . . . . . .* . . . . . .                 | -6* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Update Phase . . . . . . . . . . . . . . . . | *   |
| E.2.5* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
|        | > . . . . . . . . . . . . . .* . . . . . .           | -7* |
+--------+------------------------------------------------------+-----+
| [      | > [COTS Cost Estimation Model Status . . . . . . . . | [   |
| 19E.3] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
| {.smal | > . .]{.smallcaps}                                   | -8] |
| lcaps} |                                                      | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| *19    | > *COCOTS Model . . . . . . . . . . . . . . . . . .  | *   |
| E.3.1* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19E |
|        | > . . . . . . . . . . . . . . . . . . .* . . . . . . | -9* |
|        | > .                                                  |     |
+--------+------------------------------------------------------+-----+
| *19    | > *Other COTS Cost Models . . . . . . . . . . . . .  | *1  |
| E.3.2* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . .* . . . . . .                   | 11* |
+--------+------------------------------------------------------+-----+
| [      | > [COTS Cost Estimation/Pricing Equations . . . . .  | [1  |
| 19E.4] | > . . . . . . . . . . . . . . . . . . . . . .        | 9E- |
| {.smal | > .]{.smallcaps}                                     | 13] |
| lcaps} |                                                      | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Qualification Costs . . . . . . . . . . . .  | *1  |
| E.4.1* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . .* . . . . . . .                 | 15* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Adaptation Costs . . . . . . . . . . . . . . | *1  |
| E.4.2* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . .* . . . . . .                 | 15* |
+--------+------------------------------------------------------+-----+
| *19    | > *Glue Code Assembly Costs . . . . . . . . . . . .  | *1  |
| E.4.3* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . .* . . . . . . .                       | 15* |
+--------+------------------------------------------------------+-----+
| *19    | > *Increased Application Effort Due to COTS          | *1  |
| E.4.4* | > Volatility . . . . .* . . . . . . . .              | 9E- |
|        |                                                      | 16* |
+--------+------------------------------------------------------+-----+
| *19    | > *Software System with COTS Components Costs . . .  | *1  |
| E.4.5* | > . . . . . . . . .* . . . . . .                     | 9E- |
|        |                                                      | 16* |
+--------+------------------------------------------------------+-----+
| [      | > [Impact of COTS on System Testing . . . . . . . .  | [1  |
| 19E.5] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
| {.smal | > . . . . .]{.smallcaps} .                           | 19] |
| lcaps} |                                                      | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| [      | > [COTS Support . . . . . . . . . . . . . . . . . .  | [1  |
| 19E.6] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
| {.smal | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 19] |
| lcaps} | > .]{.smallcaps} .                                   | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| *19    | > *Configuration Management Data . . . . . . . . . . | *1  |
| E.6.1* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > .* . . . . . .                                     | 19* |
+--------+------------------------------------------------------+-----+
| *19    | > *Maintenance Policy . . . . . . . . . . . . . . .  | *1  |
| E.6.2* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . . . . . . . .* . . . . . .     | 20* |
+--------+------------------------------------------------------+-----+
| *19    | > *Sustainability . . . . . . . . . . . . . . . . .  | *1  |
| E.6.3* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . . . . . . . . . . . . . . .* . | 20* |
|        | > . . . . .                                          |     |
+--------+------------------------------------------------------+-----+
| [      | > [COTS Software Risks . . . . . . . . . . . . . . . | [1  |
| 19E.7] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
| {.smal | > . . . . . . . . . . . . . . . . . .]{.smallcaps}   | 21] |
| lcaps} |                                                      | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Functionality or Performance . . . . . . . . | *1  |
| E.7.1* | > . . . . . . . . . . . . . . . . . . . . . . .* . . | 9E- |
|        | > . . . . .                                          | 21* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Interoperability . . . . . . . . . . . . . . | *1  |
| E.7.2* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . . . . .* . . . . . . .         | 22* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Evolution . . . . . . . . . . . . . . . . .  | *1  |
| E.7.3* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . . . . . . . . . . .* . . . . . | 23* |
|        | > . .                                                |     |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Vendor Behavior . . . . . . . . . . . . . .  | *1  |
| E.7.4* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . .* . . . . . . .               | 24* |
+--------+------------------------------------------------------+-----+
| *19    | > *COTS Risk Summary . . . . . . . . . . . . . . . . | *1  |
| E.7.5* | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
|        | > . . . . . . . . . . . . . . . .* . . .             | 25* |
+--------+------------------------------------------------------+-----+
| [      | > [Licensing Schemes . . . . . . . . . . . . . . . . | [1  |
| 19E.8] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
| {.smal | > . . . . . . . . . . . . . . . . . . . .            | 25] |
| lcaps} | > .]{.smallcaps} .                                   | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+
| [      | > [Summary . . . . . . . . . . . . . . . . . . . . . | [1  |
| 19E.9] | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 9E- |
| {.smal | > . . . . . . . . . . . . . . . . . . . . . . . . .  | 27] |
| lcaps} | > . . . . .]{.smallcaps} . .                         | {.s |
|        |                                                      | mal |
|        |                                                      | lca |
|        |                                                      | ps} |
+--------+------------------------------------------------------+-----+

# 19E. Commercial-Off-The-Shelf (COTS) Software {#e.-commercial-off-the-shelf-cots-software .unnumbered}

###### 19E.1 [Introduction]{.underline}

This appendix summarizes the special considerations related to
incorporating COTS (commercial-off-the-shelf) software components into a
system. These considerations vary so much from standard software
development approaches that special versions of software cost estimating
models are being created to deal with the peculiarities. If a COTS
software product is to be used as a stand-alone item, then some
subsections such as those considerations related to "glue code" do not
apply. For a COTS software product that will be integrated into a
system, all the sections of this appendix apply. The majority of this
introduction is extended from the model rationale used by the University
of Southern California (USC), Center for Software Engineering (under the
direction of Dr. Boehm) for the new COTS version of COCOMO (discussed in
section 19E.3).

One of the more significant changes in software development practice
over the past twenty years is the greatly increased emphasis being
placed on building systems incorporating pre-existing software in order
to keep overall development and maintenance costs as low as possible.
One source of pre-existing software is commercial vendors who supply
self-contained off-the-shelf components that can be plugged into a
larger software system to provide capability that would otherwise have
to be custom-built. The two primary distinguishing characteristics of
COTS software are 1) that its source code is not available to the
application developer (software developer that will incorporate a COTS
component into a larger system), and 2) that its evolution is not under
the control of the application developer.

The rationale for building COTS based systems is that development time
is reduced by taking advantage of existing, market proven, vendor
supported products, thereby reducing overall system development costs.
In the case of using such components as databases and operating systems,
this is almost certainly true. However, there are little data available
concerning the relative costs of using the component-based approach. Due
to a of the lack of access to product source code and a lack of control
over product evolution, there is a trade-off that results using the COTS
approach. New software development time can indeed be reduced; however,
the cost of software component integration generally increases.
Moreover, if integrating COTS components, there will be additional
system costs to negotiate, manage, and track licenses to ensure
uninterrupted operation of the system. Moreover, using COTS software
also brings with it a host of unique risks quite different from those
associated with software developed in-house.

The true cost of integrating a COTS software component into a larger
system includes the traditional costs associated with new software
development such as the cost of requirements definition, design, code,
test, and software maintenance. In addition, the cost of integration
includes: the cost of licensing and redistribution rights; royalties;
effort needed to understand the COTS software; pre-integration
assessment and evaluation; post-integration certification of compliance
with mission critical or safety critical requirements; indemnification
against faults or damage caused by vendor supplied components; and costs
incurred due to incompatibilities with other needed software and/or
hardware.

The following sections explain the system development model phases, cost
estimation model status, cost estimation/pricing equations,
testing/supportability/risk considerations, and licensing schemes for
COTS software.

###### 19E.2 [COTS Software Five Panel Model]{.underline}

The COTS model pictured in Figure 19E-1 represents various activities
that address different aspects of turning a set of COTS components into
a COTS-based system. This model was obtained from the Software
Engineering Institute (SEI) at Carnegie Mellon University, who have
authored a number of Special Reports and hosted Workshops devoted to
COTS software. This section will cover in greater detail than section
19.2.4, the model's five phases and outline the some of the cost/risk
considerations associated with them.

> Figure 19E-1. COTS Five Panel Model

**Error! Not a valid link.**

####### 19E.2.1 COTS Market Phase

The COTS Market phase deals with the market survey and analysis
activities that determine the viable candidates for a particular
component, from both a business and a technical perspective. Discussing
the many aspects regarding a market analysis is outside the scope of
this appendix. More information can be obtained by referring to DRAFT
Air Force Handbook 60-113, Buying COTS Handbook
(http://www.safaq.hq.af.mil/eng_ind_env_policy/syseng/policies/
dcotshbc.html). The Air Force Handbook has an extensive section and
attachment on conducting a COTS market analysis.

####### 19E.2.2 COTS Qualify Phase

The **Qualify** phase activities investigate the hidden interfaces and
other characteristics and features of the candidate products. Per SEI in
their article "Component-Based Software Development/COTS Integration,"
component qualification is a process of determining \"fitness for use\"
of previously-developed components that are being applied in a new
system context. Qualification of a component can also extend to include
qualification of the development process used to create and maintain it
(for example, ensuring algorithms have been validated, and that rigorous
code inspections have taken place).

Continuing with the SEI article, there are two aspects of component
qualification: discovery and evaluation. For discovery, the properties
of a component are identified. Such priorities include component
functionality (what services are provided) and other aspects of a
component\'s interface (such as the use of standards.) These properties
also include quality aspects that are more difficult to isolate, such as
component reliability, predictability, and usability. In some
circumstances, it is also reasonable to discover \"non-technical\"
component properties, such as the vendor\'s market share, past business
performance, and process maturity of the component developer\'s
organization. Discovery is a difficult and ill-defined process, with
much of the needed information being difficult to quantify and, in some
cases, difficult to obtain.

The most difficult problem in the qualify phase is to determine the
characteristics of the available COTS products so that a product which
best meets the requirements may be selected. The result of this
discovery process is to reveal the necessary information to make a
selection and identify possible sources of conflict and overlap, so that
the component can be effectively assembled and evolved. The following
excerpt from a Crosstalk article describes the selection process.

> "Experience shows that the selection process for one major product can
> require three to six months of calendar time, multiple engineers and
> programmers, access to sophisticated suites of hardware and software
> environments, and will likely entail the purchase of vendor-provided
> training classes."
>
> *March 1998 STSC Crosstalk*

The rationale behind this time and expense are summarized by Richard D.
Stutzke in his paper "Cost Factors for COTS Integration." Typically the
COTS capabilities which are of interest to the designer are the
completeness of the functions provided, the COTS product's architecture,
and the maturity and expected life of the product. The product's
architecture is particularly important since COTS products often make
assumptions about the environment in which they will operate. The
product\'s architecture dependencies also affect the partitioning of the
components of the COTS product and their dependencies. These
dependencies can have significant cost impacts when the various
components are integrated.

Dr. Stutzke states that during the evaluation process, the analyst must
build a mental model which relates the general and specific knowledge
for both the system being built and the COTS product being considered
for inclusion into that system. He referred to three models that the
analyst must construct to understand a software program:

-   Structure ("top-down hierarchy of elements)

-   Control flow ("program")

-   Data flow ("situation")

A key point concerning these models is that knowledgeable people are
essential to correctly evaluate COTS components expeditiously.

####### 19E.2.3 COTS Adapt Phase

Because individual components are written to meet different
requirements, and are based on differing assumptions about their
context, components often must be adapted when used in a new system. A
March 1998 article in STSC Crosstalk on COTS states that COTS software
does not require coding but does require integration with other
components. As a result, it starts the life cycle as a partially
developed component. The design, construction, and integration and test
development stages must be recast to accommodate early COTS software
integration and testing as well as to develop "Glue Code" (interface
software, configuration files, scripts, utilities, and data files) which
is required to make the COTS software deliver its intended
functionality.

Referring again to the SEI article, components must be adapted based on
rules that ensure conflicts among components are minimized. The degree
to which a component\'s internal structure is accessible suggests
different approaches to adaptation:

-   **White Box.** Access to source code allows a component to be
    significantly rewritten to operate with other components.

-   **Grey Box.** Source code of a component is not modified but the
    component provides its own extension language or application
    programming interface (API).

-   **Black Box.** Only a binary executable form of the component is
    available and there is no extension language or API.

Each of these adaptation approaches has its own positives and negatives;
however, White box approaches, because they modify source code, can
result in serious maintenance and evolution concerns in the long term.
Wrapping, bridging, and mediating (included as part of the "Glue Code")
are specific programming techniques used to adapt Grey- and Black-box
components.

####### 19E.2.4 COTS Assembly Phase

The **Assembly** phase is the integration of the adapted components into
a software architectural infrastructure. Components must be integrated
through some well-defined infrastructure that provides the binding that
forms a system from the disparate components. This infrastructure
supports component assembly and coordination, and differentiates
architectural assembly from ad hoc "glue". Referring again to the March
1998 STSC Crosstalk article, waiting until late in the development
process to test and integrate COTS products, particularly those that are
complex, will not give adequate time to master all their intricacies and
complexities. COTS product testing and integration activities must be
interwoven into more of the development process stages.

Extended from Dr. Stutzke's paper, Table 19E-1 summarizes two types of
potential problems that may occur during the assembly phase. First, it
is very difficult to determine exactly how hard it will be to tailor and
integrate the COTS product into the new system. As indicated in the
table, the steps are first to obtain a detailed, understanding of the
component, then the component must be modified and tested. In some cases
the product must be documented as well.

##### Table 19E-1. Managing Potential Assembly Phase Problems

+-----------------------------------------------------------------------+
| Predicting how hard it will be to integrate the code                  |
|                                                                       |
| -   Obtain a detail understanding                                     |
|                                                                       |
| -   Modify                                                            |
|                                                                       |
| -   Test                                                              |
|                                                                       |
| -   Document (if required)                                            |
|                                                                       |
| Incorporating new versions during integration                         |
|                                                                       |
| -   Cost                                                              |
|                                                                       |
| -   Strategy                                                          |
+-----------------------------------------------------------------------+

The second potential problem is that new versions of the COTS products
are often released during the integration process. This can affect the
cost of development, therefore some sort of strategy is needed to manage
this. The likelihood of new versions increases if the project lasts
longer that one year.

The particular process used by the development team can significantly
affect the cost of integrating COTS components. The three important
process areas are installing and integrating the component initially,
installing new versions of the components which become available during
the integration process, and tracking and controlling the baselines.

####### 19E.2.5 COTS Update Phase

The **Update** phase acknowledges that new versions of components will
replace older versions; in some cases, components may be replaced by
different components with similar behavior and interfaces. These
replacement activities may require that glue code be rewritten, and they
suggest the advantage of well-defined component interfaces that reduce
the extensive testing otherwise needed to ensure that the operation of
unchanged components is not adversely affected.

Per the SEI article, component-based systems may seem relatively easy to
evolve and upgrade since components are the unit of change. To repair an
error, an updated component is swapped for its defective equivalent,
treating components as plug-replaceable units. Similarly, when
additional functionality is required, it is embodied in a new component
that is added to the system. This is a highly simplistic (and
optimistic) view of system evolution however. The new component will
never be identical to its predecessor and must be thoroughly tested,
both in isolation and in combination with the rest of the system. As a
result, replacement of one component with another is often a
time-consuming and arduous task. Wrappers must typically be rewritten,
and side-effects from changes must be found and assessed.

Dr. Stutzke concluded that the "volatility" of COTS products is a source
of significant uncertainty and risk, increasing in magnitude as the
length of the system's planned service life increases. A COTS product is
volatile in two ways: features and price. COTS products are
market-driven and evolve rapidly in response to consumer and competitive
pressures. The vendor adds (and sometimes deletes) features based on
many factors. Regarding pricing, maintenance pricing is the most
volatile (assuming the initial purchase occurs shortly after the
evaluation is completed.)

Changes in maintenance price are primarily of interest for estimating
the operating costs of the deployed system. Typically, vendors encourage
the user to discontinue use of a product they want to retire by
increasing the annual maintenance fees (to cover the reduced number of
active users.) The may also offer attractive price reductions on the
purchase price of new, replacement products. The obsolescence of product
features and/or changes in maintenance price may cause significant
redesign and refurbishment of the deployed system long before its useful
life is over. Of course, these costs impact the lifecycle costs of the
system.

###### 19E.3 [COTS Cost Estimation Model Status]{.underline}

The USC's Center for Software Engineering under the direction of Dr
Boehm is in the process of developing a version of COCOMO tailored for
the unique requirements of estimating COTS software. This model is
called COCOTS (COnstructive COTS). COCOTS is considered experimental and
evolving. The latest information on this model can be obtained at
website http://sunset.usc.edu/COCOTS/cocots.html. The contents of this
section were extended from the information obtained from the COCOTS web
site. COCOTS' goal is the development of a comprehensive COTS
integration cost modeling tool. The approach taken was to first examine
a wide variety of sources in an attempt to identify the most significant
factors driving COTS integration costs, and then to develop a
mathematical form for such a model.

####### 19E.3.1 COCOTS Model

The current model provides insight into the most important factors that
should be considered when estimating the cost of integrating COTS
components, regardless of the specific tool or methodology used to
perform that estimation. "The model represents a prototype
implementation of the first of four submodels being proposed for COCOTS,
namely, the glue code submodel. The model is not yet mature enough for
the estimates it provides to be used with a high level of confidence.
However, the cost parameters contained within the model and the criteria
used to rate those parameters as described in the user guide certainly
offer much insight into the questions a software cost estimator should
be considering when working on a system to be built with COTS
components."

The USC COTS integration cost model version 1.0 takes the following
form:

SIZE = KSLOC (1.0 + BRAK/100)

$$\text{PM} = A \ast \left( \text{SIZE} \right)^{\beta} \ast \prod_{i = 1}^{\text{13}}\text{EM}_{i}$$

COST = (PM)\*(\$\$/PM)

> **Table 19E-2. Definition of Symbols**

  ------------ -------------------------------------------------------------
  **Symbol**   **Description**

  A            A linear scaling constant to provide an accurate effort
               estimate when all EM are nominal, provisionally set to 12.0.

  β            A nonlinear scaling constant that accounts for the influence
               of factors that have exponential effects, provisionally set
               to 1.0.

  BRAK         Breakage: Percentage of COTS glue code thrown away due to
               requirements volatility.

  KSLOC        Size of COTS component glue code expressed in thousands of
               source lines of code.

  PM           Person Months of estimated COTS integration effort.

  \$\$/PM      Estimated average labor rate per person-month.

  EM           The thirteen Effort Multipliers or cost drivers, each of
               which assumes one of five possible values based upon the
               following ratings: very low, low, nominal, high, and very
               high. Nominal ratings have a multiplier value of 1.0.
  ------------ -------------------------------------------------------------

USC suggests the following procedure when attempting to use their model
for a COTS integration cost estimation exercise:

1.  Estimate the amount of glue code that is expected to be needed to
    integrate a set of COTS products into a software application. Only
    the integration or glue code linking the COTS component to the
    larger application should be included in the estimate,
    [not]{.underline} the code internal to the COTS component itself.

2.  Estimate the percentage of glue code that will be lost due to
    breakage during the integration effort. This will be a function of
    the number of COTS packages being integrated into the new system
    overall, the average number of updated product releases expected per
    COTS package over the life of the system development, and the
    average interface breakage per product release. Breakage refers to
    COTS integration code that must be reworked as a result of a change
    in system requirements. Breakage also includes integration code
    required when a new release by the vendor of a COTS product which
    necessitates that the newer version of the product be installed
    before system delivery. Breakage does not refer to code that must be
    reworked due to bugs introduced by the programmer, or due to defects
    in design. The Breakage percentage is best estimated by acquiring
    knowledge of two things: 1) the vendor's past history regarding
    releases of the COTS product in question or of similar products that
    the vendor markets, and 2) the customer's past history regarding
    demanding changes in requirements after development and COTS product
    integration has begun.

3.  Determine the effective size of the glue code development effort by
    feeding the estimates derived in steps 1 and 2 into the formula for
    SIZE.

4.  Assign each effort multiplier a rating on the given scale from very
    low to very high, which best characterizes the unique conditions
    pertaining to the COTS integration effort.

5.  Determine the overall estimated effort for this integration task by
    feeding the estimate for SIZE and the rated effort multipliers into
    the formula for Person-months (PM).

6.  Determine the estimated cost by multiplying estimated PM by the
    estimated average labor rate (\$\$/PM).

This completes the COTS integration cost estimation procedure.

####### 19E.3.2 Other COTS Cost Models

In preparing to create the COCOTS model, USC researched other available
COTS models. They discovered that quantitative COTS integration models
available for review in the public domain were almost non-existent.
Three models were discussed in the COCOTS paper, two by Science
Applications International Corp. (SAIC) and one by Loral Federal
Systems. USC's discussion of these models is summarized here to provide
further understanding into the cost drivers that influence COTS
integration costs.

One model proposed by Richard Stutzke of SAIC centered on the issue of
COTS volatility, which is, the frequency with which a COTS vendor
releases new versions of its software. (This has been identified as one
of two primary determinants in the cost of using COTS software, the
other being the actual size of the interface or "glue" code needed to
integrate a given COTS product.) His model suggested a way to quantify
the added cost associated with using a COTS product that has significant
volatility. It is included here to assist the analyst if faced with this
situation. In brief, Dr. Stutzke proposed the following formula:

Extra Cost due to Volatility = CV\*AC\*IS\*(CS + CC)

*Where :*

*CV = Component volatility (number of new releases of the COTS component
over life of the project).*

*AC = Architectural coupling (number of other components which interface
with the given COTS component).*

IS = Apparent interface size in terms of the number of entry points,
procedures, functions or other methods used to access the COTS
component, weighted by the number of arguments passed.

*CS = Cost of screening the COTS component and all the other components
with which it interfaces to determine the impact of a new release.*

*CC = Cost of making changes to impacted components.*

As of USC's writing, no attempt had been made to implement this model.
It also addresses only one aspect, although an important one, associated
with integrating COTS software.

Still another approach to modeling COTS integration costs was taken at
SAIC. This second model addressed more the end user costs of using COTS
software. The model takes this form:

$$\begin{array}{r}
\text{COTS Integration Cost } = \left\lbrack (\text{Cost of COTS}\text{ product license})\text{ * }(\text{Number of licenses}) \right\rbrack \\
\text{                                        } + \text{COTS product training cost}) \\
\text{                                        } + (\text{COTS Interface or glue code cost})
\end{array}$$

This model highlights some important sources of cost, but ignored the
details of determining the last term, the cost of developing the COTS
product glue code.

An alternate model that attempts to address the issue of estimating the
cost of developing COTS interface code has been described by Tim Ellis
of Loral Federal Systems. This model was well into the implementation
stage (when the article was written), having been calibrated to a number
of internal Loral COTS integration projects, with more continually being
added to the model's calibration database. Mr. Ellis describes the COTS
integration model in these general terms:

> Work Units = IQ \_ (Size, Drivers)
>
> Productivity = Labor-months/Work Unit
>
> Estimated Effort in LM = WU\*P

*Where:*

*Size = Size of the COTS interface or glue code in function points.*

*Drivers = Set of seventeen COTS integration cost drivers.*

*LM = Labor-months.*

*P = Productivity.*

*WU = Work units.*

The greatest influence the Loral model had on the development of the USC
COCOTS integration cost model came from its seventeen cost drivers,
which served as one of the starting points in the definition of the USC
COCOTS model drivers. The disadvantage of the Loral model is that the
total size of the COTS component is generally thought to be irrelevant,
and the Assessment & Assimilation (AA) and Percentage of Integration &
Test Modification (IM) COCOMO parameters were not considered to be
adequate descriptors of the factors affecting COTS integration efforts.

###### 19E.4 [COTS Cost Estimation/Pricing Equations]{.underline}

Estimating the cost of a software system that includes COTS components
requires the inclusion of more cost areas than typical software
development efforts. Figure 19E-2 provides a general illustration of the
standard software development costs included in the Software Cost
Estimating Process of section 19.2. Figure 19E-3 shows the four
additional cost areas when COTS components are integrated to form a
complete system. This section will discuss the general cost equations
that can be used to calculate the costs in these four areas. These two
figures and the equations in this section were adapted from a briefing
given in August 1998 by Dr. Boehm and Christopher Abts titled "COCOTS
Software Integration Cost Model."

> **Figure 19E-2. SW Cost Estimates without COTS**

Error! Not a valid link.

> Figure 19E-3. SW Cost Estimates with COTS in System

**Error! Not a valid link.**

####### 19E.4.1 COTS Qualification Costs

COTS Qualification costs typically have two parts, the initial filtering
efforts to identify candidates and then the final selection effort to
select the winning COTS vendor. The equation for the filtering effort
done during discovery takes the form:

$$$\text{Total Effort} = (\text{\# COTS Candidates})\left( \frac{\text{Average Filtering Effort}}{\text{Candidate}} \right)$

Final selection is based upon the evaluation of the candidate components
for certain functionality attributes. The general equation to estimate
this effort has the form:

$$\text{Total Effort} = \sum_{\text{Attributes}}^{}{(\text{\# COTS Candidates})}\left( \frac{\text{Ave Assessment Effort per Attribute}}{\text{Candidate}} \right)$$

####### 19E.4.2 COTS Adaptation Costs

These costs are associated with tailoring the selected COTS components
to meet requirements and take the form:

$$\text{Total Effort} = \sum_{\text{Complexity}}^{}\left( \text{\#COTS Candidates Adapted at Level} \right)_{i}\left( \text{Ave Effort at Adapting Level} \right)_{i}$$

USC uses five adaptation effort complexity levels (very low to very
high), differentiated by the difficulty of needed scripts, API
iterations, etc.

####### 19E.4.3 Glue Code Assembly Costs

This effort takes the form of the equation used in the currently
existing COCOTS Glue Code model. Section 19E.3.1 contains additional
detail on the variables in this equation. Further information and a copy
of this model can be obtained from the COCOTS website
<http://sunset.usc.edu/COCOTS/cocots.html):>

$$\text{Total Effort} = A \ast \left\lbrack \left( \text{Size} \right)\left( 1 + \text{Breakage} \right) \right\rbrack^{\beta} \ast \prod_{}^{}\left( \text{Effort Multipliers} \right)$$

*Where:*

*A = Linear scaling constant*

*Size = Size of the glue code in SLOC or Function Points*

*Breakage = Glue code due to changes in the requirements and/or COTS
volatility*

*Effort Multipliers = 13 parameters, each with a range from very low to
very high*

*β = Architectural scale factor with settings from very low to very
high.*

####### 19E.4.4 Increased Application Effort Due to COTS Volatility

Approximate Application Effort Model:

$$\text{Total Effort} = \left( \text{Application Effort} \right) \ast \left\lbrack \frac{\text{BRAK COTS}}{\text{100}} \right\rbrack \ast \left( \text{EM} \right)_{\text{COTS}}$$

*Where:*

######## BRAK COTS = % application code breakage due to COTS volatility {#brak-cots-application-code-breakage-due-to-cots-volatility .unnumbered}

*BRAK = % application code breakage otherwise*

*EM = Product of Effort Multipliers*

Using the above four equations added together (which form the basis of
COCOTS) would provide an estimate for the COTS software integration
effort. This effort could then be converted to a cost estimate by
multiplying it by the average cost of a person-month. There are other
COTS costs that also must be added in: the costs to obtain and support
the COTS components (COTS SW), the maintenance costs for the COTS
integration software (COTS SW integration) which can be estimated using
the completed version of COCOTS and the costs associated with any
required COTS hardware (COTS HW).

####### 19E.4.5 Software System with COTS Components Costs

The total cost of a software system that includes COTS components could
be estimated by adding together the following:

**Table 19E-3 Total Cost of Software System**

  ----------------------------------- -----------------------------------
  **Cost Components**                 **Cost Estimation Model**

  New SW development (not COTS)       SW cost models in Appendix D

  New SW maintenance (not COTS)       SW cost models in Appendix D

  COTS SW integration                 COCOTS

  COTS SW integration maintenance     COCOTS

  COTS SW                             See below

  COTS HW                             See below
  ----------------------------------- -----------------------------------

Estimating the cost of obtaining the COTS software for each COTS
component would take the following form:

$$\begin{array}{r}
\text{ COTS SW = Cost}\lbrack\text{acquisition office}\rbrack\text{ +Cost}\lbrack\text{licenses}\rbrack \\
\text{                    }\left\{ \text{= Cost}\lbrack\text{\#licenses,\#features,\#platforms}\rbrack \right\}\text{ +} \\
\text{                   Cost}\lbrack\text{implementation}\rbrack\left\{ \text{= Cost}\lbrack\text{training}\rbrack\text{ + Cost}\lbrack\text{install}\rbrack \right\}\text{ + } \\
\text{                  Cost}\lbrack\text{Op}\text{.}\text{ \& Maint}\text{.}\rbrack\left\{ \text{= Cost}\lbrack\text{maint}\text{.}\text{  licenses}\rbrack\text{ + Cost}\lbrack\text{support}\rbrack \right\}
\end{array}$$

Estimating the cost of obtaining the COTS Hardware would take the
following form:

$$\begin{array}{r}
\text{COTS HW } = \text{ Cost}\lbrack\text{acquisition office}\rbrack + \\

\end{array}$$

*Where:*

*i = number of processors, storage, workstation, communications*

Unit costs vary by quantity, platform, and time. if all the items are
not procured at the same time, there will be a need to consider
time-phasing of acquisition, implementation, operations & maintenance.
The biggest challenge will be the complex, dynamic COTS price
structures.

###### 19E.5 [Impact of COTS on System Testing]{.underline}

This section (extended from the draft Air Force Handbook 60-113, "Buying
COTS"), is included in order that the analyst may consider these two
questions when pricing a proposal or developing a cost estimate. Does
the proposal/estimate cover these tests? Is some testing missing or is
there too much testing? An important advantage of COTS software is
reduced delivery time. This is realized through the elimination of the
research and development phase of the acquisition cycle and subsequent
development test and evaluation. In general, testing is not required
when the software will be used and maintained by similarly skilled
people in the same environment as that for which it was designed and
existing data (contractor or other sources) provides reasonable answers
to performance and supportability issues. If COTS software components
are being integrated into a larger system, then system integration
testing and selected component testing would be required. Operational
testing is strongly recommended if operational or support environments
differ from the contractor\'s. Further, preselection testing in the
Qualify phase should be conducted to minimize technical, operating and
support risk.

For stand-alone COTS in a commercial-like (same-environment)
application, no testing should be required if the application is the
same as commercial use of the item. If there is any doubt, an analyst
should acquire and qualify a test sample prior to selection. After
delivery of a successful item, any deficiencies that appear should be
covered by the vendor\'s commercial warranty.

For COTS embedded in a larger system, feasibility testing to qualify a
test sample should be done prior to selection and integration into the
system. Development Test and Evaluation (DT&E) of the complete system is
required. Hardware and software integration tests should be conducted as
well as user testing (typically, operational test and evaluation (OT&E))
at the system level. COTS deficiencies that appear during or after
system-level testing should be covered by the vendor\'s commercial
warranty. Special considerations for specific testing areas are as
follows:

**Development Testing.** Development testing should be conducted at the
system level but not at the COTS assembly level since COTS is already
developed. If an item does not meet performance standards/requirements,
select other COTS or go for a non-COTS design effort.

**Operational Testing.** Selection of COTS items does not automatically
mean there will be no need for operational testing. However, operational
testing for COTS components (or stand-alone COTS) should be limited or
waived if market investigation data and, as applicable, pre-selection
qualification testing will satisfy the requirements. Operational
testing, as previously mentioned, should be conducted for a system with
integrated COTS components.

**Supportability Evaluation.** The impact of COTS on system
supportability probably requires testing that should include reliability
testing, organizational level maintainability testing, human factors
engineering tests, standardization measurements and safety analysis.

**Compatibility Testing.** Follow-on compatibility testing in fielded
systems could be conducted for all significant hardware or software
revisions to a COTS item. This testing should verify that the interfaces
are not violated, and be completed before the Government accepts the
change. Reviewing the vendor\'s service bulletins and engineering change
orders (ECOs) can help identify which changes would require testing.

**Quality Assurance Provisions.** A quality assurance provision should
be specified for each system-level functional and physical requirement
in the contract. However, quality control of individual COTS items
should be covered by the vendor\'s commercial warranties.

###### 19E.6 [COTS Support]{.underline}

The rapidly changing market and technology of COTS items means that it
will have a limited useful life cycle. In their draft Handbook 60-113,
the "Buying COTS," the Air Force considers a number of important factors
that could impact the cost in the support of commercial items. The
Handbook recommends that support requirements should be defined before
SIR release. Support requirements should also include an up-front
definition of the system support requirements to the item level, a
lifetime support strategy and appropriate contract language to implement
the support strategy. Acquisition contracts should include support such
as maintenance, support equipment and training for the total expected
life cycle as well as pre-planned product improvements or when
replacements are needed.

A market analysis should be performed while the requirements document is
still in draft form. If COTS items are to be embedded in a larger system
by a prime contractor, be sure that the prime contractor acquires all
the required information from the proposed vendors.

The preferred method of support for COTS products is through local
acquisition and locally-acquired contractor service. Any decision for
life-cycle contractor logistics support must be accompanied by adequate
planning. In the absence of a formal contractor support agreement, plan
for an appropriate level of organic management and support for COTS
assets.

When determining the support for a commercial item, it is recommended to
evaluate original contractor\'s support, alternate vendors\' support,
and Government support with regards to impact on competition,
operational requirements, total support costs, and support availability.

Three major areas of concern in COTS support are configuration
management and data; maintenance policy; and sustainability.

####### 19E.6.1 Configuration Management Data 

Except under very unusual circumstances, it is not wise to demand full
design disclosure engineering data, for system support. Even if the
vendor was agreeable to selling the data, it would quickly be obsolete
because the government would not have design control. Contractors can
unilaterally change the design as necessary to suit their customer or
market requirements. COTS designs should be documented to the system
interfaces level.

For COTS software, the government should acquire and maintain
appropriate licensing and subscription services (vendor field change
orders and software releases) throughout the life of the system. If the
Government secures change authority, it must clearly define the limits
of what the Government can change, and restrict alterations in any way
that would void the licensing or subscription service. Independent
software changes by the Government, or a Government freeze to an earlier
COTS software version, will turn COTS into a unique product.

####### 19E.6.2 Maintenance Policy 

Employment of COTS will usually lock the government into two-level
maintenance: organizational and depot. The most desirable support
concept is contract repair, preferably via competition. A good strategy
is to price support options while the original buy is still in
competition. The riskiest course is an in-house depot repair, which
should be selected only under exceptional conditions because it will
have to be a specialized operation if even possible at all. The vendor
might refuse to sell the data and tools to develop a capability.

####### 19E.6.3 Sustainability 

Because of the design control the Government has had over in-house
developmental software, many systems have been maintained and upgraded
to serve multiples of their original design life. Market pressures
usually send commercial designs more quickly into obsolescence not so
easily dealt with. It is not uncommon for this to occur within five
years, which means that retrofit funding should be routinely projected
approximately at the time of original system fielding. There should be
cost estimates for maintenance and eventual replacement of COTS
software. Licensing and subscription service represent a significant
annual cost, and replacement cost will also be considerable.

###### 19E.7 [COTS Software Risks]{.underline}

This section is extended from the USC COCOTS model's "COTS Integration
Modeling Study", dated June 1997. There are four key COTS integration
characteristics that make COTS integration significantly different from
other forms of software development (including maintenance). These
characteristics require traditional approaches to software development
to be significantly revised when considering program risk. The
characteristics are:

-   No control over a COTS product's functionality or performance.

-   Most COTS products are not designed to interoperate with each other.

-   No control over a COTS product's evolution.

-   COTS vendor behavior varies widely.

####### 19E.7.1 COTS Functionality or Performance 

If the source code can be modified, it's not really COTS and its future
becomes the Government's responsibility. Even as black boxes, big COTS
products have formidable complexity: Microsoft people have indicated
that Windows 95 has 25,000 entry points.

[Resulting Pitfalls]{.underline}

-   **Using the waterfall model on a COTS integration project.** With
    the waterfall model, the Government specifies requirements, and
    these determine the capabilities. With COTS products, the situation
    is reversed: the capabilities determine the \"requirements\" or the
    delivered system features.

-   **Using evolutionary development with the assumption that every
    undesired feature can be changed to fit the needs.** COTS vendors do
    change features, but they respond to the overall marketplace and not
    to individual users.

-   **Believing that advertised COTS capabilities are real.** COTS
    vendors may have had the best of intentions when they wrote the
    marketing literature, but that doesn't help when the advertised
    feature isn't there.

[Resulting Recommendations]{.underline}

-   **Use risk management and risk-driven spiral-type process models.**
    Assess risks via prototyping, benchmarking, reference checking, and
    related techniques. Focus each spiral cycle on resolving the most
    critical risks.

-   **Perform the equivalent of a \"receiving inspection\" upon initial
    COTS receipt** to ensure that the COTS product really does what it
    is expected to do.

-   **Keep requirements negotiable until the system's architecture and
    COTS choices stabilize.**

-   **Involve all key stakeholders in critical COTS decisions.** These
    can include users, customers, developers, testers, maintainers,
    operators, or others as appropriate.

####### 19E.7.2 COTS Interoperability 

Interoperability problems can cause COTS integration cost and schedule
overruns by factors of four to five.

[Resulting Pitfalls]{.underline}

Lack of COTS interoperability exacerbates each of the previously cited
pitfalls. Some additional direct pitfalls are:

-   **Premature commitment to incompatible combinations of COTS
    products.** This can happen in many ways: haste, desire to show
    progress, politics, or uncritical enthusiasm with features or
    performance. Short-term emphasis on rapid application development is
    another source of this pitfall.

-   **Trying to integrate too many incompatible COTS products.** Four
    can be too many. In general, trying to integrate more than a
    half-dozen COTS products from different sources would indicate a
    high-risk program.

-   **Deferring COTS integration till the end of the development
    cycle.** This puts the most uncontrollable problem on the critical
    path as delivery is approached.

-   **Committing to a tightly-coupled subset of COTS products with
    closed, proprietary interfaces.** These restrict the downstream
    options; once committed, it's hard to back out.

[Resulting Recommendations]{.underline}

The previously cited recommendations on risk-driven processes and
co-evolving requirements and architecture are appropriate here. In
addition:

-   **Include demonstrations of COTS interoperability and scalability as
    risks to be resolved.**

-   **Go for open architectures and COTS substitutability.** In the
    extremely fast-moving software field, the ability to adapt rapidly
    to new "best-of-breed" COTS products is competitively critical.

####### 19E.7.3 COTS Evolution 

Again, COTS vendors respond to the overall marketplace and not to
individual users. Upgrades are frequently not downward compatible. Old
releases become obsolete and unsupported by the vendor. If COTS
architectural mismatch does not happen initially, COTS architectural
drift can easily occur later. Current COTS-intensive systems often have
higher software maintenance costs than traditional systems, but good
practices can make them lower.

[Resulting Pitfalls]{.underline}

Lack of evolution controllability exacerbates each of the previously
cited pitfalls. Some additional direct pitfalls are:

-   **\"Snapshot\" requirements specifications and corresponding
    point-solution architectures.** These are not good practices for
    traditional systems; with uncontrollable COTS evolution, the
    maintenance headaches become even worse.

-   **Under-staffing for software maintenance,** and lack of COTS
    adaptation training for maintenance personnel.

-   **Tightly coupled, independently evolving COTS products.** Just two
    of these will make maintenance difficult; more than two is much
    worse.

-   **Assuming that uncontrollable COTS evolution is just a maintenance
    problem.** It can attack development schedules and budgets as well.

[Resulting Recommendations]{.underline}

The previously-cited risk-driven and architecture-driven recommendations
are also appropriate here. In addition:

-   **Stick with dominant commercial standards.** These make COTS
    product evaluation and substitutability more manageable.

-   **Use likely future system and product line needs as well as current
    needs as COTS selection criteria.** These can include portability,
    scalability, distributed processing, user interface media, and
    various kinds of functionality growth.

-   **Use flexible architectures facilitating adaptation to change.**
    These can include message/event-based software bus, encapsulation,
    and layering.

-   **Carefully evaluate COTS vendors' track records with respect to
    predictability of product evolution.**

-   **Establish a pro-active system release strategy, synchronizing COTS
    upgrades with system releases.**

####### 19E.7.4 COTS Vendor Behavior 

Vendor behavior varies widely with respect to support, cooperation, and
predictability. Sometimes a COTS vendor is not even the developer, just
a value-added re-seller. Given the three major sources of COTS
integration difficulty above, an accurate assessment of a COTS vendor's
ability and willingness to help out with the difficulties is
tremendously important. Experience indicates that the value of a COTS
vendor's support follows a convex curve with respect to the vendor's
size and maturity. Small vendors often lack the capability to support
the user; very large vendors have the capability, but not the
motivation.

####### 19E.7.5 COTS Risk Summary

Per Dr. Stutzke, designers and managers should realize that there may be
a large hidden price to be paid for the use of COTS components
incorporated into a software system. The most frequently encountered
problem is the need to update the COTS product during integration. Such
updates often cause incompatibles to arise between COTS products that
previously worked together. Increases in the price of the annual license
and maintenance agreements can affect operating costs. Liabilities for
damage caused by the failure of a COTS product may be very costly. More
difficult to predict is the loss of critical features as the vendor
upgrades the product in response to market pressures (including
withdrawal of the product from the market, in the extreme case). Such
losses can cause significant redesign of the deployed system long before
its useful life is over.

###### 19E.8 [Licensing Schemes]{.underline}

Referring again to Dr. Stutzke's article, the license terms for the COTS
products can affect the cost of the overall system and even affect the
choice of the architecture. Dr. Stutzke adapted a memo by Gibson and
Mankofsky which grouped their license types into the six types shown in
Table 19E-4. The names used were defined to be as descriptive as
possible since there is no standard terminology used by vendors to refer
to these types of licenses. This makes it extremely difficult to compare
the costs of products offered by different vendors.

> Tab1e 19E-4. Types of COTS Software License

+-----------------------------------------------------------------------+
| **Universal Site**                                                    |
+-----------------------------------------------------------------------+
| -   Covers use on all machines at a geographic location               |
+-----------------------------------------------------------------------+
| -   Annual (Payment of maintenance fee provides yearly update)        |
+-----------------------------------------------------------------------+
| -   Perpetual (No maintenance fee. Repurchase required.)              |
+-----------------------------------------------------------------------+
| -   Number of users is unlimited                                      |
+-----------------------------------------------------------------------+
| **Site**                                                              |
+-----------------------------------------------------------------------+
| -   Covers one machine/platform type at a given location              |
+-----------------------------------------------------------------------+
| -   Number of users is unlimited                                      |
+-----------------------------------------------------------------------+
| -   May be annual or perpetual                                        |
+-----------------------------------------------------------------------+
| **Node-Locked**                                                       |
+-----------------------------------------------------------------------+
| -   Covers use on a single machine (tied to machine's serial number)  |
+-----------------------------------------------------------------------+
| -   May be annual or perpetual                                        |
+-----------------------------------------------------------------------+
| **Floating**                                                          |
+-----------------------------------------------------------------------+
| -   Allows multiple users on a client/server network                  |
+-----------------------------------------------------------------------+
| -   Controlled by one or more "1icense servers\"                      |
+-----------------------------------------------------------------------+
| -   Buy seats (\"sockets\") or tokens (\"resource units tied to       |
|     platform type)                                                    |
+-----------------------------------------------------------------------+
| **Metered**                                                           |
+-----------------------------------------------------------------------+
| -   Usage measured and charged (or limited to some specified amount   |
|     per day, month etc.                                               |
+-----------------------------------------------------------------------+
| -   Can be combined with node locked, floating, platform or token     |
|     schemes                                                           |
+-----------------------------------------------------------------------+
| **Public Domain**                                                     |
+-----------------------------------------------------------------------+
| -   Pay distribution charge                                           |
+-----------------------------------------------------------------------+
| -   May pay registration fee or donation (\"shareware")               |
+-----------------------------------------------------------------------+
| -   May be free to non-profit organization or Government agency.      |
|     (Must purchase the rights for commercial resale)                  |
+-----------------------------------------------------------------------+

The license costs are primarily of interest in making the \"buy or
build\" decision during the Product Design phase or the Proposal phase
(since this is when the price must be determined). The cost of
developing the products must be traded against the cost of purchasing
the necessary number of copies and integrating them into the system.
In-house developed products have a significant non-recurring engineering
cost which is paid once. All subsequent copies of the component may;
however, be used for essentially no cost for the life of the system,
regardless of how many copies of the product are produced. For a COTS
product, the non-recurring engineering cost is absorbed by the vendor
but the buyer (developer of the new system) must pay a license fee for
each copy of the component which is used. Complex trades-off analysis
must be made between functionality, cost, schedule, risk, etc.,
involving not just the development and production cost but also the
operating costs of the deployed system when performing a make (develop)
or buy (COTS) decision.

Other factors may affect the cost of a COTS product. First, discounts
may be available if multiple copies are purchased (volume discounts. )
Second, reduced prices may be offered if several related products are
purchased together ("bundling\"). Third, sometimes the first year of
maintenance is provided free with the initial purchase of the product.
Fourth, if the product is utilized during the development process, it is
possible that an additional warranty must be purchased prior to delivery
of the system to the customer. Fifth, there may be liability costs in
the event COTS components cause failure in the system during operations.
Such costs could cover damage to equipment, injures to personnel and
loss of service. Last, there may be refurbishment costs associated with
the loss of critical functions as the product evolves in response to
market pressures. The functions needed may be removed from later
versions of the product.

Multiple types of licenses must be mixed to obtain the necessary number
of copies of a product. Sometimes a full license is purchased to obtain
a complete set of documentation and several other \"right to use"
licenses are purchased to allow copies of the \"full\" product to be
used on other platforms. Such mixing and matching can have significant
impacts on the total cost of a system.

###### 19E.9 [Summary]{.underline}

The utilization of COTS software components/products is intended to
reduce the cost and schedule of software development programs. If the
COTS component is a stand-alone product, it probably will. However, for
integrating COTS components into a larger system, there are many factors
that must be considered. This appendix has outlined the primary
considerations and the basic cost equations that apply to COTS software
cost estimating and pricing.

# Bibliography {#bibliography .unnumbered}

Abdel-Hamid, Tarek, and Stuart E. Madnick. [Software Project Dynamics:
An Integrated Approach]{.underline}. Englewood Cliffs, NJ:
Prentice-Hall, 1991.

Abts, Christopher M. "COTS Software Integration Cost Modeling Study."
Los Angeles, CA: University of Southern California, June 1997.

Albanese, Frank Jr. "The Application of Regression-Based and Function
Point Software Sizing Techniques to Air Force Programs*."* AFIT Thesis
GCA/LSY/88S-1. Air Force Institute of Technology, 1988.

Albrecht, A. J., and J. E. Gaffney. \"Software Function, Source Lines of
Code, and Development Effort Prediction: A Software Science
Validation.\" [IEEE Transactions on Software Engineering]{.underline}
Vol SE-9, No.6, November 1983.

ANSIJ-STD-016. [Software Development]{.underline}. New York: American
National Standards Institute, 1995

Ayres, Bradley J., and William M. Rock. "Development of a Standard Set
of Software Indicators for Aeronautical Systems Center." AFIT Thesis
GSS/ENC/92D-1. Air Force Institute of Technology, September 1992.

Bailey, Elizabeth K., et al. [A Descriptive Evaluation of Automated
Software Cost-Estimation Models]{.underline}. Alexandria, VA: Institute
For Defense Analysis, October 1986.

Barber, Brent L. "Investigative Search of Quality Historical Software
Support Cost Data and Software Support Cost-Related Data." AFIT Thesis
GSS/LSY/91D-1. Air Force Institute of Technology, 1991.

Boehm, Barry W. \"A Spiral Model of Software Development and
Enhancement, Tutorial: Software Engineering Project Management."
Abstract. Washington, DC: IEEE, 1988.

\_\_\_\_\_. [Software Engineering Economics]{.underline}. Englewood
Cliffs, NJ: Prentice-Hall, 1981.

Boehm, Barry W. and Christopher M. Abts. "COCOTS Software Integration
Cost Model: An Overview." Presentation handouts, 1998.

\_\_\_\_\_. "COTS/NDI Software Integration Cost estimation & USC-CSE
COTS Integration Cost Calculator v2.0 User Guide." Los Angeles, CA:
University of Southern California, September 1997.

Boehm, Barry W., et al. \"COCOMO 2.0 Program.\" Presentation handouts.
ISPA 1994 Conference, Munich, Germany, 1994.

\_\_\_\_\_. \"The COCOMO 2.0 Software Cost Estimation Model.\" Abstract.
International Society of Parametric Analysts, 1995.

\_\_\_\_\_. \"An Overview of the COCOMO 2.0 Software Cost Model.\"
Abstract. Software Technology Conference, 1995.

Bozoki, G. J. [SSM User\'s Guide]{.underline}. Redwood City, CA: GJB
Associates, 1984.

Carnegie Mellon Distance Education Home Page. Slides.
http://www.sei.cmu.edu/mse/msd/html: Carnegie Mellon University, 1998.

Carney, David J. and John Foreman. "Component-Based Software
Development/COTS Integration." Software Technology Review Search Results
(http://www.sei.cmu.edu/str/descriptions/cbsd_body.html): Carnegie
Mellon University, 1998.

Carney, David J. and Patricia A. Oberndorf. "The Commandments of COTS:
Still in Search of the Promised Land." Crosstalk, The Journal of Defense
Software Engineering Vol 10, No.5, May 1997.

Cheadle, William G. \"DOD-STD-2167 Impacts on Software Development.\"
[ISPA Journal of Parametrics]{.underline} Vol VI, No.4, December 1986.

Christensen, David S., and Daniel V. Ferens. \"Using Earned Value for
Performance Measurement on Software Development Projects.\" [Acquisition
Quarterly]{.underline} Spring, 1995.

College of Business Administration Website. IS322 course material.
www.cba.uc.edu: The University of Cincinnati, February 1998.

Computer Science Telecommunications Program Website. CS457 course
material. www.ctsp.umkc.edu: University of Missouri-Kansas City,
February 1998

DeMarco, Tom. [Controlling Software Projects]{.underline}. Englewood
Cliffs, NJ: Prentice-Hall, 1982.

DOD-STD-1521B. [Technical Reviews and Audits for Systems, Equipment,
Munitions, and Computer Software]{.underline}. 4 June 1985.

DOD-STD-1703. [Software Products Standards]{.underline}. 12 February
1987.

DOD-STD-2167A. [Defense System Software Development]{.underline}*.* 29
February 1988.

Ferens, Daniel V. \"A Common Sense Approach to Software Cost
Estimation.\" [ISPA Journal of Parametrics]{.underline} Vol VI, No.2,
June 1986.

\_\_\_\_\_. \"Software Support Cost Models: Quo Vadis?\" [ISPA Journal
of Parametrics]{.underline} 4.4, December 1984.

Ferens, Daniel V., and Mark T. Hunter. \"Use of Cost plus Award Fee
Contracts for Software Development Efforts.\" [Journal of Cost
Analysis,]{.underline} Spring 1996.

Fulton, Richard, and Sherry Stukes. [Space and Missiles Center Software
Database User\'s Manual Version 1.0]{.underline}*.* Oxnard, CA:
Management Consulting and Research, 1994.

Gaffney, John E. & Durek, Thomas A. [Software Reuse\--Key to Enhanced
Productivity; Some Quantitative Models SPC-TR-88-015]{.underline}.
Herndon, VA: Software Productivity Consortium, April 1988.

Galorath Associates. [SEER-SEM User\'s Manual]{.underline}. Los Angeles,
CA: Galorath Associates, September 1994.

\_\_\_\_\_. [SEER-SEM User\'s Manual Release 4.5]{.underline}. Updated.
El Segundo, CA: Galorath Associates, September, 1996.

\_\_\_\_\_. [SEER-SSM User\'s Guide]{.underline}. Marina del Rey, CA:
Galorath Associates, 1991.

Galorath, Dan, and Karen McRitchie. \"Software Size Measures: Source
Lines of Code and Functions.\" Presentation handouts. Marina del Rey,
CA: Galorath Associates, 1992.

Garnus, David (Editor). [IFPUG Function Point Counting Practices Manual
Release 3.4]{.underline}. Westerville, OH: IFPUG, July 1992.

George A., and Roy C. Russell. "Software Cost Estimating Models: A
Comparative Study of What the Models Estimate*."* AFIT Thesis
GCA/LAS/93S-4. Air Force Institute of Technology, 1993.

Gerlich, Rainer, and Ulrich Denskat. \"A Cost Estimation Model for
Maintenance and High Reuse.\" [Proceedings of the European Software Cost
Modeling Conference 1994 Ivrea, Italy]{.underline}. 1994.

Glass, Robert L., and Ronald A. Noiseux. [Software Management
Guidebook]{.underline}. Englewood Cliffs, NJ: Prentice-Hall, 1981.

Henderson, Garland S. "The Application of Function Points to Predict
Source Lines of Code for Software Development*."* AFIT Thesis
GCA/LSY/92S-4. Air Force Institute of Technology, 1992.

Herbert, Chris E., and Robert W. Ratliff. [Database Management System
and Calibration Tool]{.underline}*.* Denver, CO: Martin Marietta
Astronautics Group, April 1993.

Humphrey, Watts S. [Managing the Software Process]{.underline}*.*
Reading, MA: Addison-Wesley, 1990.

IEEE-STD-729. [IEEE Standard Glossary of Software Engineering
Terms]{.underline}. 1983.

Ikatura, Minoru, and Akio Takayanagi. \"A Model For Estimating Program
Size and Its Evaluation.\" [Proceedings of the Sixth International
Conference on Software Engineering]{.underline} 1982.

International Function Point User\'s Group. [IFPUG Function Point
Counting Practices Manual Version 4.0]{.underline}. Westerville, OH:
IFPUG, 1995

Institute for Systems Analysis Corporation. [ISA Cost Expert User\'s
Guide]{.underline}. Bethesda, MD: Institute for Systems Analysis, 1990.

ITT Research Institute. [Test Case Study: Evaluating the Cost of Ada
Software Development]{.underline}. Lanham, MD: ITT Research Institute,
April 1989.

Jensen, Randall W. \"An Improved Macro-Level Software Development
Resource Estimation Model.\" [Proceedings of the Fourteenth Asilomar
Conference on Circuits, Systems, and Computers]{.underline} 1981.

Jones, Capers. "Programming Languages Table." Burlington, MA: Software
Productivity Research, March 1995.

\_\_\_\_\_. "What Are Function Points." Burlington, MA: Software
Productivity Research, March, 1995.

\_\_\_\_\_. *[Applied Software Measurement]{.underline}*. New York:
McGraw-Hill, 1991.

\_\_\_\_\_. [Programming Productivity]{.underline}. New York:
McGraw-Hill, 1986.

McCabe, Thomas J., and G. Gordon Schulmeyer. \"System Testing Aided By
Structural Analysis.\" [Proceedings of the 1982 IEEE
COMPSAC]{.underline} November 1982.

MIL-STD-480B. [Configuration Control-Engineering Changes, Deviations and
Waivers]{.underline}. 17 July 1992.

MIL-STD-490. [Specification Practices]{.underline}. 31 August 1995.

MIL-STD-498. [Software Development and Documentation]{.underline}. 5
December 1994.

MIL-STD-499A. [Military Standard: Engineering Management]{.underline}. 1
May 1974.

Mosemann, Lloyd K., II, et al. [Guidelines for Successful Acquisition
and Management of Software Intensive Systems]{.underline}, Hill AFB, UT:
Software Technology Support Center, September 1994.

Musa, John D. et. Al. Software Reliability: Measurement, Prediction,
Application. New York: McGraw-Hill, 1987.

Ourada, Gerald L. *"*Software Cost Models: A Calibration, Validation,
and Comparison." AFIT Thesis GSS/LSY/91D-11. Air Force Institute of
Technology, 1991.

Parikh, G., and N. Zvegintzov. \"The World of Software Maintenance.\"
[Tutorial on Software Maintenance]{.underline}. Piscataway, NJ: IEEE
Computer Society Press, 1983.

Park, Robert E. \"An Open Letter to Cost Model Evaluators.\" [ISPA
Journal of Parametrics]{.underline} Vol IX, No.3, October 1989.

Paulk, M.C., et al. [Capability Maturity Model for Software, Version 1.1
(CMU/SEI-93-TR-24)]{.underline}. Pittsburgh, PA: Carnegie Mellon Univ.,
February 1993.

Pressman, Roger S. [Software Engineering: A Practitioner\'s
Approach]{.underline}. 3rd ed. New York: McGraw-Hill, 1992.

PRICE Systems. [ForeSight 1.0 Evaluation Copy]{.underline}. Computer
software. Mt. Laurel, NJ: Lockheed-Martin Corporation, June 1997.

\_\_\_\_\_. [The Central Equations of the PRICE Software Cost
Model]{.underline}*.* Moorestown, NJ: PRICE Systems, 1988.

\_\_\_\_\_. [PRICE-S Reference Manual]{.underline}. 3d ed. Moorestown,
NJ: Martin-Marietta, October 1993.

Putnam, L. H., and A. Fitzsimmons. \"Estimating Software Costs.\"
[Datamation]{.underline} September-November 1979.

Putnam, Lawrence H., and Ware Myers. [Measures of
Excellence]{.underline}*.* Englewood Cliffs, NJ: Prentice-Hall, 1992.

Quantitative Software Management Corporation. [Size Planner User\'s
Manual Version 1.0]{.underline}. McLean, VA: QSM, 1987.

\_\_\_\_\_. [SLIM 3.0 For Windows User\'s Manual]{.underline}. McLean,
VA: Quantitative Software Management, 1993.

Ratliff, Robert W., et al. [SASET 3.0 User\'s Guide]{.underline}.
Denver, CO: Martin Marietta Astronautics Group, April 1993.

Reifer, Donald J. [ASSET-R Version 2.0 User\'s Manual]{.underline}.
Torrance, CA: Reifer Consultants, Inc., January 1989.

\_\_\_\_\_. [SoftCost-Ada User\'s Manual, Version 2.2]{.underline}.
Torrance, CA, Reifer Consultants, Inc., 1991.

\_\_\_\_\_. [SoftCost-R User\'s Manual Version 8.0]{.underline}.
Torrance, CA: Reifer Consultants, Inc., 1989.

Resource Calculations, Inc. [SoftCost-OO User\'s Guide Version
3.1]{.underline}. Denver, CO: Resource Calculations, Inc., 1994.

Roetzheim, William H., and Reyna A. Beasley. \"Best Practice in Software
Project Cost and Schedule Estimating.\" [Cost Xpert User\'s
Manual]{.underline}. Jamul, CA: Marotz, Inc., 1997.

Selby, Richard, D. Schmidt, and J. Berney. \"Metric Driven Analysis and
Feedback Systems for Enabling Empirically-Guided Software Development.\"
[Proceeding of the 13th International Conference of Software
Engineering]{.underline}, May 1991.

Selby, Richard. [Software Reuse: Emerging Technology]{.underline}.
Piscataway, NJ: IEEE Computer Society Press, 1988.

Sofstar Systems. [COSTAR User\'s Manual Version 3.00]{.underline}.
Amherst, NH: Sofstar Systems, 1989.

Software Engineering Institute. [Checklists and Criteria for Evaluating
the Cost and Schedule Estimating Capabilities of Software Organizations,
CMU/SEI-95-SR-005]{.underline}. Pittsburgh, PA: Carnegie Mellon Univ.,
January 1995.

\_\_\_\_\_. [Workshop on COTS-Based Systems,
CMU/SEI-97-SR-019]{.underline}. Pittsburgh, PA: Carnegie Mellon Univ.,
November 1997.

Software Productivity Research, Inc*.* [CHECKPOINT For Windows User\'s
Guide Release 2.2.2]{.underline}. Burlington, MA: Software Productivity
Research, 1993.

\_\_\_\_\_. [KnowledgePLAN User\'s Guide Version 2.0]{.underline}.
Burlington, MA: Software Productivity Research, April 1997.

Stutzke, Richard D. "Cost Factors for COTS Integration", [Proceedings of
the 10th International COCOMO User\'s Conference]{.underline}. October
1995.

\_\_\_\_\_. \"Software Cost Estimating Technology: A Survey\",
[Proceedings of the 10th International COCOMO User\'s
Conference]{.underline}. October 1995.

SYSCON Corporation. [Avionics Software Support Cost Model: Final Report
(AFWAL-TR-82-1173)]{.underline}. Washington, DC: SYSCON, 1983.

Thibodeau, Robert. [An Evaluation of Software Cost Estimating
Models]{.underline}. Huntsville, AL: General Research Corporation, 10
April 1981.

United States. Department of Defense. Joint Logistics Commanders.
[Practical Software Measurement\--A Guide to Objective Program
Insight]{.underline}. 27 March 1996.

\_\_\_\_\_. Software Technology Support Center. [Report on Project
Management and Software Cost Estimation Technologies]{.underline}. Hill
AFB, UT: STSC*,* April 1995.

\_\_\_\_\_. U. S. Air Force Acquisition Systems Engineering Homepage.
"DRAFT Air Force Handbook 60-113, Buying Commercial Off-The-Shelf"
http://www.safaq.hq.af.mil/eng_ind_env_policy/syseng/policies/dcotshbc.html:
SAF/AQRE, October 1998.

\_\_\_\_\_. U. S. Air Force Cost Analysis Agency. [REVIC Software Cost
Estimating Model User\'s Manual, Version 9.0]{.underline}. Springfield,
VA: NTIS, 18 February 1991.

\_\_\_\_\_. U. S. Air Force Data and Analysis Center for Software
Homepage. "*Function Point FAQ."* http://mach10.rome.kaman.com: DACS,
September 1997.

\_\_\_\_\_. U. S. Air Force Material Command "White Paper #2 on Methods
for Evaluating Similar Items." Dayton, OH: AFMC, January 1996.

\_\_\_\_\_. U. S. Air Force. [Space Systems Cost Analysis Group,
Software Methodology Handbook]{.underline}. Springfield, VA: NTIS, June
1995.

\_\_\_\_\_. U. S. Naval Center for Cost Analysis. Software Development
Estimating Handbook. Arlington, VA: NCCA, February 1998.

\_\_\_\_\_. U. S. Navy. Naval Command, Control, Surveillance Center,
RDT&E Division Software Engineering Process Office. [Software Size, Cost
and Schedule Estimation Process]{.underline}. Springfield, VA: NTIS,
June 1996.

United States. Department of Energy Financial Assistance Homepage. "10
CFR 600.145 Cost and price analysis." http://www.pr.doe.gov/
f600toc.html: DOE, August 1998.

United States. Federal Aviation Administration Acquisition Management
System Homepage. FAA Pricing Handbook, 1998. http://fast.faa.gov/
v898/index.htm.

\_\_\_\_\_\_. Federal Aviation Administration. FAA Cost Estimating
Handbook, 1998.

United States. General Accounting Office. [Air Traffic Control: Improved
Cost Information Needed to Make Billion Dollar Modernization Investment
Decisions]{.underline}. Washington: GPO, January 1997.

United States. General Services Administration. "Contract Pricing
Reference Guides." Federal Acquisition Internet Homepage.
http://www.gsa.gov/fai Washington, August 1998.

United States. National Aeronautics and Space Administration.
[Parametric Cost Estimating Handbook]{.underline}. Springfield, VA:
NTIS, Fall 1995.

\_\_\_\_\_. Software Engineering Laboratory. Cost and Schedule
Estimation Study Report (SEL-93-002). Greenbelt, MD: SEL, November 1993.

University of Illinois Computer Science Home Page. CS302 course
material. www-courses.cs.uiuc.edu: University of Illinois at
Urbana-Champaign, 1998

University of Southern California Center for Software Engineering COCOTS
Home Page. http://sunset.usc.edu/COCOTS/cocots.html: University of
Southern California, 1998.

Wheaton, Marilee J. \"Functional Software Sizing Methodology.\" [ISPA
Journal of Parametrics]{.underline}. Vol VI, No.1, March 1986.

Whetstone, Mark J. "Developing Software Size Estimating Relationships
Based on Functional Descriptions of the Software." AFIT Thesis
GSM/LSY/86S-24. Air Force Institute of Technology, 1986.

Youll, David P., [Making Software Development Visible]{.underline}.
Chichester,

England: John Wiley and Sons, 1990.