**CHAPTER VI**

**AUDITING PARAMETRIC ESTIMATES**

**A MANAGEMENT VIEWPOINT**

**CHAPTER VI**

**AUDITING PARAMETRIC ESTIMATES**

The following paper written by Michael Thibault, Deputy Director, DCAA,
provides background from an auditor\'s point of view on auditing
parametric cost estimating techniques.

**[COST ESTIMATING RELATIONSHIPS: A DCAA PERSPECTIVE]{.underline}**

**Introduction**

The U.S. Defense Contract Audit Agency (DCAA) was established in 1965.
DCAA\'s mission is to perform all necessary contract audits for the
Department of Defense and, when requested, to perform contract audit
services on a reimbursable basis for other government agencies. In
essence, DCAA provides accounting and financial advisory services for
procurement and contract administration activities. Contract audit
activities include providing professional advice on accounting and
financial matters to assist in the negotiation, award, administration,
repricing, and settlement of contracts. DCAA audits are conducted in
accordance with generally accepted government auditing standards
established by the General Accounting Office.

This paper discusses the government\'s expectations when defense
contractors use parametric cost‑estimating relationships for estimating
government contract costs. The paper also emphasizes that companies must
meet all the adequacy criteria set out in the Federal Acquisition
Regulations (FAR) and applicable supplements to obtain approval for
their estimating systems. Companies must apply the same criteria to
their parametric cost‑estimating relationships to ensure they are
acceptable for use in estimating systems.

DCAA believes now, as it has always believed, that parametric estimating
techniques using cost‑estimating relationships are acceptable in the
appropriate circumstances for proposing costs on government contracts.
DCAA is ready and willing to work with industry in the evolution of
parametrics. Operation Desert Shield and Operation Desert Storm
dramatically demonstrated that our government must be capable of
responding quickly to changing procurement requirements. Parametric
systems can help us do just that. Future estimating systems must be
responsive, accurate, and cost effective.

**Background**

DCAA was issuing official guidance on parametric systems as early as
1978. Parametrics was broadly defined as a technique that employs one or
most cost‑estimating relationships to estimate costs associated with
developing, manufacturing, or modifying an end item. In the 1980s, DCAA
auditors reported an increase in the number of contractors using
parametric cost estimating. DCAA developed and issued audit guidance to
assist the field auditors in this new area. Studies conducted by DCAA;
the Office of the Secretary of Defense Cost Analysis Improvement Group;
Headquarters, Air Force Contract Management Division; Headquarters,
Aeronautical Systems Division; and the Space Systems Cost Analysis Group
provided the basis for DCAA audit guidance issued in 1982.

**Parametric Criteria**

This guidance was also the subject of an article written for the Spring
1982 issue of Journal of Parametrics published by the International
Society of Parametric Analysts. Charles 0. Starret, Jr., then‑Director
of the Defense Contract Audit Agency, wrote the article entitled
\"Parametric Cost Estimating \-- An Audit Perspective.\" The guidance
contained in that article is essentially the same as the guidance given
to DCAA auditors today. It reiterates DCAA\'s long‑held view that
parametrics is an acceptable estimating technique. The 1982 article
included the criteria a contractor should apply before submitting a
contract price proposal using parametrics. The criteria are still on
point today, and they are:

1\) Logical relationships

2\) Significant statistical relationships

3\) Verifiable data

4\) Reasonably accurate predictions

5\) Proper system monitoring

**Logical Relationships**

Contractors are expected to demonstrate that cost‑to‑noncost‑estimating
relationships are logical. \"Logical relationship\" is often difficult
to determine in a finite sense, yet is very important. DCAA\'s primary
concern in this area is that a contractor consider all reasonable
logical estimating alternatives and not use only the first apparent set
of variables. Contractor analysis may disclose multiple alternatives
that appear logical. Statistical testing should be used to help identify
the best alternative.

**Significant Statistical Relationships**

Contractors are also expected to demonstrate that a significant
statistical relationship exists among the variables used in a parametric
cost‑estimating relationship. There are several statistical methods such
as regression analysis that can be used to validate a cost‑estimating
relationship; however, no single uniform test can be specified.
Statistical testing may vary depending on an overall risk assessment and
the unique nature of a contractor\'s parametric data base and the
related estimating system. Proposal documentation should describe the
statistical analysis performed, including the contractor\'s explanation
of why the cost‑estimating relationship is statistically valid.

**Verifiable Data**

There must be a system in place for verifying data used for parametric
cost‑estimating relationships. In many instances, the auditor will not
have previously evaluated the accuracy of noncost data used in
parametric estimates. For monitoring and documenting noncost variables,
contractors may have to modify existing information systems or develop
new ones. Information that is adequate for day-to-day management needs
may not be reliable enough for contract pricing. Data used in parametric
estimates must be accurately and consistently available over a period of
time, and easily traced to or reconciled with source documentation.

**Reasonably Accurate Predictions**

The contractor\'s demonstration that the parametric cost‑estimating
relationships predict costs with a reasonable degree of accuracy is also
important. The key is that if the contractor\'s analysis of historical
estimating and cost performance data shows that the parametric
estimating system is as accurate as a discrete estimating system, then
the government has increased assurance of receiving a fair and
reasonable price.

As with any estimating relationship derived from prior history, it is
essential for the contractor to document that the work being estimated
using parametric cost‑estimating relationships is comparable to the
prior work from which the parametric data base was developed.

**Proper System Monitoring**

The contractor should also ensure that cost‑to‑noncost parametric rates
and factors will be monitored periodically in the same manner as is
expected for cost‑to‑cost rates and factors. Because of improved
technology, production changes, or better pricing alternatives,
cost‑estimating relationships can and do change. The contractor should
be prepared to revalidate any parametric cost‑estimating relationship
whenever system monitoring discloses that the relationship has changed.

**Audit Planning and Requirements**

The old expression, \"The more things change, the more things stay the
same,\" seems to apply. Government procurement procedures may change to
accommodate changes in the services and products it buys, but the basic
procurement goal of getting products and services for fair and
reasonable prices remains the same. And so it goes with auditing.
Whether DCAA audits proposed costs for space stations or for missiles,
DCAA\'s basic aim of ensuring that the proposed costs are allowable,
allocable, and reasonable and therefore acceptable for government
reimbursement is still the same. How DCAA accomplishes its audit
objective varies with the sophistication of contractor accounting and
estimating systems.

Auditors begin the audit by ensuring they have the requisite familiarity
with DCAA guidance on estimating systems and techniques. This guidance
is contained in DCAA\'s Contract Audit Manual (CAM), which is available
to the general public. The auditor then proceeds to do the following:

1\) Ensure they are familiar with the company\'s estimating policies and
procedures.

2\) Identify the estimating methods used to develop the proposal.

3\) Determine that the supporting cost and pricing data for the
individual proposal was derived in accordance with the contractor\'s
estimating system and is in compliance with applicable regulations.

The auditor plans the audit scope using what is known about the
contractor. For example, the audit scope will vary depending upon the
estimating methods the contractor uses. In addition, the auditor will
consider the following types of questions:

\* What is the dollar amount and type of contract contemplated?

\* Has the contractor established strong internal controls and sound
accounting and estimating systems?

\* What kinds of testing does the contractor do to ensure compliance
with these systems?

\* What does our prior audit experience tell us about the contractor\'s
internal controls or estimating practices?

Audit planning requires the auditor to answer all of these questions and
to make a determination regarding the government\'s risk. Judgment is
then exercised in deciding the degree of risk that the estimate could be
materially misstated. This assessment of risk will be used to decide
what audit procedures to employ.

The auditor identifies the method of estimating the contractor uses to
determine the kind of support that should be available. A contractor
could be using any or all of the following methods:

1\) Detailed \-- also known as the bottoms‑up approach. This method
divides proposals into their smallest component tasks and are normally
supported by detailed bills of material.

2\) Comparative \-- develops proposed costs using like items produced in
the past as a baseline. Allowances are made for product dissimilarities
and changes in such things as complexity, scale, design, and materials.

3\) Judgmental \-- subjective method of estimating costs using estimates
of prior experience, judgment, memory, informal notes, and other data.
It is typically used during the research and development phase when
drawings have not yet been developed.

Parametric estimating techniques may be used in conjunction with any of
these methods. Whatever the method selected by the contractor, it must
comply with applicable laws and regulations. The laws and regulations
most often encountered in dealing with parametrics are:

1\) CAS 401, \"Consistency in Estimating, Accumulating, and Reporting
Costs\"

2\) Truth in Negotiations Act (TINA)

3\) FAR 15.800, Price Negotiation

4\) DFARS 215.811, \"Estimating Systems\"

Cost Accounting Standards (CAS) provides guidance in accounting for
contract costs at larger contractors. CAS 401 requires that a
contractor\'s estimating practices be consistent with those governing
the accumulation and reporting of costs during contract performance.
Some contractors see parametrics as being inconsistent with CAS 401.
Contractors must ensure both cost and noncost information used in
estimating is separately accumulated and reported as required by CAS
401.

The purpose of the Truth in Negotiations Act, 10 U.S.C. 2306(f), is to
provide the government with all facts available to the contractor at the
time it certified the cost or pricing data was accurate, complete, and
current. Parametric estimates must meet the same basic disclosure
requirements under the act as discrete estimates. Although the
principles are no different, proposals supported in whole or in part
with parametric estimating will have different types of cost or pricing
data than traditional discrete cost estimates.

Fundamental to the definition of cost or pricing data are \"all facts
\... which prudent buyers and sellers would reasonably expect to have a
significant effect on price negotiations\" (FAR 15.801). Reasonable
parallels may be drawn between the data examples provided in FAR for
discrete estimating approaches and the type of data pertinent to
parametric estimating approaches. The contractor is also expected to
provide all factual data for the parametric cost estimates. This data
must be accurate, complete, and current.

Many contractors use parametric cost estimating for supplementary
support or validation of estimates developed using other methods. This
requires judgment in selecting which data will be used in developing the
total cost estimate relied upon for the price proposal. In
distinguishing between fact and judgment, FAR states the certificate of
cost or pricing data \"does not constitute a representation as to the
accuracy of the contractor\'s judgment on the estimate of future costs
or projections. It does apply to the data upon which the contractor\'s
judgment or estimate was based\" (FAR 15.804‑4b). Thus, if a contractor
develops a proposal using both parametric data and discrete estimates,
it would be prudent to disclose all pertinent facts to avoid later
questions about completeness of the submission.

Auditors are also required to evaluate estimating systems of major
Department of Defense (DoD) contractors. This includes ensuring that, if
parametric estimating procedures are part of the estimating system, they
are properly disclosed. Of key concern to the auditor in evaluating the
estimating systems\' disclosure of parametric procedures are the
following:

1\) Do the procedures clearly establish guidelines for when parametric
techniques would be appropriate?

2\) Are there guidelines to ensure the consistent application of
estimating techniques?

3\) Is there proper identification of sources of data and the estimating
methods and the rationale used in developing cost estimates?

4\) Do the procedures ensure that relevant personnel have sufficient
training, experience, and guidance to perform estimating tasks in
accordance with the contractor\'s established procedures?

5\) Is there internal review of and accountability for the adequacy of
the parametric estimating techniques, including the comparison of
projected results to actual results and an analysis of any differences?

DCAA believes parametric estimating approaches are acceptable when they
are properly implemented. Auditors encounter it most often as a
technique used in conjunction with other estimating methods. For
example, parametrics are often used for estimating costs of scrap and
other such factors. The majority of the proposals audited are not
developed solely based on parametric estimating techniques. The use of
parametric estimating is most appropriate in such circumstances where
historical data is not available as when the program is at the
engineering concept stage, or when no bill of materials exists and the
program definition is unclear. Contractors with good parametric cost
estimating systems analyze their perspective proposal to determine the
appropriate estimating technique for each part of the work breakdown
structure.

**Observations and Suggestions**

The contractor can consider some observations made by DCAA auditors as
to the pitfalls contractors fall victim to when employing parametric
techniques. The first is when a contractor fails to do a cost‑benefit
analysis before implementing an elaborate parametric estimating model.
Key questions for any contractor considering implementing a complex
parametric model are:

1\) How often can we reasonably expect to use it?

2\) How much time can we expect to save?

3\) What are the costs of maintaining the model?

4\) Will the model produce the necessary precision?

Contractors should be satisfied that implementation and monitoring costs
do not outweigh the benefit of reduced estimating costs. Moreover, it is
critical that the environment is appropriate for the use of parametrics.
It would not be prudent to rely exclusively on parametric techniques to
estimate costs when directly applicable historical cost data are
available. Such is the case of follow‑on production for the same or
similar hardware. Contractors manufacturing mature weapon systems
already have a record of the actual costs. The use of parametric
estimating may be appropriate for certain aspects of follow‑on
production, however, the contractor should disclose any data that may
have a significant impact on cost. The exclusive use of parametrics is
generally not appropriate for economic forecasting of such elements as
labor and indirect cost rates. For parametric estimates, the contractor
must ensure that any changes in accounting practices are accounted for
in the estimate and that labor and indirect cost rates are appropriately
applied.

Another problem encountered is contractors failing to properly disclose
their parametric estimating practices. Auditors have experienced
instances where the first time a parametric model is disclosed is during
the evaluation of the proposal. This is often too late. As mentioned
earlier, larger DoD contractors have an obligation to disclose in
writing their estimating procedures. Making proper and timely disclosure
will minimize problems and expedite the negotiation process.

Contractors can take the lead in helping to streamline the oversight
process. In a few words, they should practice self‑governance! DCAA has
been a leading proponent of the self‑governance program. Self‑governance
is intended to encourage contractors to establish and maintain good
systems of internal control in key areas, including estimating systems.
It requires contractors to provide their own oversight \-- to detect
system weaknesses and take corrective action as necessary. This
initiative recognizes that prudent contractors already have the means in
place to ensure their operations are efficient and cost effective.

DCAA has another initiative that contractors should consider as a part
of streamlining the oversight process. It is called \"coordinated audit
planning.\" DCAA defines coordinated audit planning as a voluntary
process wherein the DCAA auditor and the contractor\'s internal and
external auditors consider each other\'s work in determining the nature,
timing, and extent of his or her own auditing procedures. Coordinated
audit planning considers the extent to which reliance can be placed upon
work performed by the other auditor to minimize duplication of audit
effort. In addition, this process strengthens the evaluation of internal
control systems.

Understanding estimating systems controls, assessing risk, and
transaction testing are common objectives of DCAA, and the internal and
external auditors associated with contractors. This often results in
duplicative audit procedures. In the coordinated audit planning
conducted to date, DCAA found that the instances of duplication are
significant. DCAA auditors are more than willing to rely on the work
done by contractor internal and external auditors, providing DCAA has
the opportunity to evaluate and test their work.

Contractors are expected to establish and maintain reliable estimating
systems. Departmental procurement officials, Members of Congress, and
the average American citizen hold Defense contractors to high and
exacting standards. The funds involved can be enormous.

The expectations are equally high for the auditor whose job it is to
protect the taxpayer\'s interest. The DCAA auditor must comply with the
American Institute of Certified Public Accountants\' Statements on
Auditing Standards and the GAO\'s Government Auditing Standards. These
standards require that the auditor be independent in fact and in
appearance. They also require the auditor exercise a healthy degree of
professional skepticism. These requirements, however, should not render
the DCAA auditor and the contractor enemies. Such polarized and
adversarial relationships are dysfunctional and not in either party\'s
best interest.

Both parties are taking significant actions to improve relationships.
These changes are producing a culture change that is very positive:
positive because contractors are beginning to more fully accept their
responsibilities; positive because auditors are more effectively
communicating audit plans and objectives.

Defense procurement is taking on a new, streamlined look in the 1990s.
Government and industry are both concerned with quicker, less costly
means of procuring goods and services. This is one reason why parametric
methods continue to stir up so much interest. Parametric techniques,
properly applied, can assist contractors and the government alike in
streamlining the acquisition process. In addition, the ability of the
government to place greater reliance on contractor oversight of
contractor systems will also result in meeting procurement needs more
timely.

**Summar*y***

In today\'s and tomorrow\'s procurement environment, a great challenge
facing us all is the development of a cooperative work climate conducive
to quickly acquiring quality products and services at fair and
reasonable prices. New estimating techniques such as parametrics can cut
estimating costs.

Adequate estimating systems, fully supported and self‑governed by
industry, can cut audit costs. Quick estimating and audit turnaround
times can cut procurement costs. All of this, however, requires
communication and teamwork. Whether our buying effort is for innovative
space equipment or for recurring maintenance, we must all work to meet
the challenge.

Mike Thibault\'s message to the parametric community is clear:
Parametric techniques are acceptable cost estimating methodologies given
that the following five criteria exist: logical relationships,
verifiable data, a significant statistical relationship (correlation)
exists, techniques produce accurate predictions, and they are easy to
monitor and support.

He goes on to say that the auditor needs to consider the adequacy of the
parametrics cost estimating system and related internal controls, which
includes: the audit trail, sufficiency of documentation, currency and
sources of data, procedures for calibration and validation, and the
appropriateness of parametrics use. Since historical data is normally
used as the basis for all estimating, the auditor will use basic
auditing techniques to verify that costs are current, accurate, and
complete. Mr. Thibault completely dispels the myth that the audit
community would not accept the use of appropriate parametric cost
estimating techniques for firm business proposals.

DCAA\'s complete audit guidance is included in its Contract Audit
Manual, Chapter 9‑1000. This chapter is included in its entirety in
Appendix C.

**[ESTIMATING SYSTEM REVIEWS]{.underline}**

As part of a regulatory oversight requirement, DCAA will periodically
perform contractor estimating system reviews. The intent is to review
the requirements delineated in DFARS 215.811 and 252.215‑7003. DFARS
215.811 requires all DoD contractors to have adequate estimating
systems, requires certain large businesses to disclose their estimating
systems in writing, provides guidelines concerning the characteristics
of an adequate estimating system, and provides guidance for team
estimating system reviews. If a contractor is required to disclose their
system, all significant parametric cost estimating techniques need to be
disclosed.

It is DoD policy that contractors have estimating systems that
consistently produce well supported proposals acceptable as a basis for
negotiating fair and reasonable prices. Estimating systems should be
consistent and integrated with a contractor\'s related management
systems, and be subject to applicable financial control systems. To be
considered adequate, an estimating system must be established,
maintained, reliable, and consistently applied. It must also produce
verifiable, supportable and documented cost estimates.

FAR 15.8 provides the criteria for submission of cost or pricing data.
Although FAR 15.8 does not mention parametric cost estimating
techniques, there are no restrictions that preclude the use of these
techniques for firm business proposal submissions. FAR 15.804‑6, Table
15‑2, does require the offeror to submit any information reasonably
required to explain the estimating process. This means that the
contractor should clearly describe all parametric cost estimating
techniques and provide support for the data used in those techniques.

The FAR 15.804‑6, Table 15‑2, also states that for material cost
estimates, the contractor shall provide a consolidated summary of
individual material quantities (e.g., bill of material) included in the
various tasks being proposed. In some cases it is not feasible for a
contractor to provide a consolidated bill of material or parametrics
results in a better and more supportable estimate. For example, in the
research and development phase of a program, the material requirements
often times have not yet been defined, and, as a result, the preparation
of a consolidated priced bill of material is often not possible. In
these circumstances, parametric cost estimating techniques may provide
for a reasonable basis to estimate material costs in lieu of a
consolidated priced bill of material.

In any case, when a offeror uses estimating techniques other than a
consolidated priced bill of material to estimate material costs, the
offeror must adequately describe the techniques being used, provide
sufficient support to allow for an independent evaluation, and explain
why the techniques used are the best in the circumstances in order to
comply with the material cost criteria included in Table 15‑2.

DFARS 215.811‑70 delineates attributes of an adequate estimating system.
These are:

\(1\) Establishes clear responsibility for the preparation, review, and
approval of cost estimates.

\(2\) Provides a written description of the organization and duties of
personnel responsible for \... contributing to the estimating process
\...

\(3\) Ensures that relevant personnel have sufficient training,
experience and guidance \...

\(4\) Identifies sources of data and the estimating methods and
rationale used in developing cost estimates.

\(5\) Provides for appropriate supervision \...

\(6\) Provides for consistent application of estimating techniques.

\(7\) Provides for detection and timely correction of errors.

\(8\) Protects against cost duplication and omissions.

\(9\) Provides for the use of historical experience, including vendor
pricing information where appropriate.

\(10\) Requires use of appropriate analytical methods.

\(11\) Integrates information available from other management systems as
appropriate.

\(12\) Requires management review \[of the estimating system\]

\(13\) Provides for internal review of and accountability for the
adequacy of the estimating system, including the comparison of projected
results to actual results and an analysis of any differences.

\(14\) Provides procedures to update cost estimates in a timely manner.

\(15\) Addresses responsibility for review and analysis \... of
subcontract prices.

No well supported and documented parametric estimating system should be
threatened by any audit requirements, FAR or DFAR characteristics, or
any other compliance (Truth in Negotiation Act) issue. A well calibrated
and validated parametric estimating system will be compliant in all
respects. The DFARS goes on to list indicators of or conditions that may
cause significant estimating deficiencies (from 215.811‑77). Three in
particular stand out:

\(1\) Failure to ensure that relevant historical experience is available
to and utilized by cost estimators, as appropriate.

\(2\) Consistent absence of analytical support for significant proposed
cost amounts.

\(3\) Excessive reliance on individual personal judgment where
historical experience or commonly used standards are available.

The DFARS emphasis on historical experience is particularly satisfied by
well supported parametric systems. After effective calibration
activities, parametric estimates are developed that are auditable,
compliant with regulations, and suitable for the negotiation of fair and
reasonable prices between government and contractor.

[**FORWARD PRICING RATE AGREEMENTS (FPRAs**)]{.underline}

An FPRA, as defined in FAR 15.801, is a written agreement negotiated
between a contractor and the government to make certain rates and
factors available during a specified period for use in pricing contracts
or contract modifications. Such rates and factors represent reasonable
projections of specific costs that are not easily estimated for,
identified with, or generated by a specific contract, contract end item
or task. On the other hand, a Forward Pricing Agreement (FPA) is a
written agreement between a DoD contracting office and a large volume
contractor which sets forth a methodology that the contractor agrees to
follow when pricing items covered by the FPA. It differs from an FPRA in
that once established, the FPA may be used to determine the complete
final price of individual orders. A typical FPA, for example, may be
established to cover and expedite the acquisition of spares.

It is clear that any CER or parametric methodology could fall under the
umbrella of an FPRA or an FPA. A forward pricing factor is generally
represented as a percentage or ratio that is applied to an existing cost
determination or estimate in order to arrive at another, usually
related, cost determination or estimate. A costing rate used to convert
labor hours to dollars is just such a factor and is truly a CER.
Parametric costing is done all the time. An FPRA for a parametric model
is a natural next step.

Therefore, when repetitive use of single CERs or parametric cost models
is envisioned by the contractor, an FPRA should be established.
Procedures for establishing such a FPRA should follow the basic
guidelines listed in FAR Part 15.809‑3 and should include the following:

a\. Timely submission by the contractor, at least yearly, of an
adequately supported FPRA proposal.

b\. Requirement for clear definitions of all elements of cost (dependent
variables) which will be developed by the CERs included in the FPRA and
the applicable bases (independent variables). Additionally, the proposal
must indicate the period of applicability of the proposed CERs and the
conditions under which the CERs will be used in price proposals.

c\. The proposed CER\'s or parametric cost models should be evaluated in
terms of the ten criteria listed in the Table VI‑1. The answer to the
majority of these questions should be affirmative ‑ to support the need
for a CER or Cost Model.

d\. Tracking of the negotiated CERs and parametric cost models included
in the FPRA is required to test their validity as estimating tools.
Guidelines for monitoring FPRAs are included in Part 15.809‑5 and the
DCAA Contract Audit Manual (Vol. 1, Chapter 9‑200).

It is the contractor\'s responsibility to identify the rate(s) to be
included under the umbrella of the FPRA or FPA and to specify the latest
data already submitted in accordance with the agreement. All data
submitted in connection with the agreement, updated as necessary, form a
part of the total data that the offeror certifies to be current,
accurate, and complete at the time of agreement on price for an initial
contract or for a contract modification.

**TABLE VI-I**

**[CRITERIA]{.underline}** **[QUESTIONS TO ANSWER]{.underline}**

Cost Benefit Is it more cost beneficial to have a CER for the cost
element than to generate a discreet estimate?

Predicts Well Do you anticipate the CER or parametric cost model to be a
good predictor of actual and reasonable cost?

Supportability Is the data that supports the development of the CER
auditable and/or traceable? (This can be from official accounting data
or other supporting data (e.g., subcontractor files)

Usability Can the CER be applied within the bounds of the Estimating
System?

Customer Acceptance Is the CER accepted by the PCO, DPRO, the CER owner,
etc.?

Management Acceptance Does the CER satisfy the contractor\'s
requirements?

Ownership Can a group/individual be identified who would provide a
discrete input if the element of cost was not estimated within the CER?

Applicability What is the intent for application of this CER or
parametric cost model (e.g., changes, restructures, etc.)?

Effective Implementation Are the persons doing the modeling/ analysis
sufficiently trained and experienced to use the tools correctly?

**[WHAT TO LOOK FOR IN A PARAMETRIC ESTIMATE]{.underline}**

With the proliferation of parametric cost estimating models and tools,
both commercial models and "home grown" versions, it is impossible to
describe what to look for in every model and cost parameter. However,
some generalizations can be made. The following is a summary, since most
of this has already been covered in the Handbook.

As a cost estimator or analyst, knowing and understanding the audit
guidance described in this chapter is a good start. If you follow the
steps delineated earlier in this chapter, the cost analysis or audit
battle is more than half won.

If, as an analyst, you are confronted with a parametric cost estimate,
you should take a few steps to ensure a fair review. These are :

-   Understand the cost model used. Do not hesitate to ask questions and
    to consult with the experts. Most commercial model builders welcome
    calls from users, analysts and auditors. You do not have to have a
    user's understanding of the model, just a general knowledge about
    how it works. Remember, there's no such thing as a stupid question.

-   Review the program inputs to the model. Is the schedule correct?
    Does the WBS adequately describe the product being estimated? Is
    anything missing? Are there duplications? Does the WBS follow the
    statement of work?

-   Review the technical inputs to the model with your own engineers or
    technical department, and your program office. Check them for
    reasonableness and benchmark them against your own experience and
    the experience of the resident experts.

-   Understand the model's cost drivers. Generally, there are a few
    select parameters that are the predominate cost estimating factors
    that drive cost. Many of the others are just "tweaks" to the model.
    Concentrate on the cost drivers when performing your analysis. Your
    may wish to combine the other minor "environmental" factors together
    as one factor.

-   Be aware of the assumptions the cost estimator made when he/she
    built the model. Are they reasonable? Would you have made the same
    assumptions? If not, why not?

-   Be knowledgeable of the historical cost basis for the model, if any.
    Be sure to review the source documentation. Be wary of any model
    used that has no basis in historical cost. How was the data "tuned"
    or normalized? Were data outliers disregarded? If so, why? Was the
    model calibrated? How was the calibration performed? Were any
    universal or "standard" cost factors used? Would they be applicable
    in this case?

-   Question how the future environment might be different from the
    historical case. Have these differences been accounted for? Would
    you account for them differently? If so, how?

-   Is the estimated program expected to push the state of the
    technology (state of the art)? How is this planned to be
    accomplished? How is it accounted for in the estimate? Is it
    seriously considered, or glossed over? Advances in the state of the
    art normally have significant impacts on the cost drivers. Does this
    show up in the parametric estimate, or are the historical factors
    used without adjustment? It is important to use technical experts on
    this item. For instance, if a contractor proposes using a technology
    that has yet to be proven in the laboratory, program trouble may lie
    ahead and result in inadequate estimates. Consult with your
    technical people on this and related technical concerns.

-   Review the track record of the estimator or the estimating
    organization. What is their past performance? Have their estimates
    been reliable?

-   Understand the economics involved with the model. What are the
    effective costing rates used? Are they reasonable? Do they reflect
    the organization and skill levels being estimated?

-   Identify what cost factors have been "tweaked" and why. Focus on the
    "big ticket" items. Using expert opinion and "rules of thumb," are
    any significant cost factors outside the range of reasonableness?
    For instance, it is very easy to calibrate a software cost model's
    cost (hours or dollars) per line of software code. Is the CER
    reasonable for the estimate? However, different models may define a
    software line of code differently. It is therefore vital to
    understand the definitions used in the model you are evaluating. If
    the model violates your concept of reasonableness, then question it.

-   Last, don't be intimidated. Most (especially commercial) models are
    quite user friendly and the model builders are helpful. Other people
    have learned how to use them, and you are as smart as they are. The
    most difficult part is learning the language and gaining the
    experience over time to develop your skills.

**Rules Of Thumb**

Rules of thumb are general rules, CER's or other cost factors that are
widely used, but do not apply precisely to and specific case. Each
estimating organization will have its own peculiar set. Most long time
estimators can develop a rapid estimate (a ROM) based on their own
personal rules of thumb. These rules come from their own experience.
Examples of rules of thumb are the "default" factors that populate many
parametric models including the commercial ones. If no factor input is
available, the model uses a universally derived factor - one taken from
a universal data base or expert opinion.

Although there is nothing wrong with the use of these rules per se, the
danger is evident. The rules are only norms or benchmarks, and they will
never apply to a specific estimate. Their value lies in the comparison
of a factor in a model to the universal case. Too much deviation should
be investigated. Why is the actual case so different from the rule? Or,
stated differently, why doesn't the rule apply here?

For instance, if you believe that a line of software code (for a
particular type of software) should take about one-half hour to write
(on average), and the parametric model indicates two hours, then some
investigation is in order. Although either CER could be the correct one,
there is far too much deviation from the norm, and suspicions are
aroused. However, if the model indicates six tenths of an hour that
compares to your one-half hour, the rule had performed a "sanity" check
for you. But the use of the rule does not preclude a thorough cost
analysis if you have identified a cost driver.

Rules of thumb are important, but they must be used wisely. Develop your
own list. This can be accomplished through experience, or by consulting
with experts in the field. Many rules exist embedded within commercial
parametric models, and are available to the users and modelers. In any
event, it is important to have a list of benchmark factors. You'll
probably need a different list for each organization you deal with.
Organizational and product differences will demand it. But remember that
there are no shortcuts or magic formulas, so use your rules as a quality
check, or for quick, rough order of magnitude estimates.

**An Example**

What follows is an example of some of the above discussion. The example
here uses the software cost estimating model, REVIC.

The fundamental REVIC equation is: Man Months = A\*(KDSI)^B^\*ΠFi,
where,

A and B are coefficients of several Software Development Modes (for
instance, embedded , organic, Ada);

KDSI is Thousand Delivered Source Instructions; and;

ΠFi is the product of various environmental factors the model uses to
adjust the cost estimates. These environmental factors include, for
example: analyst and programmer capabilities, applications experience,
programming language, storage constraints, requirements volatility,
reliability requirements, data base size, product complexity, the use of
modern programming practices and software tools, platform (airborne,
space, etc.), schedule compression, etc. Each factor is given a factor
value from very low through nominal to high.

Suppose you were given the following cost estimate:

REVIC = IMS EP 1 = 3.312(DSI/1000)^1.2^\*ΠFi

IMS EP 1 = (3.312)(1337/1000)^1.2^\*.874 = 4.1MM

What would you do with it? The basis of estimate (BOE) tells you this:
"The Integrated Management System Evaluation Package for Release 1 (IMS
EP 1) requires 1337 new lines of embedded source code of new algorithms
and uses complex interfaces. Our superior programming staff using
state-of-the-art software tools, modern programming practices, and
possessing significant application experience allows the product of the
environmental factors to be less (13%) than nominal." What do you notice
about this estimate?

First, this particular estimate is relatively small (4.1MM). You may
wish to move on to the more important cost issues. However, you decide
to "spot check" this one. You know very little about REVIC, so you turn
to the resident expert for advice. He/She confirms that the technical
description in the BOE is correct, so the model's values for A and B are
the correct ones. You make a quick calculation to compare the estimate
with a rule of thumb supplied by your resident expert.

> 4.1MM \* 155hrs./MM 1337DSI = .475hrs/DSI

Your resident expert informs you that he believes this factor to be
quite low, possibly two times. He also indicates that the DSI number
needs to be verified for accuracy, since it is a major cost driver for
this estimate. He is also suspicious of the "\....less (13%) than
nominal" environmental factors product. (Nominal here he believes should
equal at least 1.5 due to a partial space platform involved.) Therefore
13% appears to be a mistake of some kind, but it will have to be
investigated. He also argues that since this is a competitive proposal
that's almost 80% software, all software estimates may be "aggressive,"
and should be carefully reviewed. The model does not appear to have been
calibrated except for the vague references "\....superior programming
staff\...." in the BOE. The product of the environmental factors seems
to be based upon judgment, although likely an educated one.

You decide to carefully review the other REVIC cost estimates in the
proposal, so you borrow a REVIC User's Guide in order to become more
proficient.

Although this is a relatively easy example, it demonstrates the general
idea of what to look for in a parametric estimate.