-----------------------------------------------------------------------
  **Module Title**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ---------------------
  ![](media/image1.wmf){width="3.9506944444444443in"   Slide \# 7-1
  height="2.9680555555555554in"}                       

  ---------------------------------------------------- ---------------------

**Module 7: Performance-Based Agreements**

**Module Outline**

  ------------------------------------------------- --------- ---------------------
  Topic                                                       Time Frame

  Introduction/Objectives                           5 minutes 

  Elements of a Performance-Based Agreement (PBA)   120       
                                                    minutes   

  Budget Analysis                                   120       
                                                    minutes   

  Review                                            5 minutes 

  **Estimated/Approximate Time for Module           **4       
  Completion**                                      hours**   
  ------------------------------------------------- --------- ---------------------

  -----------------------------------------------------------------------
  **Topic 1: Introduction/Objectives**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image2.wmf){width="3.9506944444444443in"   Slide \# 7-2
  height="2.9680555555555554in"}                       

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This module will discuss the recognition of the forms of cost         |
| estimating, cost analysis, reconciliation of cost estimates,          |
| financial planning, formulating financial programs and budgets,       |
| budget analysis/execution, benefit-cost analysis, earned value        |
| management (EVM), and other methods of performance measurement        |
| required for a performance-based agreement (PBA).                     |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Introduction/Objectives (continued)**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image4.wmf){width="3.9506944444444443in"   Slide \# 7-3
  height="2.9680555555555554in"}                       

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| > Briefly review the module agenda by introducing the topics that     |
| > will be covered. Topics include:                                    |
|                                                                       |
| Topic 2: Elements of a Performance-Based Agreement (PBA)              |
|                                                                       |
| Topic 3: Budget Analysis                                              |
|                                                                       |
| Topic 4: Review                                                       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Introduction/Objectives (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image6.wmf){width="3.9506944444444443in"   Slide \# 7-4
  height="2.9680555555555554in"}                       

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| [Objectives]{.underline}                                              |
|                                                                       |
| At the end of this module, you will be able to:                       |
|                                                                       |
| -   Explain how to appropriately administer the establishment of a    |
|     negotiated baseline of performance for an acquisition.            |
|                                                                       |
| -   Describe the negotiations for the required level of support at a  |
|     cost consistent with available support funding.                   |
|                                                                       |
| (continued on the next slide)                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Introduction/Objectives (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image7.wmf){width="3.9506944444444443in"   Slide \# 7-5
  height="2.9680555555555554in"}                       

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| [Objectives (continued)]{.underline}                                  |
|                                                                       |
| At the end of this module, you will be able to:                       |
|                                                                       |
| -   Explain the application of management actions required of program |
|     managers who are involved in acquisition of services.             |
|                                                                       |
| -   Define the different methods of performance measurement as part   |
|     of program management.                                            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Introduction/Objectives (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the introduction and objectives After     |
| addressing any questions the participants may have, inform them that  |
| we will be moving on to the next topic: Elements of a                 |
| Performanced-Based Agreement.                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image8.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  the introduction and objectives?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image9.wmf){width="3.9506944444444443in"   Slide \# 7-6
  height="2.9680555555555554in"}                       

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| As presented in detail in FAI Government Specific Courses I and II,   |
| to be considered performance-based, an acquisition should have, at a  |
| minimum, four elements. This topic serves as a review of these        |
| elements.                                                             |
|                                                                       |
| The first element an acquisition must have to be considered a PBA is  |
| a **Performance Work Statement**. This element can be considered      |
| "defining the problem."                                               |
|                                                                       |
| The second element an acquisition must have to be considered a PBA is |
| **Measurable Performance Standards**:                                 |
|                                                                       |
| -   Define what is considered acceptable performance                  |
|                                                                       |
| -   Determine whether performance outcomes have been met              |
|                                                                       |
|     (continued)                                                       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| (continued from previous page)                                        |
|                                                                       |
| The third element an acquisition must have to be considered a PBA is  |
| **Remedies and Incentives**:                                          |
|                                                                       |
| Procedures that address how to address performance that does not meet |
| required performance standards                                        |
|                                                                       |
| Incentives may be used to encourage performance that will exceed      |
| minimum performance standards                                         |
|                                                                       |
| -   Not mandatory                                                     |
|                                                                       |
| -   Incentives and remedies compliment each other                     |
|                                                                       |
| The final element an acquisition must have to be considered a PBA is  |
| a **Performance Assessment Plan**:                                    |
|                                                                       |
| Describes how contractor performance will be:                         |
|                                                                       |
| -   Measured                                                          |
|                                                                       |
| -   Assessed                                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly review and discuss the benefits |
| of a PBA acquisition.                                                 |
+-----------------------------------------------------------------------+

![](media/image8.png){width="0.49930555555555556in"
height="0.48819444444444443in"} What are some benefits of a PBA?

Answer:

There are many benefits, including:

-   Increased likelihood of meeting program (mission) needs

-   Focus on intended results, not process

-   Better value and enhanced performance

-   Shifts the performance risk to the contractor

-   Contractor flexibility in proposing solutions

-   Shared effectiveness permitting innovation and cost effectiveness

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The benefits of a PBA can be further articulated in specific          |
| objectives that can be achieved by describing requirements as         |
| performance outcomes in the acquisition. These objectives include:    |
|                                                                       |
| Maximize Performance                                                  |
|                                                                       |
| Maximize Innovation and Competition                                   |
|                                                                       |
| Encourage and Promote Use of Commercial Services                      |
|                                                                       |
| Shift Performance Risk to Contractor                                  |
|                                                                       |
| Potential Savings                                                     |
|                                                                       |
| Let us consider each of these objectives in more detail.              |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The PBA objective, **Maximize Performance**, accomplishes the         |
| following benefits for the program manager:                           |
|                                                                       |
| Allows contractor to deliver the required service by following the    |
| contractor's own best practices                                       |
|                                                                       |
| Prime focus is the end result                                         |
|                                                                       |
| -   Contractors can adjust their processes, as appropriate,           |
|     throughout the life of the contract without being burdened with   |
|     contract modifications provided the delivered service (outcome)   |
|     remains in accordance with the contract                           |
|                                                                       |
| -   Use of incentives provides motivation for contractors to provide  |
|     best performance possible                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The PBA objective, **Maximize Competition and Innovation**,           |
| accomplishes the following benefits for the program manager:          |
|                                                                       |
| Encourages innovation from the supplier base by using performance     |
| requirements                                                          |
|                                                                       |
| -   Maximizes opportunities for competitive alternatives vs.          |
|     agency-directed solutions                                         |
|                                                                       |
| -   Attracts a broader industry base                                  |
|                                                                       |
| The PBA objective, **Encourage and Promote the Use of Commercial      |
| Services**, accomplishes the following benefits for the program       |
| manager:                                                              |
|                                                                       |
| Use of commercial procedures for services provides great benefits     |
|                                                                       |
| -   Minimizes the reporting burden                                    |
|                                                                       |
| -   Reduces use of government-unique contract clauses and similar     |
|     clauses                                                           |
|                                                                       |
| -   Also helps attract a broader industry base                        |
|                                                                       |
| The PBA objective, **Shift the Performance Risk to the Contractor**,  |
| accomplishes the following benefits for the program manager:          |
|                                                                       |
| Makes contractor responsible for achieving the objectives in the work |
| statement                                                             |
|                                                                       |
| Contractor achieves objectives through use of own best practices and  |
| processes                                                             |
|                                                                       |
| The final PBA objective, **Potential Savings**, accomplishes the      |
| following benefit for the program manager:                            |
|                                                                       |
| Use of performance requirements has been proven to result in          |
| demonstrated cost savings during the life of a program                |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Now that we've reviewed the elements of a PBA and the corresponding   |
| objectives and benefits for this acquisition strategy, let's consider |
| in more detail how the first element, Performance Work Statement, is  |
| crucial to the PBA. The PWS essentially "defines the problem" to be   |
| solved by the acquired services.                                      |
|                                                                       |
| The program manager (and other technical specialists, possibly)       |
| serves as the principle technical expert for the acquisition. He/she  |
| is usually the most familiar with the requirement and best able to    |
| identify technical trade-offs, as well as determine whether the       |
| requirement can be met with a commercial solution.                    |
|                                                                       |
| The program manager is typically responsible for preparing the        |
| Performance Work Statement or the Statement of Objectives.            |
|                                                                       |
| Development of the Performance Work Statement (or Statement of        |
| Objectives) requires careful consideration and analysis of the        |
| financial aspects of the acquisition.                                 |
|                                                                       |
| These financial aspects are covered in the next topic of this module. |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| Next, let's consider some other elements of a PBA as they relate to   |
| the VCF Acquisition Program Management Business Case .                |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image10.png) **Activity:**                                  |
|                                                                       |
| **We need to have the groups select a point in time and then answer   |
| the questions.**                                                      |
|                                                                       |
| Activity Purpose: This activity provides participants with an         |
| opportunity to analyze the VCF Acquisition Program Management         |
| Business Case relative to two PBA elements: Measurable Performance    |
| Standards and Remedies and Incentives.                                |
|                                                                       |
| (*This module activity continues the business case that will be used  |
| throughout this course to apply various concepts and reinforce        |
| learning through the use of critical thinking and application         |
| skills.*)                                                             |
|                                                                       |
| Activity Length: 60 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Divide participants into four groups (or the number of groups     |
|     required to keep size equal to or less than five participants per |
|     group). Provide each group with a flip chart and markers.         |
|                                                                       |
| 2.  Allow participants 5 - 10 minutes to individually review the *VCF |
|     Acquisition Program Management Business Case* found in their      |
|     Participant Guide and on the following pages of this Instructor   |
|     Guide.                                                            |
|                                                                       |
| 3.  Assign half the groups "Measurable Performance Standards" and the |
|     other half "Remedies and Incentives."                             |
|                                                                       |
| 4.  Have the small groups answer the following questions relative to  |
|     their assigned PBA element: Measurable Performance Standards."    |
|     Groups should record their repsonses on the flip chart for        |
|     presentation to the whole group.                                  |
|                                                                       |
|     a.  You are the program manager for this acquisition. What are    |
|         some of the decision points you might use to identify         |
|         measurable performance standards?                             |
|                                                                       |
|     b.  How would you utlilize your team to develop these standards?  |
|                                                                       |
|     c.  What types of methods would you use for measurement of the    |
|         standards?                                                    |
|                                                                       |
|     d.  Where do you feel the actual program manager (and/or team)    |
|         failed with regard to performance standards for this          |
|         acquisition?                                                  |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image11.png) **Activity:(continued)**                       |
|                                                                       |
| Activity Instructions (continued):                                    |
|                                                                       |
| 5.  Have the small groups answer the following questions relative to  |
|     their assigned PBA element: Remedies and Incentives." Groups      |
|     should record their responses on the flip chart for presentation  |
|     to the whole group.                                               |
|                                                                       |
|     a.  You are the program manager for this acquisition. What are    |
|         some of the decision points you might use to identify         |
|         remedies and incentives?                                      |
|                                                                       |
|     b.  How would you utilize your team to develop these remedies and |
|         incentives?                                                   |
|                                                                       |
|     c.  What types of methods would you use for determining           |
|         applicability?                                                |
|                                                                       |
|     d.  Where do you feel the actual program manager (and/or team)    |
|         failed with regard to remedies and incentives for this        |
|         acquisition?                                                  |
|                                                                       |
| 6.  Have a spokesperson for each of the small groups present the      |
|     findings to the whole group.                                      |
|                                                                       |
| 7.  Use the answers given to generate a large group discussion to     |
|     elicit other opinions about the question. There may be more than  |
|     one correct answer or viewpoint for each question. **Note:**      |
|     *Relevant answers for each question are found on the following    |
|     page of this Instructor Guide.*                                   |
|                                                                       |
| 8.  After all the questions have been discussed, summarize the        |
|     activity by telling participants that there is not necessarily a  |
|     "right and wrong" when managing a program. The important aspect   |
|     is to make sure that the program manager exercises due diligence  |
|     and considers the applicability of the program management         |
|     components taught as part of this topic.                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[Introduction: FBI Shares the Blame]{.underline}

The VCF was supposed to automate the FBI\'s paper-based work
environment, allow agents and intelligence analysts to share vital
investigative information, and replace the obsolete Automated Case
Support (ACS) system. Instead, the FBI claims, the VCF\'s contractor,
Science Applications International Corp. (SAIC), delivered 700 000 lines
of code so bug-ridden and functionally off target that the bureau had to
scrap the \$170 million project, including \$105 million worth of
unusable code. However, various government and independent reports show
that the FBI shares the blame for the project\'s failure.

[The Bureau Environment: Lots of Paper]{.underline}

The bureau\'s 12,400 agents work out of 56 field offices and 400
satellite - or resident agency - offices, as well as 51 Legal Attaché
offices scattered across the globe in U.S. embassies and consulates.
Field agents interview witnesses, develop informants, conduct
surveillance, hunt for clues, and collaborate with local law enforcement
to find and arrest criminals. Agents document every step and
methodically build case files. They spend a tremendous amount of time
processing paperwork, faxing and FedEx-ing standardized memo and
requisition forms through the approval chain. Once the appropriate
supervisors sign off on the form, it goes back to the agent, who gives
it to a clerk to enter into the ACS system. From there, the paper form
is filed as part of the official record of the case. This system of
forms and approvals stretches back to the 1920s.

[July 2000: Trying to Bring Order to Chaos]{.underline}

According to the DOJ\'s Office of the Inspector General, the FBI had
13,000 computers that could not run modern software. Most of the 400
resident agency offices were connected to the FBI intranet with links
about the speed of a 56-kilobit-per-second modem. Many of the bureau\'s
network components were no longer manufactured or supported. And agents
couldn\'t e-mail U.S. Attorney offices, federal agencies, local law
enforcement, or each other; instead, they typically faxed case-related
information.

\[1\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[July 2000: Trying to Bring Order to Chaos (continued)]{.underline}

In 2000 the bureau finally began to deal with its outdated IT systems.
At the time, under the direction of Louis J. Freeh, the bureau had
neither a CIO nor documentation detailing its IT systems, much less a
plan for revamping them. The task of creating such a plan fell to former
IBM executive Bob E. Dies, who became assistant director in charge of
the FBI Information Resources Division on 17 July 2000. He was the first
of five officials who, over the next four years, would struggle to lead
the FBI\'s sprawling and antiquated information systems and get the VCF
project under way.

In September 2000, Congress approved \$379.8 million over three years
for what was then called the FBI Information Technology Upgrade Project.
Eventually divided into three parts, the program became known as
Trilogy. The Information Presentation Component would provide all 56 FBI
field offices, some 22,000 agents and support staff, with new Dell
Pentium PCs running Microsoft Office, as well as new scanners, printers,
and servers. The Transportation Network Component would provide secure
local area and wide area networks, allowing agents to share information
with their supervisors and each other.

But the User Applications Component, which would ultimately become the
VCF, staked out the most ambitious goals. First, it was to make the five
most heavily used investigative applications---the Automated Case
Support system, IntelPlus, the Criminal Law Enforcement Application, the
Integrated Intelligence Information Application, and the Telephone
Application---accessible via a point-and-click Web interface. Next, it
would rebuild the FBI\'s intranet. Finally, it was supposed to identify
a way to replace the FBI\'s 40-odd investigative software applications,
including ACS.

[Spring 2001: Trilogy Contracts Awarded]{.underline}

In May and June 2001, the bureau awarded Trilogy contracts to two major
U.S. government contractors: DynCorp, of Reston, Va., for the hardware
and network projects, and to SAIC for software. All three Trilogy
components were to be delivered by the middle of 2004. The FBI used
cost-plus-award fee contracts for the hardware, networks, and software.

\[2\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[September 2001: Inadequate System Exposed; the Virtual Case File is
Born]{.underline}

On 4 September 2001, Robert S. Mueller III became the tenth director in
FBI history. One week later, terrorists pulverized New York City\'s
World Trade Center and a piece of the Pentagon. The inability of FBI
agents to share the most basic information about Al Qaeda\'s U.S.
activities blew up into a front-page scandal. Within days, the FBI\'s
pathetic technology infrastructure went from being so much arcane trivia
to a subject of daily fulmination by politicians and newspaper
columnists. In the face of intense public and congressional pressure,
Mueller shifted Trilogy into high gear.

Recognizing the limitations of the new web interface feature of Trilogy
and ACS, Robert J. Chiaradio the FBI\'s executive assistant director for
administration decided that the Bureau needed an entirely new database,
graphical user interface, and applications, which would let agents
search across various investigations to find relationships to their own
cases. The new case management system would host millions of records
containing information on everything from witnesses, suspects, and
informants to evidence such as documents, photos, and audio recordings.
To address concerns being raised by intelligence experts and lawmakers
in the wake of 9/11, these records would be accessible to both the
FBI\'s agents and its intelligence analysts. The new system was dubbed
the Virtual Case File.

With no detailed description of the FBI\'s processes and IT
infrastructure to go by (i.e., an enterprise architecture), a team of
FBI agents led by Larry Depew began to characterize investigative
processes such as witness interviews and surveillance operations and map
them to the FBI\'s software and databases. Depew, a veteran Special
Agent from the Trenton, New Jersey office, was a self taught database
programmer. Over a six-week period in the fall of 2001, the group
defined how agents worked, how they gathered information, and how that
information was fed into ACS. Working with engineers from SAIC, they
drew up diagrams and flowcharts of how the case management system
operated then and how they wanted the new case management system, the
VCF, to operate in the future.

\[3\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[December 2001: Contract Changes and an Accelerated
Schedule]{.underline}

In December 2001, the FBI asked SAIC to stop building a Web front end
for the old programs. Instead, SAIC was asked to devise a new
application, database, and graphical user interface to completely
replace ACS.

To formally define what users needed the VCF to do for them, SAIC
embarked on a series of Joint Application Development (JAD) sessions. In
these meetings, the FBI's team of agents and experts got together with a
group of SAIC engineers to hash out what functions the VCF would
perform. Ideas captured in these sessions formed the basis of the
requirements document that guided SAIC\'s application designers and
programmers.

In January 2002, the FBI requested an additional \$70 million to
accelerate Trilogy; Congress went further, approving \$78 million.
DynCorp committed to delivering its two components by July 2002. SAIC
agreed to deliver the initial version of the VCF in December 2003
instead of June 2004.

SAIC and the FBI were now committed to creating an entirely new case
management system in 22 months. While the Trilogy contracts were changed
to reflect the aggressive new deadlines, neither the original software
contract nor the modified one specified any formal criteria for the FBI
to use to accept or reject the finished VCF software. Furthermore, those
contracts specified no formal project schedules at all, let alone
milestones that SAIC and DynCorp were contractually obligated to meet on
the way to final delivery.

In reaction to the new deadline, SAIC broke its VCF development group
into eight teams, working in parallel on different functional pieces of
the program. The eight threads would later prove difficult for SAIC to
combine into a single system. Nevertheless, Rick Reynolds, vice
president and operations manager for SAIC, defended the decision to
change tactics. \"People forget the urgency that we were under and our
customer was under. And we were right beside them,\" he declared. \"We
were in the foxhole together.\"

[March 2002: Defining Requirements]{.underline}

Over a six-month period, the JAD team met in two-week sessions. After a
two-week JAD session finished, a two-week feedback cycle would begin.
SAIC provided Depew\'s team with information gleaned from the session,
including needs statements, flow charts, and meeting minutes. Depew\'s
team reviewed these materials and gave SAIC feedback while
simultaneously preparing subject matter experts for the next round of
JAD sessions, which immediately followed the feedback cycle. There were
no breaks.

\[4\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

Mueller, Dies, and Chiaradio recruited C.Z. (\"Sherry\") Higgins, a
seasoned IT program manager with experience at Lucent and AT&T, to be
the FBI's new project management executive for the Office of the
Director. The new Office of Program Management would centralize IT
management and oversee, develop, and deploy the bureau\'s most
expensive, complex, and risky projects. But her most important
assignment was to manage Trilogy. Her first move when she came on board
in March 2002 was to appoint Depew, who had no IT project management
experience, the VCF project manager.

[April 2002: Schedules Start to Slip]{.underline}

In mid-April 2002, Higgins, demanded a detailed schedule from DynCorp.
After she got it, she pulled the project teams from the FBI and DynCorp
into a meeting and went through the document. Shortly after that,
Higgins broke the news to the director: the computers and networks would
not be delivered in July of that year as had been scheduled. She told
Mueller that DynCorp didn\'t stand a chance of hitting the delivery
target, because it didn\'t have a detailed schedule that mapped out how
it would deploy, integrate, and test the new computers and networks.

[July 2002: VCF Looked Good on Paper]{.underline}

The JAD sessions had produced an exhaustively detailed requirements
document. This plan for a case-management system would combine the ACS
with two other systems: the Telephone Application, the bureau\'s central
repository of telephone records related to investigations, and parts of
the Criminal Law Enforcement Application, a repository for investigative
data about people, organizations, locations, vehicles, and
communications.

The VCF system would accept scanned documents, photographs, and other
electronic media---to simplify evidence tracking. People with the proper
credentials would be able to access that evidence from any FBI office.
The way work flowed through the bureau would change dramatically, too.
Instead of filling out a form either by hand or in a word-processing
program and then faxing or FedEx-ing the paper form to a supervisor, an
agent would fill out a form online and, with a click of the mouse, route
it to the supervisor. The document would pop up in a supervisor\'s
in-box, and the agent could track it to see if it had been approved. And
perhaps most important, information collected within a case file would
eventually be available to software applications that would compare data
among cases to search for correlations---to connect the proverbial dots.

\[5\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[Summer 2002: Detailed Requirements]{.underline}

By August 2002 SAIC had around 200 programmers on the job. Matthew
Patton joined the SAIC team as a security engineer. Patton, who was
later removed from the project after criticizing its management in an
Internet posting, was unimpressed by the 800-plus pages of requirements.
\"In a requirements document, you want to dictate the whats, not the
hows,\" Patton said. \"We need an e-mail system that can do x, and
there\'s 12 bullets. Instead, we had things like \'there will be a page
with a button that says e-mail on it.\' We want our button here on the
page or we want it that color. We want a logo on the front page that
looks like x. We want certain things on the left-hand side of the page.
They were trying to design the system layout and then the whole
application logic before they had actually even figured out what they
wanted the system to do.\"

The overly specific nature of the requirements focused developers on
their tiny piece of the puzzle. They were writing code, Patton said,
with no idea of how their piece fit with the others. This presaged the
integration problems that would later plague the project.

Patton also claimed that SAIC was determined to write much of the VCF
from scratch. This included an e-mail-like system that at least one
team, to his knowledge, was writing, even though the FBI was already
using an off-the-shelf software package, Novell\'s GroupWise, for
e-mail.

[December 2002: The March of the Change Requests]{.underline}

In December 2002, Higgins asked lawmakers to invest an additional
\$137.9 million in Trilogy. Congress approved another \$123.2 million
for a project whose total cost had now ballooned to \$581 million.

After the FBI and SAIC agreed to the baseline set of requirements, SAIC
programmers began cranking out code. The company had settled on a spiral
development methodology, an iterative approach to writing software.
Basically, SAIC programmers would write and compile a block of code that
performed a particular function, then run it to show Depew\'s agents
what it would do. The agents gave the programmers feedback, and the
programmers tried to incorporate the suggested changes. Sometimes
Depew\'s team had only two days to review a batch of code. Agents would
pull all-nighters to get the evaluation finished, \"and in the next
iteration their comments wouldn\'t be taken into account,\" Higgins
said. Sometimes, she acknowledged, these evaluations would include
changes to the requirements---functions that the agents had decided that
they needed once they saw what they were going to get. Other times the
FBI team would find bugs that needed to be fixed.

\[6\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

If there was a dispute as to whether the change could or should be made,
the agents sent an official request to the change control board,
composed of SAIC engineers and FBI personnel, for review. It wasn\'t
long before the change requests started rolling in---roughly 400 from
December 2002 to December 2003, according to SAIC.

\"Once they saw the product of the code we wrote, then they would say,
\'Oh, we\'ve got to change this. That isn\'t what I meant,\'\" said
SAIC\'s Reynolds. \"And that\'s when we started logging change request
after change request after change request.\" Reynolds added that SAIC\'s
bid on the original contract, and each subsequently revised cost
estimate, was based on there being \"minimal, minor changes\" to the
program once a baseline set of requirements had been agreed on.

Some of the changes were cosmetic---move a button from one part of the
screen to another, for instance. Others required the programmers to add
a new function to a part of the program, such as the graphical user
interface, common to all eight development threads.

[March 2003: The Network is Delayed]{.underline}

In March 2003, Computer Sciences Corp., which had acquired DynCorp that
month, told Higgins that the final deployment of the computers and
networks would be delayed until October. In August, October became
December. And in October, December became April 2004. The problem
wasn\'t the PCs, which had been trickling in since 2001, but changing
the e-mail system from Novell\'s GroupWise to Microsoft Outlook and,
obtaining the components needed to connect the field offices to the wide
area network. Higgins added that the delays were compounded by the
FBI\'s own sloppy inventories of existing networks and its
underestimation of how taxing the network traffic would be once all
22,000 users came online using their new PCs.

While the FBI and SAIC waited for the networks to go live so they could
test the VCF on a real system, changes and fixes continued. Many of the
changes had to be to be made by all eight of SAIC\'s development teams.
Arnold Punaro, SAIC executive vice president and general manager,
admitted in a posting on the company\'s Web site that in the rush to get
the program finished by December, SAIC didn\'t ensure that all of its
programmers were making the changes the same way. That inconsistency
occasionally meant that different modules of the VCF handled data in
different ways. Consequently, when one module needed to communicate with
another, errors sometimes occurred.

\"This, however, did not compromise the system,\" according to Punaro.
The real killers, he said, were \"significant management turbulence\" at
the FBI, \"the ever-shifting nature of the requirements,\" and the
agents\' \"trial-and-error, \'We will know it when we see it\' approach
to development.\"

\[7\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[September 2003: Testing Begins]{.underline}

SAIC began testing the program in the fall of 2003, and problems started
cropping up, some of which the agents had warned SAIC about over the
previous summer. SAIC officials complained to Higgins that Computer
Sciences Corp. didn\'t have its hardware and network in place, so SAIC
couldn\'t adequately test the VCF, crucial for a successful flash
cutover. They informed her that they would deliver a version of the VCF
to be in technical compliance with the terms of the contract and that
the FBI should feel free to make changes to it afterward.

[December 2003: VCF Delivery and Failure]{.underline}

On 13 December 2003, SAIC delivered the VCF to the FBI. Under the
direction of the FBI's new interim CIO, Zalmai Azmi, the FBI rejected
SAIC\'s delivery. The bureau found 17 \"functional deficiencies\" it
wanted SAIC to fix before the system was deployed. SAIC argued that at
least some of these deficiencies were changes in requirements. An
arbitrator was called in. The arbitrator found fault with both SAIC and
the FBI. Of the 59 issues and subissues derived from the original 17
deficiencies, the arbitrator found that 19 were requirements
changes---the FBI\'s fault; the other 40 were SAIC\'s errors.

While SAIC fixed bugs, Azmi, with the help of Depew\'s team, created
investigation scenarios that would take different cases from opening to
closing and tested them on the VCF. Those tests revealed an additional
400 deficiencies. \"We have requirements that are not in the final
product, yet we have capabilities in the final product that we don\'t
have requirements for,\" said Azmi.

At the same time Computer Sciences Corp was delivering the final pieces
of equipment to the FBI. By April, 22,251 computer workstations, 3408
printers, 1463 scanners, 475 servers, and new local and wide area
networks would all be up and running, 22 months later than the
accelerated schedule called for.

Azmi and SAIC had yet to agree on the VCF\'s ultimate fate, much less
when it would be deployed. Azmi rejected SAIC's proposal to take one
more year to make all the changes the FBI wanted at the cost of an
additional \$56 million.

[June 2003: Trying to Salvage Something from VCF:]{.underline}

Azmi asked SAIC to take the electronic workflow portion of the VCF, code
that was in relatively good shape, and turn that into what was
eventually called the Initial Operating Capability (IOC), at an
additional fixed price to the FBI of \$16.4 million. SAIC and the FBI
project team had six months to deliver a software package that would be
deployed to between 250 and 500 field personnel in Louisiana and at the
Hoover Building.

\[8\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

The objectives for the new project were clear: test-drive the VCF\'s
electronic workflow; see how people reacted to the graphical user
interface; create a way to translate the output from the VCF forms, into
the ACS system; check out network performance; and develop a training
program. The IOC would use Azmi\'s rigorous approach to software
development and project management, which he called the Life Cycle
Management Directive.

[June 2004: SAIC Begins the IOC Project]{.underline}

With new management in place, about 120 SAIC engineers began work on the
IOC project. The FBI and SAIC agreed to keep to a strict development
schedule, define acceptance criteria, and institute a series of control
gates. Azmi\'s project manager worked closely with his SAIC counterpart
to adhere to the baseline requirements SAIC and the FBI had agreed on
for the IOC in July, thus avoiding a death spiral of change requests. In
January, the IOC was rolled out as a pilot right on schedule.

[March 2005: IOC Pilot Fails]{.underline}

The IOC pilot ended in March. The verdict: \"Although the IOC
application was an aid to task management, its use did not improve the
productivity of most users,\" according to an internal FBI assessment.
FBI employees, particularly agents, found that the IOC actually
increased their workload. Agents filled out forms electronically and
routed them to superiors for approval, after which the electronic form
was uploaded to the ACS, still in use, to be shared with the rest of the
FBI. But to comply with the FBI\'s paper-based records management
system, the form had to be printed out, routed, signed, and filed.

[May 2005: Sentinel Rises from the Ashes of VCF]{.underline}

In May 2005 Mueller announced Sentinel, a four-phase, four-year project
intended to do the VCF\'s job and provide the bureau with a Web-based
case- and records-management system that incorporates commercial
off-the-shelf software.

[March 2006: Sentinel goes to Contract]{.underline}

The FBI awarded a \$305M contract to Lockheed Martin for Sentinel. Total
program cost is estimated to be \$425M.

[June 2007: Sentinel's First Deployment]{.underline}

The FBI and Lockheed Martin rolled out Phase 1 Sentinel capabilities to
the FBI workforce. According to a press release from the FBI phase 1
includes a "Personal Workbox, which summarizes a user's cases and
leads....and user-friendly search capabilities." Azmi announces his
intent to use agile development to deliver new capability to the desktop
every six to nine months.

\[9\]

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image12.png)**Activity (continued)**                        |
|                                                                       |
| **Key Findings**                                                      |
|                                                                       |
| Measurable Performance Standards                                      |
|                                                                       |
| a.  You are the program manager for this acquisition. What are some   |
|     of the decision points you might use to identify measurable       |
|     performance standards?                                            |
|                                                                       |
| > Best practice is to accomplish this early in the program/project.   |
| > VCF opportunities existed in Spring 2001 when Trilogy contracts     |
| > were awarded, again in December 2001 when the scope of work was     |
| > significantly changed, in April 2002 when it became clear that      |
| > schedules were slipping, in December 2002 when the change requests  |
| > began marching in, in June 2003 when the scope was changed to       |
| > salvage something from VCF, and in March 2005 when Sentinel rises   |
| > from the ashes.                                                     |
|                                                                       |
| b.  How would you utlilize your team to develop these standards?      |
|                                                                       |
| > Classic approach is for an Integrated Product Team to conduct       |
| > facilitated brainstorming sessions perhaps not unlike the March     |
| > 2002 JAD sessions. The standards would be linked to the strategic   |
| > goals for the program, with the key requirements, and with the key  |
| > criteria used for contractor selection/award.                       |
|                                                                       |
| c.  What types of methods would you use for measurement of the        |
|     standards?                                                        |
|                                                                       |
| > Four methods are available: testing, demonstration, inspection,     |
| > analysis. Each individual standard should use the most applicable   |
| > method. For example, if the performance standard was related to the |
| > need for email connectivity with agents then the measurement could  |
| > be a sampkle demonstration where an email is released to a          |
| > representative number of agents and the number of agents who        |
| > receive the meail is counted.                                       |
|                                                                       |
| d.  Where do you feel the actual program manager (and/or team) failed |
|     with regard to performance standards for this acquisition?        |
|                                                                       |
| > The case does not include evidence that performance standards were  |
| > established, and if they were, that the were uesd as a routine part |
| > of the management of the program. Failure seem to occur at every    |
| > opportunity.                                                        |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image13.png)**Activity (continued)**                        |
|                                                                       |
| **Key Findings (continued)**                                          |
|                                                                       |
| Remedies and Incentives                                               |
|                                                                       |
| a.  You are the program manager for this acquisition. What are some   |
|     of the decision points you might use to identify remedies and     |
|     incentives                                                        |
|                                                                       |
| > Best practice is to do this simultaniously with the indentification |
| > of measurable performance standards. See answer to question "a"     |
| > above.                                                              |
|                                                                       |
| b.  How would you utilize your team to develop these remedies and     |
|     incentives?                                                       |
|                                                                       |
| > See answer "b" above.                                               |
|                                                                       |
| c.  What types of methods would you use for determining applicability |
|                                                                       |
| > Use a positive feedback loop where the contractor performance is    |
| > monitored and compared with the effect of the remedies and          |
| > incentives. If the remedies and incentitives are stimulating the    |
| > desired performance then they can be retained, otherwise, they can  |
| > be adjusted.                                                        |
|                                                                       |
| d.  Where do you feel the actual program manager (and/or team) failed |
|     with regard to remedies and incentives for this acquisition?      |
|                                                                       |
| > See answer "d' above.                                               |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Elements of a Performance-Based Agreement (PBA)
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the performance-based agreements. After   |
| addressing any questions the participants may have, inform them that  |
| we will be moving on to the next topic: Budget Anaysis.               |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image8.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  performance-based agreements?**

  -----------------------------------------------------------------------

[]{.mark}

[]{.mark}

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ----------------------------------------------------- ----------------------
  ![](media/image14.wmf){width="3.9506944444444443in"   Slide \# 7-7
  height="2.9680555555555554in"}                        

  ----------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| > This topic will explore the different elements of overall budget    |
| > analysis functions and activities associated with a                 |
| > Performance-Based Agreement (PBA).                                  |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| > These include, but are not limited to:                              |
|                                                                       |
| Cost estimating                                                       |
|                                                                       |
| Cost analysis                                                         |
|                                                                       |
| Reconciliation of cost estimates                                      |
|                                                                       |
| Financial planning                                                    |
|                                                                       |
| Formulating financial programs and budgets                            |
|                                                                       |
| Budget analysis/execution                                             |
|                                                                       |
| Benefit-cost analysis                                                 |
|                                                                       |
| Earned value management (EVM) as it relates to budget analysis tasks  |
|                                                                       |
| > **Cost estimates** are one of the fundamental building blocks of    |
| > the acquisition process. The cost estimate and its supporting       |
| > budget are a part of the baseline against which a program\'s        |
| > progress and success are measured. The agency must be able to       |
| > produce and use cost estimates in order to evaluate whether a       |
| > system is affordable and consistent with the program objectives.    |
|                                                                       |
| This topic focuses on basic cost estimating tools and techniques.     |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly review and discuss the          |
| importance of cost estimates in a PBA.                                |
+-----------------------------------------------------------------------+

![](media/image8.png){width="0.49930555555555556in"
height="0.48819444444444443in"} Why are program cost estimates so
important in a PBA?

Answer:

Program cost estimates are important because:

They are the basis of program budgets and are also critical in the
day-to-day management of a program.

Current acquisition regulations require programs to maintain a program
cost estimate.

They are necessary to support agency-directed analyses, to defend
program budget requirements, or to choose between potential contractors
in source selection.

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| A life cycle cost estimate is the product of an estimating procedure  |
| that specifies the expected dollar cost to develop, procure, operate  |
| and support the system over its entire life, and finally dispose of   |
| the system.                                                           |
|                                                                       |
| Cost estimates must identify all elements involved in the             |
| development, production, operation and disposal of a system,          |
| regardless of funding source or management control.                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| There are four main cost estimating techniques used to develop cost   |
| estimates for acquisition programs: **analogy**, **parametric**,      |
| **engineering** and **actual costs**. In deciding which technique to  |
| use, it is important to consider several factors:                     |
|                                                                       |
| Availability of historical data                                       |
|                                                                       |
| Level of detail required                                              |
|                                                                       |
| Adequacy of system description                                        |
|                                                                       |
| Time/resource constraints                                             |
|                                                                       |
| No matter which estimating techniques are used, the program manager   |
| must ensure the cost estimate completely defines the program and is   |
| technically sound and reasonable. The cost estimate must be           |
| defensible with well-reasoned analysis.                               |
|                                                                       |
| Let us consider each of the four cost estimating methodologies in     |
| more detail.                                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The **analogy** method is the simplest form of estimating. When       |
| historical information is available for a similar program that has    |
| already been completed, that information can be used to estimate      |
| costs for the proposed program. The estimator makes a subjective      |
| evaluation of the differences between the new system and historical   |
| systems. An example would be estimating the cost of a new model car   |
| based on what was paid for the same makes and model 5 years ago.      |
|                                                                       |
| In many cases, it is necessary to compare subsystems of the new       |
| system to subsystems of several old systems in order to make the most |
| accurate comparisons. For example, an engineer will typically be      |
| asked to make a technical evaluation of the difference between the    |
| new system and the old system. Based on the engineer\'s evaluation, a |
| cost estimator must assess the cost impact of the technical           |
| differences.                                                          |
|                                                                       |
| The analogy method is appropriate very early in the program life      |
| cycle when the system is not yet fully defined. When using this       |
| method, it is important that differences between the existing system  |
| and the proposed system (e.g., software languages, development        |
| methodologies, complexity) are identified and their impacts           |
| estimated.                                                            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to facilitate a group discussion and       |
| review of the advantages and disadvantages of the analogy cost        |
| estimating method. Elicit participant examples of programs where they |
| have used this method to illustrate their advantages and/or           |
| disadvantages experienced with the method.                            |
+-----------------------------------------------------------------------+

![](media/image8.png){width="0.49930555555555556in"
height="0.48819444444444443in"} Given the description of the analogy
cost estimating method and your experience using this method, what are
some advantages you can think of for this methodology?

Disadvantages?

Answer:

(These answers are baseline answers provided to assist in the
discussion.)

The advantage of the analogy method is that it is based on actual
experience.

The disadvantage of the analogy method is that in many instances, no
truly similar programs exist. Subjective evaluations must be made by the
technical staff and cost estimators when determining the cost impact of
the differences between old and new systems.

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| An **engineering** or \"**bottoms-up**\" estimate is generally a very |
| detailed, costly and time-consuming process for estimating costs.     |
| This technique involves associating costs to the lowest levels of a   |
| definable work within the Work Breakdown Structure (WBS).             |
|                                                                       |
| The direct labor hours required to complete the work are estimated    |
| from engineering drawings and specifications, usually by an           |
| industrial engineer using company or general industry standards. The  |
| engineers also estimate raw materials and purchase parts              |
| requirements. The remaining cost elements, such as tooling, quality   |
| control, other direct costs, and various overhead charges including   |
| systems engineering and project management must also be factored in.  |
|                                                                       |
| The engineering cost estimate is most often used late in the          |
| development or early in the production phases. This technique         |
| encourages the contractor to do his homework early on and define all  |
| work down to the lowest level of the WBS. It is also a great process  |
| control technique at the production facility.                         |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to facilitate a group discussion and       |
| review of the advantages and disadvantages of the engineering cost    |
| estimating method. Elicit participant examples of programs where they |
| have used this method to illustrate their advantages and/or           |
| disadvantages experienced with the method.                            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image8.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Given the description of the          |
| engineering (bottoms-up) cost estimating method and your experience   |
| using this method, what are some advantages you can think of for this |
| methodology?**                                                        |
|                                                                       |
| **Disadvantages?**                                                    |
|                                                                       |
| (These answers are baseline answers provided to assist in the         |
| discussion**.)**                                                      |
|                                                                       |
| The **advantage** of the bottoms-up method provides a detailed cost   |
| estimate and can be more accurate than other methods.                 |
|                                                                       |
| Some **disadvantages** of the bottoms-up method include:              |
|                                                                       |
| -   Since detailed information is required, the bottoms-up method     |
|     tends to be time and cost intensive.                              |
|                                                                       |
| -   Historical data is not always available to support these          |
|     estimates.                                                        |
|                                                                       |
| -   There is a tendency to rely extensively on judgment.              |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The technique of using **actual cost** data from a system to          |
| **extrapolate** the future estimated costs of building the same       |
| system is probably the most accurate cost estimating method, when     |
| available. Actual cost experience on prototype units, early           |
| engineering development hardware, and early production hardware for   |
| the program under consideration should be used to the maximum extent  |
| possible.                                                             |
|                                                                       |
| This method is usually used in the late phases of acquisition, such   |
| as production and for replenishment spares.                           |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly review and review of the        |
| advantages and disadvantages of the actual cost or "extrapolation"    |
| estimating method. Elicit participant examples of programs where they |
| have used this method to illustrate their advantages and/or           |
| disadvantages experienced with the method.                            |
+-----------------------------------------------------------------------+

![](media/image8.png){width="0.49930555555555556in"
height="0.48819444444444443in"} Given the description of the actual cost
(extrapolation) estimating method and your experience using this method,
what are some advantages you can think of for this methodology?

Disadvantages?

Answer:

(These answers are baseline answers provided to assist in the
discussion.)

-   The advantage to the actual cost estimating method is that it is one
    of the most accurate cost estimating methods because it is based on
    actual costs.

-   A disadvantage of the actual cost method is that the program manager
    often does not have actual costs until late in the acquisition
    process, and therefore does not have this information available for
    budgets.

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The **parametric**, or **statistical**, cost estimating method uses a |
| database of similar systems and generates an estimate based on the    |
| relationship between cost and system performance or design            |
| characteristics. This method assumes that there is a relationship     |
| between some system parameter (such as speed, weight, thrust, etc.)   |
| and cost.                                                             |
|                                                                       |
| The parametric method is appropriate early in the program life cycle  |
| when detailed design specifications are not available, but a database |
| of like systems, performance specifications and costs is available.   |
|                                                                       |
| This method often serves as a useful check of an estimate made using  |
| another method.                                                       |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly review and review of the        |
| advantages and disadvantages of the parametric or "statistical"       |
| estimating method. Elicit participant examples of programs where they |
| have used this method to illustrate their advantages and/or           |
| disadvantages experienced with the method.                            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**

  -----------------------------------------------------------------------

![](media/image8.png){width="0.49930555555555556in"
height="0.48819444444444443in"} Given the description of the parametric
(statistical) estimating method and your experience using this method,
what are some advantages you can think of for this methodology?

Disadvantages?

Answer:

(These answers are baseline answers provided to assist in the
discussion.)

Some advantages of the parametric (or statistical) cost estimating
method include:

-   Parametric estimates are fast and easy to use.

-   Parametric estimates are easy to change to do "what--if" drills.

Disadvantages of this method are:

-   This method relies heavily on the timeliness and accuracy of the
    database.

-   Parametric estimates must rely on correlations between cost and
    performance features/characteristics.

-   Historical data is not always available to support these estimates.

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| The following activity is an optional activity the instructor can     |
| facilitate if time permits during the module.                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image15.png) **Activity (Optional)**                        |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity provides participants with an opportunity to identify   |
| and apply the most effective method of cost estimating for the VCF    |
| Acquisition Program Business Case Study.                              |
|                                                                       |
| *This module activity continues the business case that will be used   |
| throughout this course to apply various concepts and reinforce        |
| learning through the use of critical thinking and application         |
| skills.*                                                              |
|                                                                       |
| Activity Length: 30 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Conduct this activity as a class-wide discussion.                 |
|                                                                       |
| 2.  Allow participants 5 minutes to individually review the *VCF      |
|     Acquisition Program Management Business Case* found in their      |
|     Participant Guide and on the following pages of this Instructor   |
|     Guide..                                                           |
|                                                                       |
| 3.  Allow class participants 10 minutes to answer the following       |
|     questions.                                                        |
|                                                                       |
|     a.  Which of the four cost estimating methodologies (analogy,     |
|         parametric, engineering and actual costs) would make the most |
|         sense for the program manager to use at each of the following |
|         points? Spring 2001, December 2001, Summer 2002, June 2003,   |
|         and May 2005?                                                 |
|                                                                       |
|     b.  Provide a rational for each.                                  |
|                                                                       |
| 4.  Once the initial activity is completed, divide participants into  |
|     sub-groups based on the types of cost estimating methodologies    |
|     that were used (there will no more than four groups total,        |
|     assuming all of the methodologies were used by at least one       |
|     participant or more).                                             |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image16.png) **Activity (continued)**                       |
|                                                                       |
| 5.  Have each sub-group address the use of one of the cost estimating |
|     methodologies by answering to the following questions.            |
|                                                                       |
|     a.  What information is available to the program manager to       |
|         perform a cost estimation using the methodology assigned to   |
|         your sub-group?                                               |
|                                                                       |
|     b.  What additional information does the program manager need to  |
|         perform the cost estimation?                                  |
|                                                                       |
|     c.  How can the program manager obtain the needed additional      |
|         information?                                                  |
|                                                                       |
| 6.  Use the small group findings to facilitate a large group          |
|     discussion about the findings for each of the cost estimating     |
|     methodologies.                                                    |
|                                                                       |
| > **Note:** *Relevant findings for each criteria found on the         |
| > following page of this Instructor Guide.*                           |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image17.png) **Activity (continued)**                       |
|                                                                       |
| **Key Findings**                                                      |
|                                                                       |
| Cost estimating methodologies selected and the rational:              |
|                                                                       |
| Information available, needed, and method to obtain:                  |
|                                                                       |
| Spring 2001: Engineering cost estimating methodology would be used    |
| because this is the beginning of a large new program and the          |
| availability of the Government's WBS that would have been created     |
| prior to the initiation of the program. Although some historical data |
| may have existed to support analgious estimating, and some            |
| statistical data may have existed to support parametric estimating,   |
| neither of these techniques would have benefited from sufficient      |
| amounts of relevant data so early in the program. No data would have  |
| been available to support actual estimating.                          |
|                                                                       |
| December 2001: Engineering costimating methodology would be used      |
| because the scope of requested work was new and different from what   |
| the ccontractor had beeen doing (new application/database/GUI versus  |
| web front end). A detailed WBS would need to be created likely by     |
| experts within the program office who had experience with the         |
| preceeding Trilogy effort.                                            |
|                                                                       |
| Summer 2002: Actual cost estimating methodology would be used because |
| the scope of work remained unchanged and considerable information     |
| would have been available about the preceeding efforts (from the JAD  |
| sessions).                                                            |
|                                                                       |
| June 2003: Parametric estimating methodology would be used because of |
| the depth of relevant information available about code writing rates  |
| during the preceeding electronic workflow portion of VCF. Code        |
| writing rates based upon the preceeding work would need to be         |
| developed likely by the software team.                                |
|                                                                       |
| May 2005: Anology or Engineering estimating methodology would be used |
| because the scope of the new Sentinel effort would have closely       |
| resembled that of the previous VCF effort and because it represented  |
| the beginning of a large new program and the availability of the      |
| Government's WBS that would have been created prior to the initiation |
| of the program. Would likely be beneficial to have more than one cost |
| estimate prepared and then compared with eachother.                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ----------------------------------------------------- ----------------------
  ![](media/image18.wmf){width="3.9506944444444443in"   Slide \# 7-8
  height="2.9680555555555554in"}                        

  ----------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| One of the most useful aspects of earned value (EV) analysis is the   |
| interpretation of contractor performance trends. EV trend charts      |
| provide a visual means of seeing the contractor performance trends as |
| well as tracking how well they are accomplishing the work as compared |
| to the Performance Measurement Baseline (PMB) established at the      |
| beginning of the contract. .                                          |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| The activity on the following page will enable participants to apply  |
| their existing knowledge of EV analysis to performance trends and     |
| determine how the related project is performing.                      |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image19.png)**Activity**                                    |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity will cause participants to apply budget analysis        |
| perspective to earned value charts and develop opinions of how well   |
| (or poorly) the projects are performing.                              |
|                                                                       |
| Activity Length: 30 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  This activity is an individual activity with group discussion to  |
|     develop findings.                                                 |
|                                                                       |
| 2.  Have participants review each of the five slides:                 |
|                                                                       |
|     a.  Stable SPI Chart                                              |
|                                                                       |
|     b.  Unstable SPI Chart                                            |
|                                                                       |
|     c.  Stable CPI Chart                                              |
|                                                                       |
|     d.  Unstable CPI Chart                                            |
|                                                                       |
|     e.  CPI -- TCPI Trend Chart                                       |
|                                                                       |
| 3.  For each graphic (#7, 8, 10, 11, 17, shown on the next few pages  |
|     and are in the Participant Guide), participants should identify   |
|     how well or how poorly the project is performing, and be prepared |
|     to discuss their rationale for their opinions.                    |
|                                                                       |
| > <https://acc.dau.mil/docs/evm/module4/frame.htm>                    |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

![](media/image20.png){width="6.496527777777778in"
height="4.820138888888889in"}

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

![](media/image21.png){width="6.496527777777778in"
height="4.844444444444444in"}

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

![](media/image22.png){width="6.49375in" height="4.836805555555555in"}

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

![](media/image23.png){width="6.49375in" height="4.838888888888889in"}

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

![](media/image24.png){width="6.496527777777778in"
height="4.844444444444444in"}

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image25.png)**Activity**                                    |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity provides participants with an opportunity to consider   |
| the impact of cost estimation policies and practices by examining     |
| GAO-07-912 Report Exerpts for the follow-on to the FBI VCF program.   |
|                                                                       |
| (*This module activity continues the business case that will be used  |
| throughout this course to apply various concepts and reinforce        |
| learning through the use of critical thinking and application         |
| skills.*)                                                             |
|                                                                       |
| Activity Length: 45 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Conduct this activity as a class-wide discussion.                 |
|                                                                       |
| 2.  Allow participants 15 minutes to individually review the          |
|     *GAO-07-912 Report Exerpts* found in their Participant Guide and  |
|     on the following pages of this Instructor Guide..                 |
|                                                                       |
| 3.  Allow class participants 10 minutes to answer the following       |
|     questions.                                                        |
|                                                                       |
|     a.  What would your reaction as Sentinal program manager be to    |
|         the GAO report?.                                              |
|                                                                       |
|     b.  What actions, if any, would you as the Sentinal program       |
|         manager take as a reuult of the report?                       |
|                                                                       |
| 4.  Solicit the answers from the groups during a class-wide           |
|     discussion and allow individuals to answer on a volunteer basis.  |
|                                                                       |
| 5.  If necessary, elaborate on the impact of cost estimation policies |
|     and practices to ensure correct information is provided for the   |
|     whole group.                                                      |
|                                                                       |
| > **Note:** *This activity is targeted at having the participants     |
| > develop an understanding of the impact of cost estimation policies  |
| > and practices. As such it has no sample answer. Key Findings are    |
| > not applicable to this activity.*                                   |
|                                                                       |
| 6.  After all the questions have been discussed, summarize the        |
|     activity by telling participants that there is a considerable     |
|     amount benefit to a program by developing and following a         |
|     comprehensive set of cost estimation policies and practices.      |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**GAO-07-912 Report Excerpts**

**FBI Policies and Procedures Governing Sentinel Schedule and Cost
Estimates Do Not Reflect Important Best Practices:**

The FBI's policies and procedures that form the basis for Sentinel's
schedule and cost estimates are not fully consistent with reliable
estimating practices. While the FBI has issued an IT program management
handbook, related guidance, and tools that define how IT program
schedules and costs are to be estimated, this handbook and related
material do not. For example, address having a historical database of
program schedule and cost estimates to inform future estimates.

In addition, this handbook and related material do not adequately
address such schedule estimating practices as providing float time
between key activities and reserve time for high risk activities, and
they do not adequately address such cost estimating best practices as
documentation of source information. The cost estimates that the FBI has
developed for Sentinel reflect these limitations in policies,
procedures, and tools. In particular, the estimates to date did not
include all relevant costs and could not be verified by supporting
documentation.

Without well-defined policies, procedures, and supporting tools for
estimating IT programs' schedules and costs, the reliability of these
programs' respective estimates is questionable and, in the case of
Sentinel, a key basis of informed investment management is missing.

**FBI Policies and Procedures for Estimating Program's Costs Address
Some, but Not All, Best Practices:**

A reliable cost estimate is critical to the success of any IT program.
Such an estimate provides the basis for informed investment decision
making, realistic budget formulation and program resourcing, meaningful
progress measurement, proactive course correction when warranted, and
accountability for results.

According to OMB,16 programs must maintain current and well-documented
estimates of program costs, and these estimates must encompass the full
life cycle of the program. Among other things, OMB states that
generating reliable program cost estimates is a critical function
necessary to support OMB's capital programming process. Without this
capability, agencies are at risk of experiencing program cost overruns,
missed deadlines, and performance shortfalls.

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**GAO-07-912 Report Excerpts (continued)**

Our research has identified a number of best practices that are the
basis of effective program cost estimating. We have grouped these
practices into four characteristics of a high-quality and reliable cost
estimate. They are:

-   Comprehensive: The cost estimates should include both government and
    contractor costs of the program over its full life cycle, from
    inception of the program through design, development, deployment,
    and operation and maintenance to retirement of the program. They
    should also provide a level of detail appropriate to ensure that
    cost elements are neither omitted nor double counted, and they
    should document all cost-influencing ground rules and assumptions.

-   Well-documented: The cost estimates should capture in writing such
    things as the source data used and their significance, the
    calculations performed and their results, and the rationale for
    choosing a particular estimating method or reference. Moreover, this
    information should be captured in such a way that the data used to
    derive the estimate can be traced back to, and verified against
    their sources.

-   Accurate: The cost estimates should provide for results that are
    unbiased, and they should not be overly conservative or optimistic
    (i.e., should represent most likely costs). In addition, the
    estimates should be updated regularly to reflect material changes in
    the program, and steps should be taken to minimize mathematical
    mistakes and their significance. Among other things, the estimate
    should be grounded in documented assumptions and a historical record
    of cost and schedule estimating and actual experiences on other
    comparable programs.

-   Credible: The cost estimates should discuss any limitations in the
    analysis performed due to uncertainty or biases surrounding data or
    assumptions, and their derivation should provide for varying major
    assumptions and recalculating outcomes based on sensitivity
    analyses, and the associated risk and uncertainty inherent in
    estimates should be disclosed. Further, the estimates should be
    verified based on crosschecks using other methods and by comparing
    the results with independent cost estimates.

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**GAO-07-912 Report Excerpts (continued)**

The FBI's policies and procedures that govern estimating program costs
are defined in the bureau's IT Program Management Handbook, Cost-
Benefit Analysis Guide, and IT Investment Management Process. To the
bureau's credit, these documents reflect some of the previously cited
best practices.

For example, the handbook calls for cost estimates to be comprehensive
and to be life cycle in scope, including total costs (e.g., research,
development, production, training, operations and maintenance, software
licensing, and labor) over its full life cycle (from initiation to
system retirement). Moreover, FBI guidance partially provides for
documenting these estimates and ensuring their accuracy by, for example,
stating that estimating assumptions should be documented and that the
estimates are to be updated on a regular basis.

However, these policies and procedures do not reflect all of the cost
estimating best practices associated with well-documented, accurate, and
credible estimates. With respect to being well-documented, they do not
require that the sources of historical data used in the estimate be
documented and, with respect to accuracy, they do not provide for the
establishment and use of a historical database of estimating and actual
experiences on comparable programs.

Without documenting estimated data sources, the basis for the estimates,
including the circumstances surrounding the data used to derive and
whether these data have been properly normalized, cannot be understood.
This means that the reliability of the estimate for either current use
in managing a program or for inclusion in a historical database for use
in future program estimates, cannot be assured.

Further, without provision for establishing and using a historical
database, one will not be available to inform future estimates, as is
the case for the FBI. With respect to credibility, the FBI's policies
and procedures do not address the need to consider and reflect any
limitations in the analyses on which the estimates are based, or to
document any uncertainty or biases surrounding the data used.

As a result, the associated uncertainty in the estimate itself cannot be
determined, thus limiting the estimate's integrity and utility. Further,
the FBI's policies and procedures do not provide for the conduct of
risk/sensitivity analyses18 and disclosure of the associated risk and
uncertainty of the estimates. Thus, estimates will not include important
information to inform program decision making, such as the range of
potential costs surrounding the point estimate and the reasons behind
this range.

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**GAO-07-912 Report Excerpts (continued)**

FBI Office of the CIO officials agreed that these practices are not
included in the bureau's policies and procedures that form the basis for
IT program cost estimates and that they need to be. Until an effective
basis for cost estimating is in place and employed, FBI IT programs,
like Sentinel, will likely not have reliable cost estimates to properly
inform investment decision making and the risk of actual program cost
performance not tracking closely to estimates will be increased.

Our analysis of Sentinel cost estimates19 revealed reliability issues.
In particular, none of the estimates are comprehensive in that they each
omit relevant costs. For example, one estimate does not include
government or support contractor costs and, according to program
officials, another estimate does not include technology refresh, certain
government labor costs, or inflationary costs.

In addition, these estimates cannot be considered fully accurate or well
documented. For example, according to program officials, none of the
estimates was derived using a historical database reflecting actual and
estimated costs on similar programs. Further, none of the estimates had
a fully documented estimating methodology, although some parts of one
cost estimate were documented.

Also, none of the estimates could be traced to the source of the data
that were used in developing them. These reliability concerns with the
Sentinel cost estimates are due in part to the FBI's not following its
own cost estimating policies and procedures and in part to the
previously mentioned limitations in the FBI's cost estimating policies,
procedures, and supporting tools.

As a result, the Sentinel cost estimates do not provide a sufficient
basis for informed investment decision making and do not facilitate
meaningful tracking of progress against estimates, both of which are
fundamental to effectively managing an IT program.

  -----------------------------------------------------------------------
  **Topic 3: Budget Analysis (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the budget analysis. After addressing any |
| questions the participants may have, inform them that we will be      |
| moving on to the next topic: Module Review.                           |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image8.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  budget analysis?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 4: Review**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ----------------------------------------------------- ----------------------
  ![](media/image26.wmf){width="3.9506944444444443in"   Slide \# 7-9
  height="2.9680555555555554in"}                        

  ----------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Let's review the learning objectives for this module on               |
| Performance-Based Agreements.                                         |
|                                                                       |
| -   Explain how to appropriately administer the establishment of a    |
|     negotiated baseline of performance for an acquisition.            |
|                                                                       |
| -   Describe the negotiations for the required level of support at a  |
|     cost consistent with available support funding.                   |
|                                                                       |
| (continued on next page)                                              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Review (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ----------------------------------------------------- ----------------------
  ![](media/image26.wmf){width="3.9506944444444443in"   Slide \# 7-9
  height="2.9680555555555554in"}                        (continued)

  ----------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Let's review the learning objectives for this module on               |
| Performance-Based Agreements.                                         |
|                                                                       |
| -   Explain the application of management actions required of program |
|     managers who are involved in acquisition of services.             |
|                                                                       |
| -   Define the different methods of performance measurement as part   |
|     of program management.                                            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Review (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes Module 7 -- Performance-Based Agreements.. Address any |
| questions the participants may have before moving on to Module 8 --   |
| Life Cycle Logistics.                                                 |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image8.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have about Module
  7 -- Performance-Based Agreements?**

  -----------------------------------------------------------------------