-----------------------------------------------------------------------
  **Module Title**

  -----------------------------------------------------------------------

  --------------------------------------------------- ---------------------
  ![](media/image1.wmf){width="3.957638888888889in"   Slide \# 5-1
  height="2.975in"}                                   

  --------------------------------------------------- ---------------------

**Module 5: Perform Source Selection**

**Module Outline**

  ------------------------------------------------- ----------- ---------------------
  Topic                                                         Time Frame

  Introduction/Objectives                           5 minutes   

  Source Selection Criteria (not much time spent on 80 minutes  
  this)                                                         

  Risk Assessment                                   100 minutes 

  Best Value Selection                              40 minutes  

  Formal Source Selection                           40 minutes  

  International Sourcing vs. Domestic Preferences   20 minutes  

  Price Reasonableness                              20 minutes  

  Review                                            10 minutes  

  **Estimated/Approximate Time for Module           **5 hours   
  Completion**                                      and 15      
                                                    minutes**   
  ------------------------------------------------- ----------- ---------------------

  -----------------------------------------------------------------------
  **Topic 1: Module Introduction/Objectives**

  -----------------------------------------------------------------------

  --------------------------------------------------- ----------------------
  ![](media/image2.wmf){width="3.957638888888889in"   Slide \# 5-2
  height="2.975in"}                                   

  --------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This module will discuss how to assess application of source          |
| selection criteria including risk analysis methods (Federal           |
| Acquisition Regulation {FAR} Part 15/15.3) and participate in the     |
| formulation of a source selection plan that allows for best value     |
| selection from a competitive solicitation. (FAR Part 7.1)             |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Module Introduction/Objectives (continued)**

  -----------------------------------------------------------------------

  --------------------------------------------------- ----------------------
  ![](media/image4.wmf){width="3.957638888888889in"   Slide \# 5-3
  height="2.975in"}                                   

  --------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Briefly review the module agenda by introducing the topics that will  |
| be covered. Topics include:                                           |
|                                                                       |
| -   Topic 2: Source Selection Criteria                                |
|                                                                       |
| -   Topic 3: Risk Assessment                                          |
|                                                                       |
| -   Topic 4: Best Value Selection                                     |
|                                                                       |
| -   Topic 5: Formal Source Selection                                  |
|                                                                       |
| -   Topic 6: International Sourcing vs. Domestic Preferences          |
|                                                                       |
| -   Topic 7: Price Reasonableness                                     |
|                                                                       |
| -   Topic 8: Review                                                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Module Introduction/Objectives (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  --------------------------------------------------- ----------------------
  ![](media/image6.wmf){width="3.957638888888889in"   Slide \# 5-4
  height="2.975in"}                                   

  --------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6916666666666667in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| [Objectives]{.underline}                                              |
|                                                                       |
| At the end of this module, you will be able to:                       |
|                                                                       |
| -   Given an acquisition case study, apply appropriate risk analysis  |
|     methods as part of source selection criteria.                     |
|                                                                       |
| -   Relate the specific legislation that applies to sourcing          |
|     preferences to an acquisition and explain how it impacts the      |
|     program manager's decision making authority.                      |
|                                                                       |
| -   Apply concepts of sourcing preferences and price reasonableness   |
|     to allow for best value selection in program management.          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 1: Module Introduction/Objectives (continued)**
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the introduction and objectives After     |
| addressing any questions the participants may have, inform them that  |
| we will be moving on to the next topic: Source Selection Criteria.    |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  the introduction and objectives?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria**

  -----------------------------------------------------------------------

  --------------------------------------------------- ----------------------
  ![](media/image8.wmf){width="3.957638888888889in"   Slide \# 5-5
  height="2.975in"}                                   

  --------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Typically, the program manager will be deeply involved in comparing   |
| the contractor's technical and management proposal and past           |
| performance information with the evaluation criteria in the Request   |
| for Proposal (RFP). It is important that the program manager have a   |
| detailed knowledge and firm understanding of the evaluation factors   |
| to be used in source selection.                                       |
|                                                                       |
| The source selection decision is based on a comparative assessment of |
| proposals against all evaluation factors provided in the              |
| solicitation.                                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Before a solicitation is issued, a Source Selection Plan (SSP) should |
| be prepared by the source selection integrated product team (IPT),    |
| reviewed by the Contracting Office, Source Selection Board, Source    |
| Selection Advisory Council (SSAC), and approved by the Source         |
| Selection Authority (SSA).                                            |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly review and discuss the general  |
| steps for source selection.                                           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**How would you describe the general    |
| steps for evaluation and selection of a source for contract award?**  |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| The general steps for evaluation and selection of a source for        |
| contract award are:                                                   |
|                                                                       |
| 1.  Compare each proposal to the requirements and evaluation criteria |
|     (factors) in the Request for Proposal (RFP).                      |
|                                                                       |
| 2.  Conduct price and/or cost analysis to determine fair and          |
|     reasonable price.                                                 |
|                                                                       |
| 3.  Compare/contrast the merits (strengths and weaknesses) of each    |
|     proposal.                                                         |
|                                                                       |
| 4.  Select the source for contract award.                             |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Evaluation factors inform offerors of the importance the agency       |
| attaches to various aspects of a proposal. Evaluation factors are a   |
| list, in order of relative importance, of those aspects of a proposal |
| that will be evaluated both quantitatively and qualitatively.         |
|                                                                       |
| The best value continuum (BVC) is a recognition that the agency       |
| always seeks to obtain the best value in negotiated acquisitions      |
| using any one or a combination of source selection approaches.        |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly review and discuss the BVC      |
| concept and its application.                                          |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**How is the BVC applied during the     |
| consideration of a proposal's evaluation factors?**                   |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| At one end of the continuum is the low priced technically acceptable  |
| strategy. At the other end is a process by which elements of a        |
| proposed solution can be traded off against each other to determine   |
| the solution that provides the agency with the overall best value.    |
|                                                                       |
| Tradeoffs are conducted according to the source selection evaluation  |
| factors and sub-factors.                                              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Generally, there should be a limited number of evaluation factors.    |
| Evaluation factors should cause the evaluation process to highlight   |
| significant differences among proposals in areas of primary interest  |
| to the procuring agency.                                              |
|                                                                       |
| It is counterproductive to have a large number of evaluation factors, |
| or subdivisions of individual factors, because they force             |
| unnecessarily detailed assessment of relatively unimportant things.   |
| This obscures significant differences among proposals due to an       |
| averaging of pluses and minuses at the lowest levels of indenture.    |
|                                                                       |
| Evaluation factors are established at the discretion of the SSA, but  |
| always include cost and quality as per FAR 15.304(c).                 |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Cost** is always a factor in source selection. In systems           |
| acquisitions, for the purpose of selecting a source, the \"cost\" the |
| agency is concerned about is the most probable life cycle cost to the |
| agency of the proposed system.                                        |
|                                                                       |
| -   This does not eliminate the need for the government to establish  |
|     the reasonableness of the proposed price or estimated             |
|     cost-plus-fee of the contemplated contract.                       |
|                                                                       |
| **Quality** is usually evaluated with technical, management or past   |
| performance criteria. Subject to certain limitations and thresholds,  |
| past performance and participation of small disadvantaged business    |
| concerns is also evaluated.                                           |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| FAR Part 15 prescribes the policies and procedures governing          |
| competitive and noncompetitive negotiated acquisitions, over the      |
| Simplified Acquisition threshold, generally over \$100,000. A         |
| contract awarded using other than a sealed bidding process is a       |
| negotiated contract.                                                  |
|                                                                       |
| FAR Subpart 15.1 addresses Source Selection Processes and Techniques. |
| It describes some of the acquisition processes and techniques that    |
| may be used to design competitive acquisition strategies suitable for |
| the specific circumstances of the acquisition.                        |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| FAR 15.308 states that                                                |
|                                                                       |
| *"The source selection authority's (SSA) decision shall be based on a |
| comparative assessment of proposals against all source selection      |
| criteria in the solicitation. While the SSA may use reports and       |
| analyses prepared by others, the source selection decision shall      |
| represent the SSA's independent judgment. The source selection        |
| decision shall be documented, and the documentation shall include the |
| rationale for any business judgments and tradeoffs made or relied on  |
| by the SSA, including benefits associated with additional costs.      |
| Although the rationale for the selection decision must be documented, |
| that documentation need not quantify the tradeoffs that led to the    |
| decision."*                                                           |
|                                                                       |
| When using FAR 15 procedures, there are two approaches to source      |
| selection: lowest priced technically acceptable (FAR 15.101-2) and    |
| trade-off (FAR 15.101-1).                                             |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following questions to provide a description of when it is    |
| appropriate to use the two approaches for source selection: lowest    |
| priced technically acceptable and trade-off.                          |
|                                                                       |
| Use participant responses to facilitate discussion for each of the    |
| two approaches based on experience with both.                         |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**When would a lowest price technically |
| acceptable source selection process be appropriate?**                 |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| The lowest price technically acceptable source selection process is   |
| appropriate when best value is expected to result from selection of   |
| the technically acceptable proposal with the lowest evaluated price.  |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**When would the use of a trade-off     |
| source selection process be appropriate?**                            |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| A trade-off process is appropriate when it may be in the best         |
| interest of the agency to consider award to other than the lowest     |
| priced offeror or other than the highest technically rated offeror.   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The \"rules\" for the best-value and tradeoff process (and the degree |
| of documentation required) depend on two factors:                     |
|                                                                       |
| -   The rules for the specific acquisition process being used         |
|                                                                       |
| -   The rules the agency sets in the solicitation                     |
|                                                                       |
|     *For example, when conducting a negotiated procurement, the       |
|     complex processes of FAR Subpart 15.1, \"Source Selection         |
|     Processes and Techniques,\" and FAR Subpart 15.3, \"Source        |
|     Selection,\" apply. Conversely, when using Federal Supply         |
|     Schedule contracts, the simpler provisions at FAR 8.404 would     |
|     apply to the procurement. However, if the agency writes FAR       |
|     15-type rules into a Request for Quote under Federal Supply       |
|     Schedule contracts, the rules in the RFQ control.*                |
|                                                                       |
| Once the source selection criteria have been established, it is       |
| possible to make a "best value" selection for the acquisition source, |
| if best value is determined by the program manager to be the most     |
| advantageous method for the acquisition.                              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image9.png) **Activity:**                                   |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity provides participants with an opportunity to consider   |
| types of source selection factors that would be appropriate for the   |
| Virtual Case File (VCF) acquisition business case. **Note:** *The     |
| instructor should role play the position of the Source Selection      |
| Board and respond to participant critieria from that perspective.*    |
|                                                                       |
| (*This module activity continues the business case that will be used  |
| throughout this course to apply various concepts and reinforce        |
| learning through the use of critical thinking and application         |
| skills.*)                                                             |
|                                                                       |
| Activity Length: 60 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Divide the participants into small groups.                        |
|                                                                       |
| 2.  Assign each group one of the four evaluation criteria: technical, |
|     management, past performance, and cost.                           |
|                                                                       |
| 3.  Allow participants 5 -- 10 minutes to individually review the     |
|     *VCF Acquisition Program Management Business Case* found in their |
|     Participant Guide and on the following pages of this Instructor   |
|     Guide.                                                            |
|                                                                       |
| 4.  Allow each group 10 minutes to answer the following question for  |
|     the criteria assigned.                                            |
|                                                                       |
|     a.  What types of source selection criteria should the program    |
|         manager identify and use for the evaluation of proposals that |
|         led to the Spring 2001 award? Provide a rational using        |
|         information fron the case for each of the criteria?           |
|                                                                       |
| 5.  Have a spokesperson from each group describe the criteria and     |
|     rational.                                                         |
|                                                                       |
| 6.  If necessary, elaborate on the group's criteria and rational to   |
|     ensure correct information is provided for the whole group.       |
|                                                                       |
| 7.  Continue the process for each group until all groups have had a   |
|     chance to discuss their criteria and rational.                    |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image10.png) **Activity(continued)**                        |
|                                                                       |
| 8.  Pose the following question to the whole group, and allow         |
|     individuals to answer on a volunteer basis.                       |
|                                                                       |
|     a.  What are some effective types of source selection factors you |
|         have identified for programs/projects you've managed in the   |
|         past?                                                         |
|                                                                       |
| 9.  Use the answers given to generate a large group discussion to     |
|     elicit other opinions about the question. There may be more than  |
|     one correct answer or viewpoint for the question.                 |
|                                                                       |
| > **Note:** *Relevant example answers for each question are found on  |
| > the following page of this Instructor Guide.*                       |
|                                                                       |
| 10. After all the questions have been discussed, summarize the        |
|     activity by telling participants that there is not necessarily a  |
|     "right and wrong" when managing a program. The important aspect   |
|     is to make sure that the program manager exercises due diligence  |
|     when types of source selection criteria to be utlized for the     |
|     program.                                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[July 2000: Trying to Bring Order to Chaos (continued)]{.underline}

In 2000 the bureau finally began to deal with its outdated IT systems.
At the time, under the direction of Louis J. Freeh, the bureau had
neither a CIO nor documentation detailing its IT systems, much less a
plan for revamping them. The task of creating such a plan fell to former
IBM executive Bob E. Dies, who became assistant director in charge of
the FBI Information Resources Division on 17 July 2000. He was the first
of five officials who, over the next four years, would struggle to lead
the FBI\'s sprawling and antiquated information systems and get the VCF
project under way.

In September 2000, Congress approved \$379.8 million over three years
for what was then called the FBI Information Technology Upgrade Project.
Eventually divided into three parts, the program became known as
Trilogy. The Information Presentation Component would provide all 56 FBI
field offices, some 22,000 agents and support staff, with new Dell
Pentium PCs running Microsoft Office, as well as new scanners, printers,
and servers. The Transportation Network Component would provide secure
local area and wide area networks, allowing agents to share information
with their supervisors and each other.

But the User Applications Component, which would ultimately become the
VCF, staked out the most ambitious goals. First, it was to make the five
most heavily used investigative applications---the Automated Case
Support system, IntelPlus, the Criminal Law Enforcement Application, the
Integrated Intelligence Information Application, and the Telephone
Application---accessible via a point-and-click Web interface. Next, it
would rebuild the FBI\'s intranet. Finally, it was supposed to identify
a way to replace the FBI\'s 40-odd investigative software applications,
including ACS.

[Spring 2001: Trilogy Contracts Awarded]{.underline}

In May and June 2001, the bureau awarded Trilogy contracts to two major
U.S. government contractors: DynCorp, of Reston, Va., for the hardware
and network projects, and to SAIC for software. All three Trilogy
components were to be delivered by the middle of 2004. The FBI used
cost-plus-award fee contracts for the hardware, networks, and software.

\[1\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[September 2001: Inadequate System Exposed; the Virtual Case File is
Born]{.underline}

On 4 September 2001, Robert S. Mueller III became the tenth director in
FBI history. One week later, terrorists pulverized New York City\'s
World Trade Center and a piece of the Pentagon. The inability of FBI
agents to share the most basic information about Al Qaeda\'s U.S.
activities blew up into a front-page scandal. Within days, the FBI\'s
pathetic technology infrastructure went from being so much arcane trivia
to a subject of daily fulmination by politicians and newspaper
columnists. In the face of intense public and congressional pressure,
Mueller shifted Trilogy into high gear.

Recognizing the limitations of the new web interface feature of Trilogy
and ACS, Robert J. Chiaradio the FBI\'s executive assistant director for
administration decided that the Bureau needed an entirely new database,
graphical user interface, and applications, which would let agents
search across various investigations to find relationships to their own
cases. The new case management system would host millions of records
containing information on everything from witnesses, suspects, and
informants to evidence such as documents, photos, and audio recordings.
To address concerns being raised by intelligence experts and lawmakers
in the wake of 9/11, these records would be accessible to both the
FBI\'s agents and its intelligence analysts. The new system was dubbed
the Virtual Case File.

With no detailed description of the FBI\'s processes and IT
infrastructure to go by (i.e., an enterprise architecture), a team of
FBI agents led by Larry Depew began to characterize investigative
processes such as witness interviews and surveillance operations and map
them to the FBI\'s software and databases. Depew, a veteran Special
Agent from the Trenton, New Jersey office, was a self taught database
programmer. Over a six-week period in the fall of 2001, the group
defined how agents worked, how they gathered information, and how that
information was fed into ACS. Working with engineers from SAIC, they
drew up diagrams and flowcharts of how the case management system
operated then and how they wanted the new case management system, the
VCF, to operate in the future.

\[2\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[December 2001: Contract Changes and an Accelerated
Schedule]{.underline}

In December 2001, the FBI asked SAIC to stop building a Web front end
for the old programs. Instead, SAIC was asked to devise a new
application, database, and graphical user interface to completely
replace ACS.

To formally define what users needed the VCF to do for them, SAIC
embarked on a series of Joint Application Development (JAD) sessions. In
these meetings, the FBI's team of agents and experts got together with a
group of SAIC engineers to hash out what functions the VCF would
perform. Ideas captured in these sessions formed the basis of the
requirements document that guided SAIC\'s application designers and
programmers.

In January 2002, the FBI requested an additional \$70 million to
accelerate Trilogy; Congress went further, approving \$78 million.
DynCorp committed to delivering its two components by July 2002. SAIC
agreed to deliver the initial version of the VCF in December 2003
instead of June 2004.

SAIC and the FBI were now committed to creating an entirely new case
management system in 22 months. While the Trilogy contracts were changed
to reflect the aggressive new deadlines, neither the original software
contract nor the modified one specified any formal criteria for the FBI
to use to accept or reject the finished VCF software. Furthermore, those
contracts specified no formal project schedules at all, let alone
milestones that SAIC and DynCorp were contractually obligated to meet on
the way to final delivery.

In reaction to the new deadline, SAIC broke its VCF development group
into eight teams, working in parallel on different functional pieces of
the program. The eight threads would later prove difficult for SAIC to
combine into a single system. Nevertheless, Rick Reynolds, vice
president and operations manager for SAIC, defended the decision to
change tactics. \"People forget the urgency that we were under and our
customer was under. And we were right beside them,\" he declared. \"We
were in the foxhole together.\"

[March 2002: Defining Requirements]{.underline}

Over a six-month period, the JAD team met in two-week sessions. After a
two-week JAD session finished, a two-week feedback cycle would begin.
SAIC provided Depew\'s team with information gleaned from the session,
including needs statements, flow charts, and meeting minutes. Depew\'s
team reviewed these materials and gave SAIC feedback while
simultaneously preparing subject matter experts for the next round of
JAD sessions, which immediately followed the feedback cycle. There were
no breaks.

\[3\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

Mueller, Dies, and Chiaradio recruited C.Z. (\"Sherry\") Higgins, a
seasoned IT program manager with experience at Lucent and AT&T, to be
the FBI's new project management executive for the Office of the
Director. The new Office of Program Management would centralize IT
management and oversee, develop, and deploy the bureau\'s most
expensive, complex, and risky projects. But her most important
assignment was to manage Trilogy. Her first move when she came on board
in March 2002 was to appoint Depew, who had no IT project management
experience, the VCF project manager.

[April 2002: Schedules Start to Slip]{.underline}

In mid-April 2002, Higgins, demanded a detailed schedule from DynCorp.
After she got it, she pulled the project teams from the FBI and DynCorp
into a meeting and went through the document. Shortly after that,
Higgins broke the news to the director: the computers and networks would
not be delivered in July of that year as had been scheduled. She told
Mueller that DynCorp didn\'t stand a chance of hitting the delivery
target, because it didn\'t have a detailed schedule that mapped out how
it would deploy, integrate, and test the new computers and networks.

[July 2002: VCF Looked Good on Paper]{.underline}

The JAD sessions had produced an exhaustively detailed requirements
document. This plan for a case-management system would combine the ACS
with two other systems: the Telephone Application, the bureau\'s central
repository of telephone records related to investigations, and parts of
the Criminal Law Enforcement Application, a repository for investigative
data about people, organizations, locations, vehicles, and
communications.

The VCF system would accept scanned documents, photographs, and other
electronic media---to simplify evidence tracking. People with the proper
credentials would be able to access that evidence from any FBI office.
The way work flowed through the bureau would change dramatically, too.
Instead of filling out a form either by hand or in a word-processing
program and then faxing or FedEx-ing the paper form to a supervisor, an
agent would fill out a form online and, with a click of the mouse, route
it to the supervisor. The document would pop up in a supervisor\'s
in-box, and the agent could track it to see if it had been approved. And
perhaps most important, information collected within a case file would
eventually be available to software applications that would compare data
among cases to search for correlations---to connect the proverbial dots.

\[4\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[Summer 2002: Detailed Requirements]{.underline}

By August 2002 SAIC had around 200 programmers on the job. Matthew
Patton joined the SAIC team as a security engineer. Patton, who was
later removed from the project after criticizing its management in an
Internet posting, was unimpressed by the 800-plus pages of requirements.
\"In a requirements document, you want to dictate the whats, not the
hows,\" Patton said. \"We need an e-mail system that can do x, and
there\'s 12 bullets. Instead, we had things like \'there will be a page
with a button that says e-mail on it.\' We want our button here on the
page or we want it that color. We want a logo on the front page that
looks like x. We want certain things on the left-hand side of the page.
They were trying to design the system layout and then the whole
application logic before they had actually even figured out what they
wanted the system to do.\"

The overly specific nature of the requirements focused developers on
their tiny piece of the puzzle. They were writing code, Patton said,
with no idea of how their piece fit with the others. This presaged the
integration problems that would later plague the project.

Patton also claimed that SAIC was determined to write much of the VCF
from scratch. This included an e-mail-like system that at least one
team, to his knowledge, was writing, even though the FBI was already
using an off-the-shelf software package, Novell\'s GroupWise, for
e-mail.

[December 2002: The March of the Change Requests]{.underline}

In December 2002, Higgins asked lawmakers to invest an additional
\$137.9 million in Trilogy. Congress approved another \$123.2 million
for a project whose total cost had now ballooned to \$581 million.

After the FBI and SAIC agreed to the baseline set of requirements, SAIC
programmers began cranking out code. The company had settled on a spiral
development methodology, an iterative approach to writing software.
Basically, SAIC programmers would write and compile a block of code that
performed a particular function, then run it to show Depew\'s agents
what it would do. The agents gave the programmers feedback, and the
programmers tried to incorporate the suggested changes. Sometimes
Depew\'s team had only two days to review a batch of code. Agents would
pull all-nighters to get the evaluation finished, \"and in the next
iteration their comments wouldn\'t be taken into account,\" Higgins
said. Sometimes, she acknowledged, these evaluations would include
changes to the requirements---functions that the agents had decided that
they needed once they saw what they were going to get. Other times the
FBI team would find bugs that needed to be fixed.

\[5\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

If there was a dispute as to whether the change could or should be made,
the agents sent an official request to the change control board,
composed of SAIC engineers and FBI personnel, for review. It wasn\'t
long before the change requests started rolling in---roughly 400 from
December 2002 to December 2003, according to SAIC.

\"Once they saw the product of the code we wrote, then they would say,
\'Oh, we\'ve got to change this. That isn\'t what I meant,\'\" said
SAIC\'s Reynolds. \"And that\'s when we started logging change request
after change request after change request.\" Reynolds added that SAIC\'s
bid on the original contract, and each subsequently revised cost
estimate, was based on there being \"minimal, minor changes\" to the
program once a baseline set of requirements had been agreed on.

Some of the changes were cosmetic---move a button from one part of the
screen to another, for instance. Others required the programmers to add
a new function to a part of the program, such as the graphical user
interface, common to all eight development threads.

[March 2003: The Network is Delayed]{.underline}

In March 2003, Computer Sciences Corp., which had acquired DynCorp that
month, told Higgins that the final deployment of the computers and
networks would be delayed until October. In August, October became
December. And in October, December became April 2004. The problem
wasn\'t the PCs, which had been trickling in since 2001, but changing
the e-mail system from Novell\'s GroupWise to Microsoft Outlook and,
obtaining the components needed to connect the field offices to the wide
area network. Higgins added that the delays were compounded by the
FBI\'s own sloppy inventories of existing networks and its
underestimation of how taxing the network traffic would be once all
22,000 users came online using their new PCs.

While the FBI and SAIC waited for the networks to go live so they could
test the VCF on a real system, changes and fixes continued. Many of the
changes had to be to be made by all eight of SAIC\'s development teams.
Arnold Punaro, SAIC executive vice president and general manager,
admitted in a posting on the company\'s Web site that in the rush to get
the program finished by December, SAIC didn\'t ensure that all of its
programmers were making the changes the same way. That inconsistency
occasionally meant that different modules of the VCF handled data in
different ways. Consequently, when one module needed to communicate with
another, errors sometimes occurred.

\"This, however, did not compromise the system,\" according to Punaro.
The real killers, he said, were \"significant management turbulence\" at
the FBI, \"the ever-shifting nature of the requirements,\" and the
agents\' \"trial-and-error, \'We will know it when we see it\' approach
to development.\"

\[6\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[September 2003: Testing Begins]{.underline}

SAIC began testing the program in the fall of 2003, and problems started
cropping up, some of which the agents had warned SAIC about over the
previous summer. SAIC officials complained to Higgins that Computer
Sciences Corp. didn\'t have its hardware and network in place, so SAIC
couldn\'t adequately test the VCF, crucial for a successful flash
cutover. They informed her that they would deliver a version of the VCF
to be in technical compliance with the terms of the contract and that
the FBI should feel free to make changes to it afterward.

[December 2003: VCF Delivery and Failure]{.underline}

On 13 December 2003, SAIC delivered the VCF to the FBI. Under the
direction of the FBI's new interim CIO, Zalmai Azmi, the FBI rejected
SAIC\'s delivery. The bureau found 17 \"functional deficiencies\" it
wanted SAIC to fix before the system was deployed. SAIC argued that at
least some of these deficiencies were changes in requirements. An
arbitrator was called in. The arbitrator found fault with both SAIC and
the FBI. Of the 59 issues and subissues derived from the original 17
deficiencies, the arbitrator found that 19 were requirements
changes---the FBI\'s fault; the other 40 were SAIC\'s errors.

While SAIC fixed bugs, Azmi, with the help of Depew\'s team, created
investigation scenarios that would take different cases from opening to
closing and tested them on the VCF. Those tests revealed an additional
400 deficiencies. \"We have requirements that are not in the final
product, yet we have capabilities in the final product that we don\'t
have requirements for,\" said Azmi.

At the same time Computer Sciences Corp was delivering the final pieces
of equipment to the FBI. By April, 22,251 computer workstations, 3408
printers, 1463 scanners, 475 servers, and new local and wide area
networks would all be up and running, 22 months later than the
accelerated schedule called for.

Azmi and SAIC had yet to agree on the VCF\'s ultimate fate, much less
when it would be deployed. Azmi rejected SAIC's proposal to take one
more year to make all the changes the FBI wanted at the cost of an
additional \$56 million.

[June 2003: Trying to Salvage Something from VCF:]{.underline}

Azmi asked SAIC to take the electronic workflow portion of the VCF, code
that was in relatively good shape, and turn that into what was
eventually called the Initial Operating Capability (IOC), at an
additional fixed price to the FBI of \$16.4 million. SAIC and the FBI
project team had six months to deliver a software package that would be
deployed to between 250 and 500 field personnel in Louisiana and at the
Hoover Building.

\[7\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

The objectives for the new project were clear: test-drive the VCF\'s
electronic workflow; see how people reacted to the graphical user
interface; create a way to translate the output from the VCF forms, into
the ACS system; check out network performance; and develop a training
program. The IOC would use Azmi\'s rigorous approach to software
development and project management, which he called the Life Cycle
Management Directive.

[June 2004: SAIC Begins the IOC Project]{.underline}

With new management in place, about 120 SAIC engineers began work on the
IOC project. The FBI and SAIC agreed to keep to a strict development
schedule, define acceptance criteria, and institute a series of control
gates. Azmi\'s project manager worked closely with his SAIC counterpart
to adhere to the baseline requirements SAIC and the FBI had agreed on
for the IOC in July, thus avoiding a death spiral of change requests. In
January, the IOC was rolled out as a pilot right on schedule.

[March 2005: IOC Pilot Fails]{.underline}

The IOC pilot ended in March. The verdict: \"Although the IOC
application was an aid to task management, its use did not improve the
productivity of most users,\" according to an internal FBI assessment.
FBI employees, particularly agents, found that the IOC actually
increased their workload. Agents filled out forms electronically and
routed them to superiors for approval, after which the electronic form
was uploaded to the ACS, still in use, to be shared with the rest of the
FBI. But to comply with the FBI\'s paper-based records management
system, the form had to be printed out, routed, signed, and filed.

[May 2005: Sentinel Rises from the Ashes of VCF]{.underline}

In May 2005 Mueller announced Sentinel, a four-phase, four-year project
intended to do the VCF\'s job and provide the bureau with a Web-based
case- and records-management system that incorporates commercial
off-the-shelf software.

[March 2006: Sentinel goes to Contract]{.underline}

The FBI awarded a \$305M contract to Lockheed Martin for Sentinel. Total
program cost is estimated to be \$425M.

[June 2007: Sentinel's First Deployment]{.underline}

The FBI and Lockheed Martin rolled out Phase 1 Sentinel capabilities to
the FBI workforce. According to a press release from the FBI phase 1
includes a "Personal Workbox, which summarizes a user's cases and
leads....and user-friendly search capabilities." Azmi announces his
intent to use agile development to deliver new capability to the desktop
every six to nine months.

\[8\]

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image11.png)**Activity (continued)**                        |
|                                                                       |
| **Key Findings**                                                      |
|                                                                       |
| 1.  Technical criteria (be sure to stress performance measures as     |
|     part of this criteria)                                            |
|                                                                       |
|     a.  Email connectivity -- "agents couldn\'t e-mail U.S. Attorney  |
|         offices, federal agencies, local law enforcement, or each     |
|         other; instead, they typically faxed case-related             |
|         information"                                                  |
|                                                                       |
|     b.  Apprval Process Duration -- "They spend a tremendous amount   |
|         of time processing paperwork, faxing and FedEx-ing            |
|         standardized memo and requisition forms through the approval  |
|         chain. Once the appropriate supervisors sign off on the form, |
|         it goes back to the agent, who gives it to a clerk to enter   |
|         into the ACS system. From there, the paper form is filed as   |
|         part of the official record of the case. This system of forms |
|         and approvals stretches back to the 1920s"                    |
|                                                                       |
|     c.  Understanding of Existing System -- "the bureau had neither a |
|         CIO nor documentation detailing its IT systems"               |
|                                                                       |
|     d.  Functionality -- "But the User Applications Component, which  |
|         would ultimately become the VCF, staked out the most          |
|         ambitious goals. First, it was to make the five most heavily  |
|         used investigative applications---the Automated Case Support  |
|         system, IntelPlus, the Criminal Law Enforcement Application,  |
|         the Integrated Intelligence Information Application, and the  |
|         Telephone Application---accessible via a point-and-click Web  |
|         interface. Next, it would rebuild the FBI\'s intranet.        |
|         Finally, it was supposed to identify a way to replace the     |
|         FBI\'s 40-odd investigative software applications, including  |
|         ACS"                                                          |
|                                                                       |
|     e.  Host Modern Software -- "the FBI had 13,000 computers that    |
|         could not run modern software"                                |
|                                                                       |
|     f.  Software Development -- "delivered 700 000 lines of code so   |
|         bug-ridden and functionally off target that the bureau had to |
|         scrap"                                                        |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image12.png)**Activity (continued)**                        |
|                                                                       |
| **Key Findings (continued)**                                          |
|                                                                       |
| 2.  Management criteria                                               |
|                                                                       |
|     a.  Risk Management -- "On 13 December 2003, SAIC delivered the   |
|         VCF to the FBI. Under the direction of the FBI's new interim  |
|         CIO, Zalmai Azmi, the FBI rejected SAIC\'s delivery. The      |
|         bureau found 17 \"functional deficiencies\" it wanted SAIC to |
|         fix before the system was deployed. SAIC argued that at least |
|         some of these deficiencies were changes in requirements. An   |
|         arbitrator was called in. The arbitrator found fault with     |
|         both SAIC and the FBI. Of the 59 issues and subissues derived |
|         from the original 17 deficiencies, the arbitrator found that  |
|         19 were requirements changes---the FBI\'s fault; the other 40 |
|         were SAIC\'s errors"                                          |
|                                                                       |
|     b.  Systems Integration -- "In reaction to the new deadline, SAIC |
|         broke its VCF development group into eight teams, working in  |
|         parallel on different functional pieces of the program. The   |
|         eight threads would later prove difficult for SAIC to combine |
|         into a single system. Nevertheless, Rick Reynolds, vice       |
|         president and operations manager for SAIC, defended the       |
|         decision to change tactics. \"People forget the urgency that  |
|         we were under and our customer was under. And we were right   |
|         beside them,\" he declared. \"We were in the foxhole          |
|         together.\" And "While the FBI and SAIC waited for the        |
|         networks to go live so they could test the VCF on a real      |
|         system, changes and fixes continued. Many of the changes had  |
|         to be to be made by all eight of SAIC\'s development teams.   |
|         Arnold Punaro, SAIC executive vice president and general      |
|         manager, admitted in a posting on the company\'s Web site     |
|         that in the rush to get the program finished by December,     |
|         SAIC didn\'t ensure that all of its programmers were making   |
|         the changes the same way. That inconsistency occasionally     |
|         meant that different modules of the VCF handled data in       |
|         different ways. Consequently, when one module needed to       |
|         communicate with another, errors sometimes occurred"          |
|                                                                       |
|     c.  Scope and Change Management -- "SAIC\'s bid on the original   |
|         contract, and each subsequently revised cost estimate, was    |
|         based on there being \"minimal, minor changes\" to the        |
|         program once a baseline set of requirements had been agreed   |
|         on" and \"the ever-shifting nature of the requirements,\" and |
|         the agents\' \"trial-and-error, \'We will know it when we see |
|         it\' approach to development.\"                               |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image13.png)**Activity (continued)**                        |
|                                                                       |
| **Key Findings (continued)**                                          |
|                                                                       |
| 2.  Management criteria (continued)                                   |
|                                                                       |
|     d.  EVM Management -- "Furthermore, those contracts specified no  |
|         formal project schedules at all, let alone milestones that    |
|         SAIC and DynCorp were contractually obligated to meet on the  |
|         way to final delivery" and "She told Mueller that DynCorp     |
|         didn\'t stand a chance of hitting the delivery target,        |
|         because it didn\'t have a detailed schedule that mapped out   |
|         how it would deploy, integrate, and test the new computers    |
|         and networks"                                                 |
|                                                                       |
|     e.  Project Management                                            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| 3.  Past performance criteria                                         |
|                                                                       |
|     a.  Workflow Systems -- "The VCF system would accept scanned      |
|         documents, photographs, and other electronic media---to       |
|         simplify evidence tracking. People with the proper            |
|         credentials would be able to access that evidence from any    |
|         FBI office. The way work flowed through the bureau would      |
|         change dramatically, too. Instead of filling out a form       |
|         either by hand or in a word-processing program and then       |
|         faxing or FedEx-ing the paper form to a supervisor, an agent  |
|         would fill out a form online and, with a click of the mouse,  |
|         route it to the supervisor. The document would pop up in a    |
|         supervisor\'s in-box, and the agent could track it to see if  |
|         it had been approved. And perhaps most important, information |
|         collected within a case file would eventually be available to |
|         software applications that would compare data among cases to  |
|         search for correlations---to connect the proverbial dots"     |
|                                                                       |
|     b.  Systems Integration -- see 2b.                                |
|                                                                       |
| 4.  Cost criteria                                                     |
|                                                                       |
|     a.  Cost Per User -- "The bureau\'s 12,400 agents work out of 56  |
|         field offices and 400 satellite - or resident agency -        |
|         offices, as well as 51 Legal Attaché offices scattered across |
|         the globe in U.S. embassies and consulates"                   |
|                                                                       |
|     b.  Cost Realism -- "In December 2002, Higgins asked lawmakers to |
|         invest an additional \$137.9 million in Trilogy. Congress     |
|         approved another \$123.2 million for a project whose total    |
|         cost had now ballooned to \$581 million"                      |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 2: Source Selection Criteria (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the source selection criteria. After      |
| addressing any questions the participants may have, inform them that  |
| we will be moving on to the next topic: Risk Assessment.              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  the Source Selection Criteria?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image14.wmf){width="3.957638888888889in"   Slide \# 5-6
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| [Risk]{.underline} is a measure of the inability to achieve overall   |
| program objectives within defined cost, schedule, and performance     |
| constraints. Risk can impact:                                         |
|                                                                       |
| -   Program costs                                                     |
|                                                                       |
| -   Program schedules                                                 |
|                                                                       |
| -   Technical performance                                             |
|                                                                       |
| Risk should be measured in terms of the probability, or likelihood,   |
| of failing to achieve a particular outcome and the consequences, or   |
| impact, of failing to achieve that outcome.                           |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following questions to briefly review and discuss the types   |
| of risk and how it can affect a program. Answers are provided however |
| the participant responses may be broader than these examples.         |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are some risks that could impact |
| program costs?**                                                      |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Risks that could impact program costs include increases in material   |
| prices, higher than anticipated labor rates, and other factors that   |
| can change current program cost estimates. Consider including one     |
| that occurs quite a bit, higher amounts of labor than planned.        |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are some risks that could impact |
| program schedules?**                                                  |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Risks that could impact program schedule include late deliveries,     |
| political pressure, and changing requirements (user needs the system  |
| sooner or user adds requirements).                                    |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are some risks that could impact |
| a program's technical performance?**                                  |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Risks that could impact technical performance include use of new or   |
| exotic materials or processes, use of unproven technology, and use of |
| new applications to meet demanding user requirements.                 |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Risk management is the act or practice of controlling the risk events |
| (possible adverse outcomes) that affect the program. It is the        |
| process of \"crystal-balling\" possible future problems and taking    |
| actions to head off or mitigate those problems. It includes:          |
|                                                                       |
| -   Identifying, analyzing, and tracking risk events                  |
|                                                                       |
| -   Evaluating the likelihood of the occurrence and consequences of   |
|     risk events                                                       |
|                                                                       |
| -   Defining risk-handling plans                                      |
|                                                                       |
| -   Performing continuous assessments and monitoring to determine how |
|     risks change during the life of the program                       |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Risk Management is an integral part of program management             |
| responsibility. It requires all team members to use a disciplined     |
| approach so that risk is reduced to an acceptable level. This is done |
| by assessing and handling the risks associated with the design,       |
| manufacturing, technology, test, and support functions that are part  |
| of systems acquisition.                                               |
|                                                                       |
| A good risk management program can enhance program management         |
| effectiveness and provide managers with an important tool for         |
| reducing a system\'s life cycle costs.                                |
|                                                                       |
| The application of risk management is particularly important during   |
| the early stages of any program as objectives are established, and    |
| the acquisition strategy is developed. All of these activities        |
| require an acceptance of some level of risk and the development of    |
| plans to manage the risk.                                             |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Risk management in a program management office may be either          |
| **centralized** or **decentralized**.                                 |
|                                                                       |
| -   In a **centralized** risk management approach, a dedicated group  |
|     conducts risk management efforts. The program manager establishes |
|     the team and it is responsible for all aspects of risk            |
|     management. The team writes the risk management plan, conducts    |
|     assessments, evaluates risk-handling options, and monitors        |
|     progress. Although this approach may be necessary early in a      |
|     program, it tends to minimize the concept that risk management is |
|     a responsibility shared by all member of the acquisitions team.   |
|                                                                       |
| -   In the **decentralized** risk management approach, assigned       |
|     personnel conduct risk management as part of their normal jobs    |
|     within a program IPT. The decentralized approach is recommended   |
|     because it is compatible with most agencies' acquisition policy   |
|     and generally results in an efficient use of personnel resources. |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The type of organization used will depend in large part on the nature |
| and size of the program and the desires of the program manager.       |
| Regardless of which organization is used, the activities and          |
| responsibilities must be clearly stated and assigned to individuals.  |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The following guidelines apply to **all** risk management             |
| organizations:                                                        |
|                                                                       |
| -   The program manager is ultimately responsible for planning,       |
|     allocating resources and executing risk management.               |
|                                                                       |
| -   The program manager must oversee and participate in the risk      |
|     management process.                                               |
|                                                                       |
| -   Risk management is a team function, and requires the involvement  |
|     of all program management office subordinate organizations to     |
|     avoid confusion, misdirection and wasted resources.               |
|                                                                       |
| Once the risk management organization has been identified, the        |
| program manager is ready to move forward with the task of risk        |
| analysis.                                                             |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Risk analysis is a technical and systematic process to examine        |
| identified risks, isolate causes, determine the relationship to other |
| risks, express the risk rating (e.g., high, medium or moderate, and   |
| low) based on probability and consequences, and prioritize risks in a |
| \"watch\" list.                                                       |
|                                                                       |
| Risk analysis usually begins with a detailed study of the risk events |
| that have been identified to gather information for determining the   |
| probability of each event occurring, and the impact on cost, schedule |
| and performance if it does occur.                                     |
|                                                                       |
| Risk analysis assessments are either:                                 |
|                                                                       |
| -   Qualitative assessments                                           |
|                                                                       |
| -   Quantitative assessments                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to briefly discuss the difference between  |
| the two types of risk assessments: qualitative and quantitative.      |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are the differences between a    |
| qualitative and quantitative risk assessment?**                       |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Qualitative assessments are normally somewhat subjective and are      |
| based on detailed information that may come from such sources as      |
| comparisons with similar systems, lessons learned, experience,        |
| prototype and/or engineering model development, tests,                |
| experts/specialists, and modeling and simulation. Consider adding an  |
| example such as: The odor of the fluid may be objectionable.          |
|                                                                       |
| Quantitative assessments use some form of numerical analysis, such as |
| Monte Carlo simulations, decision analysis, and sensitivity analysis. |
| Consider adding an example such as: The temperature of the device may |
| exceed 150 degrees.                                                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image15.wmf){width="3.957638888888889in"   Slide \# 5-7
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| -   Risk analysis results in the ability to develop a risk rating.    |
|     The risk rating is a measure of the probability of a risk event   |
|     occurring, and subsequent impact if it does occur.                |
|                                                                       |
| -   A simple method for representing risk ratings for risk events is  |
|     a risk matrix similar to the one shown on slide 5-7 (above).      |
|                                                                       |
| -   In the example, the probability or likelihood of the risk event   |
|     occurring is shown on the \"y\" axis, and the impact or           |
|     consequence of its occurrence is shown on the \"x\" axis.         |
|     Consequences will be shown in terms of impacts to cost, schedule  |
|     and performance as applicable. **Note:** *Colors and adjectives   |
|     are commonly used to symbolize risk; not numbers or letters as    |
|     used to create the "x" and "y" axis characteristics on the matrix |
|     graphic.*                                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image16.wmf){width="3.957638888888889in"   Slide \# 5-8
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| When developing a risk rating matrix, the key concept of **risk       |
| tolerance** must be incorporated. This concept can be illustrated as  |
| simply as asking the question, "What color are the boxes in the risk  |
| matrix and why?"                                                      |
|                                                                       |
| For some projects where the risk tolerance is low there may only be   |
| one or a few green boxes in the lower left hand corner of the matrix. |
| *For example, If a schedule is not met the rocket launches without a  |
| product.*                                                             |
|                                                                       |
| Conversely, if the program bears a high tolerance for risk, then a    |
| majority of the boxes would be green in the risk rating matrix.       |
|                                                                       |
| ![](media/image17.png){width="0.6506944444444445in"                   |
| height="0.6791666666666667in"}*For example, if a schedule is not met  |
| the product will have to wait and be shipped an hour later on the     |
| next scheduled truck.*                                                |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image18.wmf){width="3.957638888888889in"   Slide \# 5-9
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The last step in analysis, prioritization, is the ranking of risk     |
| events to identify risk events requiring greater management attention |
| and to help determine the order of risk handling actions. The         |
| prioritization is documented in the watch list.                       |
|                                                                       |
| Risk handling actions fall into one of four responses (as defined in  |
| the 30th Annual Project Management Institute 1999 Seminars &          |
| Symposium):                                                           |
|                                                                       |
| -   Mitigate - seeking to reduce the size of the risk exposure to     |
|     below an acceptable threshold                                     |
|                                                                       |
| -   Transfer - seeking to transfer ownership and/or liability to a    |
|     third party                                                       |
|                                                                       |
| -   Avoid -- seeking to eliminate uncertainty                         |
|                                                                       |
| -   Accept - recognizing residual risks and devising responses to     |
|     control and monitor them                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The generic process described for risk management and analysis is     |
| typically accomplished using any number of standard risk analysis     |
| methods that are appropriate for the situation.                       |
|                                                                       |
| In some cases, one or more techniques could be applied                |
| simultaneously.                                                       |
|                                                                       |
| Standard risk analysis methods include:                               |
|                                                                       |
| -   Process Risk Assessment                                           |
|                                                                       |
| -   Program Documentation Evaluation Risk Identification              |
|                                                                       |
| -   Threat and Requirements Risk Assessment                           |
|                                                                       |
| -   Product (Work Breakdown Structure {WBS}) Risk Assessment          |
|                                                                       |
| -   Cost Risk Assessment                                              |
|                                                                       |
| -   Quantified Schedule Risk Assessment                               |
|                                                                       |
| -   Expert Interviews                                                 |
|                                                                       |
| -   Analog Comparison/Lesson Learned Studies                          |
|                                                                       |
| -   Risk Prioritization and Risk Aggregation                          |
|                                                                       |
| Each of these methods is discussed further in this topic.             |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Tell participants this listing of risk analysis methods is found in   |
| their Participant Guide.                                              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Process Risk Assessment** is used to assess (identify and analyze)  |
| the technical risks resulting from the contractor's processes. It is  |
| based on describing the risk areas contained in the various technical |
| processes, e.g., design, test and production, and specifying the      |
| methods for reducing risk in each area.                               |
|                                                                       |
| It should be used in preparation for major milestone acquisition      |
| decisions and source selection. It can be used independently or in    |
| conjunction with other risk assessment techniques.                    |
|                                                                       |
| The Pros and Cons for Process Risk Assessment are:                    |
|                                                                       |
| -   Pros -- Addresses pervasive and important sources of risk in most |
|     acquisition programs; uses fundamental engineering principles and |
|     proven procedures to reduce technical risks                       |
|                                                                       |
| -   Cons -- Success of risk reduction efforts will depend on          |
|     contractor's willingness and ability to make an effort to replace |
|     any deficient engineering practices and procedures with best      |
|     industrial practices                                              |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used process risk assessment?**          |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Program Documentation Evaluation Risk Identification** provides a   |
| methodology for the comparison of key acquisition documents and plans |
| to ensure they are consistent and traceable to one another.           |
|                                                                       |
| If the content (activities, schedules, requirements, specifications,  |
| etc.) do not flow from or support the contents of those above, below  |
| or adjacent to it, there is a strong chance that risk will be         |
| introduced into the acquisition or that known risks will not be       |
| adequately addressed.                                                 |
|                                                                       |
| The Pros and Cons for Program Documentation Risk Identification are:  |
|                                                                       |
| -   Pros -- Reduces the risk that known risks will not be adequately  |
|     addressed; improves the quality of the acquisition documentation  |
|                                                                       |
| -   Cons -- Comparison of acquisition documentation and plans should  |
|     be performed by a small team of who are intimately familiar with  |
|     the content of the total acquisition                              |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used program documentation evaluation    |
| risk identification?**                                                |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Threat and Requirements Risk Assessment** describes an approach to  |
| assess risk associated with requirements and threats. It identifies   |
| those requirements and threat elements that are risk drivers.         |
|                                                                       |
| The requirements assessment process focuses on: determining if        |
| operational requirements are properly established, ensuring that      |
| requirements are stable and achievable, the operating environment is  |
| adequately described, addressing logistics and suitability needs, and |
| determining if requirements are too constrictive, thereby identifying |
| a specific solution.                                                  |
|                                                                       |
| The Pros and Cons for Threat and Requirements Risk Assessment are:    |
|                                                                       |
| -   Pros -- Process addresses uncertainty in threat accuracy and      |
|     stability, sensitivity of design and technology to threat,        |
|     vulnerability of the system to threat countermeasures, and        |
|     vulnerability of the program to intelligence penetration          |
|                                                                       |
| -   Cons -- User must be actively involved in the process from the    |
|     beginning in order to gain a complete understanding of the        |
|     requirements and threat to preclude misunderstandings and help    |
|     identify risk areas and corresponding early risk planning         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used threat and requirements risk        |
| assessment?**                                                         |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Product (WBS) Risk Assessment** identifies those risks associated   |
| with a given system concept and design. It differs from the process   |
| technique as it focuses on the resulting product versus the           |
| contractor's engineering and manufacturing processes.                 |
|                                                                       |
| The technique is used to identify and analyze risk in the following   |
| critical areas: design and engineering, technology, logistics,        |
| concurrency and manufacturing.                                        |
|                                                                       |
| The project WBS serves as the starting Point to describe contract     |
| work that will be done, and the resulting product, and determines the |
| critical risk events in each risk area.                               |
|                                                                       |
| The Pros and Cons for Product (WBS) Risk Assessment are:              |
|                                                                       |
| -   Pros -- Can be used independently or in conjunction with other    |
|     techniques (such as Process Risk Assessment technique); may be    |
|     used to establish a performance measure baseline for the contract |
|                                                                       |
| -   Cons -- Program IPTs need to the risk assessments and handling    |
|     strategies for risks in their portion of the WBS; may be          |
|     necessary to include outside industry experts on teams to aid in  |
|     examination of WBS elements or functional areas                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used product (WBS) risk assessment?**    |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Cost Risk Assessment** provides an acquisition-level cost estimate  |
| at completion (EAC) that is a function of performance and schedule    |
| risks.                                                                |
|                                                                       |
| It uses the results of prior low-level WBS elements and their         |
| activities risk assessments, and cost probability distributions       |
| developed for each of the elements' activities.                       |
|                                                                       |
| A probability distribution of the program-level cost EAC probability  |
| distribution function is determined, and analyzed to determine the    |
| actual risk of cost overruns and identify the cost risk drivers.      |
|                                                                       |
| The Pros and Cons of Cost Risk Assessment are:                        |
|                                                                       |
| -   Pros -- Technique can be used in any of the acquisition phases;   |
|     results in a more realistic EAC than single Point WBS elements    |
|     estimates; eliminates a major cause of misunderstanding (use of   |
|     Point estimates); permits identification of technical performance |
|     or schedule causes of cost risk; provides a sound basis for the   |
|     determination of an "acceptable" level of cost risk               |
|                                                                       |
| -   Cons -- Requires close and continuous cooperation between cost    |
|     analysts and knowledgeable technical personnel; requires support  |
|     of contractor's senior management to help get valid cost data     |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used cost risk assessment?**             |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Quantified Schedule Risk Assessment** provides a means to determine |
| acquisition-level schedule risk as a function of the technical        |
| performance and cost risk associated with the various activities that |
| comprise the program.                                                 |
|                                                                       |
| It estimates the acquisition-level schedule by developing probability |
| distributions for each activity's duration, and aggregating those     |
| distributions using analytical tools.                                 |
|                                                                       |
| The resulting schedule is then analyzed to determine the actual       |
| schedule risk and to identify the schedule risk drivers.              |
|                                                                       |
| The Pros and Cons of Quantified Schedule Risk Assessment are:         |
|                                                                       |
| -   Pros -- Managers are able to estimate early in an acquisition if  |
|     there is a significant likelihood of overruning the schedule and  |
|     by how much; identifies activities that are on the highest risk   |
|     path; can be used at any phase during the acquisition             |
|                                                                       |
| -   Cons -- Probability function distribution key must be developed   |
|     for each activity as soon as the activity is included in the      |
|     master schedule; functions should be reviewed and revised         |
|     periodically; should be used by a small team of schedule analysts |
|     and technical experts that understand the significance of prior   |
|     risk performance assessments                                      |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used quantified scheduled risk           |
| assessment?**                                                         |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Expert Interviews** provides a means for collecting relevant        |
| risk-related data from subject matter experts and from personnel who  |
| are intimately involved with various aspects of the acquisition.      |
|                                                                       |
| This technique is used almost exclusively in a support role to help   |
| develop technical data required by a primary risk assessment          |
| technique, e.g., Product (WBS) Risk Assessment technique.             |
|                                                                       |
| The Pros and Cons of Expert Interviews are:                           |
|                                                                       |
| -   Pros -- Can address all functional areas that make up critical    |
|     risk areas; can be used in support of risk handling; sound and    |
|     practical means of obtaining information that is not available    |
|     from prior acquisitions                                           |
|                                                                       |
| -   Cons -- Expert opinions may be biased because of over-reliance on |
|     certain information and neglect of other information; unwarranted |
|     confidence; tendency to recall most frequent and recent events;   |
|     tendency to neglect rare events; ulterior motivation              |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used expert interviews as a means of     |
| risk assessment?**                                                    |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Analog Comparison/Lessons Learned Study** uses lessons learned and  |
| historical information about the risk associated with similar         |
| acquisitions to identify the potential risks.                         |
|                                                                       |
| It is normally used to support other primary risk assessment          |
| techniques, e.g., Product (WBS) Risk Assessment technique.            |
|                                                                       |
| The Pros and Cons of Analog Comparison/Lessons Learned Study are:     |
|                                                                       |
| -   Pros -- Key insights can be gained by examining the successes,    |
|     failures, problems and solutions of similar acquisitions          |
|     (existing or past); addresses all the functional areas that make  |
|     up the critical risk areas and processes                          |
|                                                                       |
| -   Cons -- Should be used in conjunction with other primary risk     |
|     assessment techniques; "assumes" that new acquisitions are        |
|     evolved from existing components or subsytems of other            |
|     acquisitions                                                      |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used an analog comparison / lessons      |
| learned risk assessment?**                                            |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| **Risk Prioritization/Risk Aggregation** techniques provide a means   |
| to prioritize the risks present in an acquisition and to roll up      |
| lower-level risks into a meaningful value at the critical risk area   |
| and process level.                                                    |
|                                                                       |
| Risk prioritization provides the basis for development of risk        |
| handling plans, preparation of a risk handling task sequence list,    |
| and allocation of risk handling resources.                            |
|                                                                       |
| Risk aggregation presents a higher level view of risk that may be     |
| used by senior management to assess the acquisition's present and     |
| future status.                                                        |
|                                                                       |
| The Pros and Cons of Risk Prioritization and Risk Aggregation are:    |
|                                                                       |
| -   Pros -- Use of subject matter experts in a qualitative approach   |
|                                                                       |
| -   Cons -- Require definitive criteria to evaluate the risks such as |
|     probability of failure (PF), consequence of failure (CF) or any   |
|     other such appropriate factors                                    |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following question to allow participants to illustrate the    |
| use of this risk analysis method based on prior experience.           |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**Who can provide an example of a       |
| program or project where you used risk prioritization / risk          |
| aggregation?**                                                        |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use the participant response(s) to explore how/when the risk analysis |
| method was applied and some "lessons learned" from the example.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image19.png) **Activity**                                   |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity provides participants with an opportunity to apply risk |
| assessment techniques that would be appropriate for the Virtual Case  |
| File (VCF) acquisition business case.                                 |
|                                                                       |
| Activity Length: 40 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Divide the participants into small team groups.                   |
|                                                                       |
| 2.  List the 8 risk assessment techniques: process risk assessment,   |
|     program documentation evaluation risk identification, threat and  |
|     requirements risk assessment, product (WBS) risk assessment, cost |
|     risk assessment, quantified schedule risk assessment, expert      |
|     interviews, analog comparison/lessons learned study, risk         |
|     prioritization and risk aggression.                               |
|                                                                       |
| 3.  Allow participants 5 minutes to individually review the *VCF      |
|     Acquisition Program Management Business Case* found in their      |
|     Participant Guide.                                                |
|                                                                       |
| 4.  Allow each group 15 minutes to answer the following questions.    |
|                                                                       |
|     a.  What would have been an effective risk assessment technique   |
|         for the program manager to use in the Summer 2002? Provide a  |
|         rationale using information from the case.                    |
|                                                                       |
|     b.  Using the selected technique what risks are identified? Plot  |
|         the identified risks on a Liklihood/Consequence risk rating   |
|         matrix.                                                       |
|                                                                       |
| 5.  Have a spokesperson from one group identify the selected          |
|     technique and rationale, have them present the list of identified |
|     risks, and have them present the risk rating matrix.              |
|                                                                       |
| 6.  If necessary, elaborate on the selected technique and rational to |
|     ensure correct information is provided for the whole group.       |
|     Encourage other groups to offer commenrts and to offer            |
|     alternative risk assessment techniques.                           |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image20.png) **Activity (continued)**                       |
|                                                                       |
| 7.  Use the answers given to generate a large group discussion to     |
|     elicit other opinions about the question. There may be more than  |
|     one correct answer or viewpoint for the question.                 |
|                                                                       |
| 8.  Plot the risk examples on a flip chart to create a discussion     |
|     listing of risks. As part of the discussion, be sure to clarify   |
|     the difference between risk events vs. the results of risk        |
|     events.                                                           |
|                                                                       |
| 9.  Identify the most likely risk response for each of the risks that |
|     are identified by participants: avoid, transfer, mitigate or      |
|     accept. Discuss the rationale for the suggested response(s).      |
|                                                                       |
| > **Note:** *Relevant example answers for each question are found on  |
| > the following page of this Instructor Guide.*                       |
|                                                                       |
| 10. After all the questions have been discussed, summarize the        |
|     activity by telling participants that there is not necessarily a  |
|     "right and wrong" when selecting and using a risk assessment      |
|     technique. The important aspect is to make sure that the program  |
|     manager exercises due diligence when identifying and managing     |
|     program risks.                                                    |
|                                                                       |
|     In this project's case, what probably should happen is that it    |
|     should be rejected because there is too much risk. However, for   |
|     purposes of this course the discussion should focus on actions to |
|     mitigate risk if a project is forced on the project manager; the  |
|     risk analysis should be started over because the program manager  |
|     does not know that the appropriate measures were taken prior to   |
|     the project's start.                                              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image21.png) **Activity (continued)**                       |
|                                                                       |
| **Key Findings (continued)**                                          |
|                                                                       |
| The following is a sample of a risk assessment technique and          |
| rationale for for the program manager to use in the Summer 2002.      |
|                                                                       |
| -   The *Analog Comparison/Lessons Learned Study* technique has been  |
|     selected for the program manager to use in the Summer of 2002     |
|     because of it's application to situations where information about |
|     past similar situations is available -- in this case the          |
|     preceeding history of VCF and Trilogy.                            |
|                                                                       |
| -   Note that the pros for this technique are -- Key insights can be  |
|     gained by examining the successes, failures, problems and         |
|     solutions of similar acquisitions (existing or past); addresses   |
|     all the functional areas that make up the critical risk areas and |
|     processes.                                                        |
|                                                                       |
| -   Note that the cons for this technique are - Should be used in     |
|     conjunction with other primary risk assessment techniques;        |
|     "assumes" that new acquisitions are evolved from existing         |
|     components or subsytems of other acquisitions.                    |
|                                                                       |
| The following is a sample of identified risks using the selected      |
| technique and a plot of the identified risks on a                     |
| Liklihood/Consequence risk rating matrix                              |
|                                                                       |
| 1.  Requirements creep -- "800-plus pages of requirements"            |
|                                                                       |
| 2.  Uncertain architecture -- "trying to design the system layout and |
|     then the whole application logic before they had actually even    |
|     figured out what they wanted the system to do"                    |
|                                                                       |
| 3.  Schedule inaccuracy -- "didn\'t stand a chance of hitting the     |
|     delivery target, because it didn\'t have a detailed schedule that |
|     mapped out how it would deploy, integrate, and test the new       |
|     computers and networks"                                           |
|                                                                       |
| 4.  Inexperienced program management -- "appoint Depew, who had no IT |
|     project management experience, the VCF project manager"           |
|                                                                       |
| 5.  Schedule pressure -- "agreed to deliver the initial version of    |
|     the VCF in December 2003 instead of June 2004"                    |
|                                                                       |
| 6.  Baseline lacking -- "With no detailed description of the FBI\'s   |
|     processes and IT infrastructure to go by"                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image22.png) **Activity (continued)**                       |
|                                                                       |
| **Key Findings**                                                      |
|                                                                       |
| (continued from prior page)                                           |
|                                                                       |
|                                                                       |
| --------- --------- --------- --------- --------- --------- --------- |
|             **A**                         **1, 5**                    |
|                                                                       |
|             **B**                                   **2,3**   **4**   |
|                                                                       |
|             **C**                         **6**                       |
|                                                                       |
|             **D**                                                     |
|                                                                       |
|             **E**                                                     |
|                                                                       |
|                       **1**     **2**     **3**     **4**     **5**   |
|                                                                       |
|                                                                       |
|                                                                       |
| --------- --------- --------- --------- --------- --------- --------- |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 3: Risk Assessment (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on risk assessment After addressing any      |
| questions the participants may have, inform them that we will be      |
| moving on to the next topic: Best Value Selection                     |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  risk assessment?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image23.wmf){width="3.957638888888889in"   Slide \# 5-10
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Per FAR 2.101, "Best Value" selection means that the expected outcome |
| of an acquisition that, in the agency's estimation, provides the      |
| greatest overall benefit in response to the requirement.              |
|                                                                       |
| In years past, the emphasis has been to buy the lowest priced         |
| item/service.                                                         |
|                                                                       |
| For some situations, buying lowest priced is still the approach to    |
| take, however for many of the buys, the PCO is required to look at a  |
| range of factors to determine which offer truly represents the best   |
| value.                                                                |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.40555555555555556in"                   |
| height="0.6354166666666666in"}**Instructor Notes:**                   |
|                                                                       |
| Ask the following questions to briefly review the definition of best  |
| value and some of the challenges associated with it.                  |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**How would you describe "best          |
| value"?**                                                             |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| \"Best value\" is a process used to select the most advantageous      |
| offer by evaluating and comparing factors in addition to cost or      |
| price.                                                                |
|                                                                       |
| It allows flexibility in selection through tradeoffs which the agency |
| makes between the cost and non-cost evaluation factors with the       |
| intent of awarding to the contractor that will give the government    |
| the greatest or best value for its money.                             |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are some of the challenges that  |
| get in the way of the "best value" selection process? Why?**          |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Discuss participant responses. Create a flip chart listing of the     |
| issues and challenges they perceive to be associated with best value. |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Best value in negotiated acquisitions can be obtained by using any    |
| one or a combination of source selection techniques involving the     |
| desired performance, the delivery (schedule) or the cost of the       |
| product or service being acquired. The relative importance of cost or |
| price may vary in different types of acquisitions:                    |
|                                                                       |
| -   When the requirement is clearly definable and the risk of         |
|     unsuccessful contract performance is minimal, cost or price may   |
|     play a dominant role in source selection.                         |
|                                                                       |
| -   When the requirement is less definitive and the risk is great,    |
|     then technical and past performance considerations play a more    |
|     dominant role.                                                    |
|                                                                       |
| The amount of risk (as discussed on Topic 3 of this module)           |
| anticipated is a primary determiner of the source selection approach. |
| The expected outcome is an acquisition that, in the SSA's opinion,    |
| provides the greatest overall benefit towards fulfilling the agency's |
| requirement.                                                          |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| According to FAR 15.101, Best Value Continuum                         |
|                                                                       |
| *"An agency can obtain best value in negotiated acquisitions by using |
| any one or a combination of source selection approaches. In different |
| types of acquisitions, the relative importance of cost or price may   |
| vary. For example, in acquisitions where the requirement is clearly   |
| definable and the risk of unsuccessful contract performance is        |
| minimal, cost or price may play a dominant role in source selection.  |
| The less definitive the requirement, the more development work        |
| required, or the greater the performance risk, the more technical or  |
| past performance considerations may play a dominant role in source    |
| selection."*                                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The General Accounting Office (GAO) acknowledges broad agency         |
| discretion in selection; therefore, the integrated product team       |
| evaluators and the source selection authority should expect to        |
| exercise good judgment.                                               |
|                                                                       |
| Quite simply, best-value source selection involves subjective         |
| analysis. It cannot be reduced to a mechanical, mathematical          |
| exercise.                                                             |
|                                                                       |
| -   Source selection officials have broad discretion to determine the |
|     manner and extent to which they will make use of the technical    |
|     and price evaluation results in negotiated procurements.          |
|                                                                       |
| -   In deciding between competing proposals, price/technical          |
|     tradeoffs may be made; the propriety of such tradeoffs turns not  |
|     on the difference in technical scores or ratings per se, but on   |
|     whether the source selection official\'s judgment concerning the  |
|     significance of that difference was reasonable and adequately     |
|     justified in light of the RFP evaluation scheme.                  |
|                                                                       |
| -   The discretion to determine whether the technical advantages      |
|     associated with a higher-priced proposal are worth the price      |
|     premium exists notwithstanding the fact that price is equal to or |
|     more important than other factors in the evaluation scheme.       |
|                                                                       |
| -   In a best value procurement, an agency\'s selection of a          |
|     higher-priced, higher-rated offer should be supported by a        |
|     determination that the technical superiority of the higher-priced |
|     offer warrants the additional cost involved.                      |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image24.png) **Activity**                                   |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity provides participants with an opportunity to identify   |
| common barriers to best value selection and identify types of         |
| resolutions to those barriers.                                        |
|                                                                       |
| Activity Length: 30 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Divide the participants into small groups.                        |
|                                                                       |
| 2.  Tell groups they will be given 10 minutes to identify a listing   |
|     of barriers they've experienced for best value selection and to   |
|     discuss each as a small group to develop solutions to those       |
|     barriers identified:                                              |
|                                                                       |
|     a.  "What are some of the barriers to performing a best-value     |
|         source selection?"                                            |
|                                                                       |
| 3.  Have a volunteer from a randomly selected group identify one      |
|     barrier for best value selection, describing the context in which |
|     the barrier was presented and provide the corresponding solutions |
|     the group identified for the barrier. Ask the whole group to add  |
|     to the solutions.                                                 |
|                                                                       |
| 4.  Use a flip chart to record the barriers that are identified.      |
|                                                                       |
| 5.  Repeat this process from group to group until there are no other  |
|     barriers.                                                         |
|                                                                       |
| 6.  If necessary, elaborate on the barriers and solution findings to  |
|     ensure correct information is provided for the whole group.       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection (continued)**

  -----------------------------------------------------------------------

  ------------------------------------------------------ ----------------
  ![](media/image25.wmf){width="3.957638888888889in"     Slide \# 5-11
  height="2.975in"}                                      

  ------------------------------------------------------ ----------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| Use the chart above to discuss the question posed below. Use          |
| participant comments to facilitate a group discussion about the use   |
| of best value procurement.                                            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**When is it appropriate to use best
  value as the aqcuisition procurement?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 4: Best Value Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the best value selection After addressing |
| any questions the participants may have, inform them that we will be  |
| moving on to the next topic: Formal Source Selection                  |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  the best value selecton?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image26.wmf){width="3.957638888888889in"   Slide \# 5-12
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| There are three basic levels of authority prescribed for a Source     |
| Selection Organization (SSO):                                         |
|                                                                       |
| -   Source Selection Evaluation Board (SSEB)                          |
|                                                                       |
| -   Source Selection Advisory Council (SSAC)                          |
|                                                                       |
| -   Source Selection Authority (SSA)                                  |
|                                                                       |
| The **SSEB** performs an in-depth and systematic evaluation of the    |
| proposals against the evaluation factors and sub-factors set forth in |
| the solicitation.                                                     |
|                                                                       |
| Each proposal is assessed on a stand-alone basis, with the outcome    |
| being to provide the **SSA** with sufficient information to make an   |
| informed and logical selection decision.                              |
|                                                                       |
| In addition to the SSEB, the **SSAC** must provide complete, coherent |
| and fully-supported written narrative justifications for the findings |
| and conclusions reached during the source selection process.          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| FAR 15.308 states that                                                |
|                                                                       |
| *"The SSA's decision shall be based on a comparative assessment of    |
| proposals against all source selection criteria in the solicitation.  |
| While the SSA may use reports and analyses prepared by others, the    |
| source selection decision shall represent the SSA's independent       |
| judgment. The source selection decision shall be documented, and the  |
| documentation shall include the rationale for any business judgments  |
| and tradeoffs made or relied on by the SSA, including benefits        |
| associated with additional costs. Although the rationale for the      |
| selection decision must be documented, that documentation need not    |
| quantify the tradeoffs that led to the decision."*                    |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image27.png) **Activity**                                   |
|                                                                       |
| Activity Purpose:                                                     |
|                                                                       |
| This activity provides participants with an opportunity to consider   |
| several proposals offered in response to the solicitation for the VCF |
| Business Case Study and to select the one which represents the best   |
| value.                                                                |
|                                                                       |
| *This module activity continues the business case that will be used   |
| throughout this course to apply various concepts and reinforce        |
| learning through the use of critical thinking and application         |
| skills.*                                                              |
|                                                                       |
| Activity Length: 90 minutes                                           |
|                                                                       |
| Activity Instructions:                                                |
|                                                                       |
| 1.  Divide the participants into small groups.                        |
|                                                                       |
| 2.  Allow participants 5 minutes to individually review the *VCF      |
|     Acquisition Program Management Business Case* found in their      |
|     Participant Guide and on the following pages of this Instructor   |
|     Guide..                                                           |
|                                                                       |
|     a.  Have participants develop a brief Statement of Work (SOW)     |
|         categorized into "Should", "Will" or "Must." Participants     |
|         should identify at least two or three elements in each        |
|         category for their SOW.                                       |
|                                                                       |
| 3.  Allow each group 20 minutes to.                                   |
|                                                                       |
|     a.  Formulate a company name and write a proposal using           |
|         information fron the case for submission to the FBI to be     |
|         evaluated in a best value acquisition leading to a May 2005   |
|         award? The proposal is to be legibly written on one side of a |
|         single sheet of 8.5x11 paper.                                 |
|                                                                       |
| 4.  Copy and distribute each proposal to all groups.                  |
|                                                                       |
| 5.  Allow each group 20 minutes to.                                   |
|                                                                       |
|     a.  Evaluate all proposals using best value acquisition and       |
|         select one proposal for negotiation for award. Provide a      |
|         rationale for the selection.                                  |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image28.png) **Activity**                                   |
|                                                                       |
| Activity Instructions (continued):                                    |
|                                                                       |
| 6.  Solicit the selections and rational from the groups during a      |
|     class-wide discussion and allow individuals to answer on a        |
|     volunteer basis. Use the "Should", "Will" or "Must"               |
|     categorizations to further clarify rationales.                    |
|                                                                       |
| 7.  If necessary, elaborate on the best value acquisition to ensure   |
|     correct information is provided for the whole group.              |
|                                                                       |
| > **Note:** *This activity is targeted at having the students develop |
| > an understanding of best value acquisition by exposing them to both |
| > the contractor and government perspective. As such it has no sample |
| > answer.*                                                            |
|                                                                       |
| 8.  After all the questions have been discussed, summarize the        |
|     activity by telling participants that there is a considerable     |
|     amount benefit to a program by using best value acquisition.      |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study**

[July 2000: Trying to Bring Order to Chaos (continued)]{.underline}

In 2000 the bureau finally began to deal with its outdated IT systems.
At the time, under the direction of Louis J. Freeh, the bureau had
neither a CIO nor documentation detailing its IT systems, much less a
plan for revamping them. The task of creating such a plan fell to former
IBM executive Bob E. Dies, who became assistant director in charge of
the FBI Information Resources Division on 17 July 2000. He was the first
of five officials who, over the next four years, would struggle to lead
the FBI\'s sprawling and antiquated information systems and get the VCF
project under way.

In September 2000, Congress approved \$379.8 million over three years
for what was then called the FBI Information Technology Upgrade Project.
Eventually divided into three parts, the program became known as
Trilogy. The Information Presentation Component would provide all 56 FBI
field offices, some 22,000 agents and support staff, with new Dell
Pentium PCs running Microsoft Office, as well as new scanners, printers,
and servers. The Transportation Network Component would provide secure
local area and wide area networks, allowing agents to share information
with their supervisors and each other.

But the User Applications Component, which would ultimately become the
VCF, staked out the most ambitious goals. First, it was to make the five
most heavily used investigative applications---the Automated Case
Support system, IntelPlus, the Criminal Law Enforcement Application, the
Integrated Intelligence Information Application, and the Telephone
Application---accessible via a point-and-click Web interface. Next, it
would rebuild the FBI\'s intranet. Finally, it was supposed to identify
a way to replace the FBI\'s 40-odd investigative software applications,
including ACS.

[Spring 2001: Trilogy Contracts Awarded]{.underline}

In May and June 2001, the bureau awarded Trilogy contracts to two major
U.S. government contractors: DynCorp, of Reston, Va., for the hardware
and network projects, and to SAIC for software. All three Trilogy
components were to be delivered by the middle of 2004. The FBI used
cost-plus-award fee contracts for the hardware, networks, and software.

\[1\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[September 2001: Inadequate System Exposed; the Virtual Case File is
Born]{.underline}

On 4 September 2001, Robert S. Mueller III became the tenth director in
FBI history. One week later, terrorists pulverized New York City\'s
World Trade Center and a piece of the Pentagon. The inability of FBI
agents to share the most basic information about Al Qaeda\'s U.S.
activities blew up into a front-page scandal. Within days, the FBI\'s
pathetic technology infrastructure went from being so much arcane trivia
to a subject of daily fulmination by politicians and newspaper
columnists. In the face of intense public and congressional pressure,
Mueller shifted Trilogy into high gear.

Recognizing the limitations of the new web interface feature of Trilogy
and ACS, Robert J. Chiaradio the FBI\'s executive assistant director for
administration decided that the Bureau needed an entirely new database,
graphical user interface, and applications, which would let agents
search across various investigations to find relationships to their own
cases. The new case management system would host millions of records
containing information on everything from witnesses, suspects, and
informants to evidence such as documents, photos, and audio recordings.
To address concerns being raised by intelligence experts and lawmakers
in the wake of 9/11, these records would be accessible to both the
FBI\'s agents and its intelligence analysts. The new system was dubbed
the Virtual Case File.

With no detailed description of the FBI\'s processes and IT
infrastructure to go by (i.e., an enterprise architecture), a team of
FBI agents led by Larry Depew began to characterize investigative
processes such as witness interviews and surveillance operations and map
them to the FBI\'s software and databases. Depew, a veteran Special
Agent from the Trenton, New Jersey office, was a self taught database
programmer. Over a six-week period in the fall of 2001, the group
defined how agents worked, how they gathered information, and how that
information was fed into ACS. Working with engineers from SAIC, they
drew up diagrams and flowcharts of how the case management system
operated then and how they wanted the new case management system, the
VCF, to operate in the future.

\[2\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[December 2001: Contract Changes and an Accelerated
Schedule]{.underline}

In December 2001, the FBI asked SAIC to stop building a Web front end
for the old programs. Instead, SAIC was asked to devise a new
application, database, and graphical user interface to completely
replace ACS.

To formally define what users needed the VCF to do for them, SAIC
embarked on a series of Joint Application Development (JAD) sessions. In
these meetings, the FBI's team of agents and experts got together with a
group of SAIC engineers to hash out what functions the VCF would
perform. Ideas captured in these sessions formed the basis of the
requirements document that guided SAIC\'s application designers and
programmers.

In January 2002, the FBI requested an additional \$70 million to
accelerate Trilogy; Congress went further, approving \$78 million.
DynCorp committed to delivering its two components by July 2002. SAIC
agreed to deliver the initial version of the VCF in December 2003
instead of June 2004.

SAIC and the FBI were now committed to creating an entirely new case
management system in 22 months. While the Trilogy contracts were changed
to reflect the aggressive new deadlines, neither the original software
contract nor the modified one specified any formal criteria for the FBI
to use to accept or reject the finished VCF software. Furthermore, those
contracts specified no formal project schedules at all, let alone
milestones that SAIC and DynCorp were contractually obligated to meet on
the way to final delivery.

In reaction to the new deadline, SAIC broke its VCF development group
into eight teams, working in parallel on different functional pieces of
the program. The eight threads would later prove difficult for SAIC to
combine into a single system. Nevertheless, Rick Reynolds, vice
president and operations manager for SAIC, defended the decision to
change tactics. \"People forget the urgency that we were under and our
customer was under. And we were right beside them,\" he declared. \"We
were in the foxhole together.\"

[March 2002: Defining Requirements]{.underline}

Over a six-month period, the JAD team met in two-week sessions. After a
two-week JAD session finished, a two-week feedback cycle would begin.
SAIC provided Depew\'s team with information gleaned from the session,
including needs statements, flow charts, and meeting minutes. Depew\'s
team reviewed these materials and gave SAIC feedback while
simultaneously preparing subject matter experts for the next round of
JAD sessions, which immediately followed the feedback cycle. There were
no breaks.

\[3\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

Mueller, Dies, and Chiaradio recruited C.Z. (\"Sherry\") Higgins, a
seasoned IT program manager with experience at Lucent and AT&T, to be
the FBI's new project management executive for the Office of the
Director. The new Office of Program Management would centralize IT
management and oversee, develop, and deploy the bureau\'s most
expensive, complex, and risky projects. But her most important
assignment was to manage Trilogy. Her first move when she came on board
in March 2002 was to appoint Depew, who had no IT project management
experience, the VCF project manager.

[April 2002: Schedules Start to Slip]{.underline}

In mid-April 2002, Higgins, demanded a detailed schedule from DynCorp.
After she got it, she pulled the project teams from the FBI and DynCorp
into a meeting and went through the document. Shortly after that,
Higgins broke the news to the director: the computers and networks would
not be delivered in July of that year as had been scheduled. She told
Mueller that DynCorp didn\'t stand a chance of hitting the delivery
target, because it didn\'t have a detailed schedule that mapped out how
it would deploy, integrate, and test the new computers and networks.

[July 2002: VCF Looked Good on Paper]{.underline}

The JAD sessions had produced an exhaustively detailed requirements
document. This plan for a case-management system would combine the ACS
with two other systems: the Telephone Application, the bureau\'s central
repository of telephone records related to investigations, and parts of
the Criminal Law Enforcement Application, a repository for investigative
data about people, organizations, locations, vehicles, and
communications.

The VCF system would accept scanned documents, photographs, and other
electronic media---to simplify evidence tracking. People with the proper
credentials would be able to access that evidence from any FBI office.
The way work flowed through the bureau would change dramatically, too.
Instead of filling out a form either by hand or in a word-processing
program and then faxing or FedEx-ing the paper form to a supervisor, an
agent would fill out a form online and, with a click of the mouse, route
it to the supervisor. The document would pop up in a supervisor\'s
in-box, and the agent could track it to see if it had been approved. And
perhaps most important, information collected within a case file would
eventually be available to software applications that would compare data
among cases to search for correlations---to connect the proverbial dots.

\[4\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[Summer 2002: Detailed Requirements]{.underline}

By August 2002 SAIC had around 200 programmers on the job. Matthew
Patton joined the SAIC team as a security engineer. Patton, who was
later removed from the project after criticizing its management in an
Internet posting, was unimpressed by the 800-plus pages of requirements.
\"In a requirements document, you want to dictate the whats, not the
hows,\" Patton said. \"We need an e-mail system that can do x, and
there\'s 12 bullets. Instead, we had things like \'there will be a page
with a button that says e-mail on it.\' We want our button here on the
page or we want it that color. We want a logo on the front page that
looks like x. We want certain things on the left-hand side of the page.
They were trying to design the system layout and then the whole
application logic before they had actually even figured out what they
wanted the system to do.\"

The overly specific nature of the requirements focused developers on
their tiny piece of the puzzle. They were writing code, Patton said,
with no idea of how their piece fit with the others. This presaged the
integration problems that would later plague the project.

Patton also claimed that SAIC was determined to write much of the VCF
from scratch. This included an e-mail-like system that at least one
team, to his knowledge, was writing, even though the FBI was already
using an off-the-shelf software package, Novell\'s GroupWise, for
e-mail.

[December 2002: The March of the Change Requests]{.underline}

In December 2002, Higgins asked lawmakers to invest an additional
\$137.9 million in Trilogy. Congress approved another \$123.2 million
for a project whose total cost had now ballooned to \$581 million.

After the FBI and SAIC agreed to the baseline set of requirements, SAIC
programmers began cranking out code. The company had settled on a spiral
development methodology, an iterative approach to writing software.
Basically, SAIC programmers would write and compile a block of code that
performed a particular function, then run it to show Depew\'s agents
what it would do. The agents gave the programmers feedback, and the
programmers tried to incorporate the suggested changes. Sometimes
Depew\'s team had only two days to review a batch of code. Agents would
pull all-nighters to get the evaluation finished, \"and in the next
iteration their comments wouldn\'t be taken into account,\" Higgins
said. Sometimes, she acknowledged, these evaluations would include
changes to the requirements---functions that the agents had decided that
they needed once they saw what they were going to get. Other times the
FBI team would find bugs that needed to be fixed.

\[5\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

If there was a dispute as to whether the change could or should be made,
the agents sent an official request to the change control board,
composed of SAIC engineers and FBI personnel, for review. It wasn\'t
long before the change requests started rolling in---roughly 400 from
December 2002 to December 2003, according to SAIC.

\"Once they saw the product of the code we wrote, then they would say,
\'Oh, we\'ve got to change this. That isn\'t what I meant,\'\" said
SAIC\'s Reynolds. \"And that\'s when we started logging change request
after change request after change request.\" Reynolds added that SAIC\'s
bid on the original contract, and each subsequently revised cost
estimate, was based on there being \"minimal, minor changes\" to the
program once a baseline set of requirements had been agreed on.

Some of the changes were cosmetic---move a button from one part of the
screen to another, for instance. Others required the programmers to add
a new function to a part of the program, such as the graphical user
interface, common to all eight development threads.

[March 2003: The Network is Delayed]{.underline}

In March 2003, Computer Sciences Corp., which had acquired DynCorp that
month, told Higgins that the final deployment of the computers and
networks would be delayed until October. In August, October became
December. And in October, December became April 2004. The problem
wasn\'t the PCs, which had been trickling in since 2001, but changing
the e-mail system from Novell\'s GroupWise to Microsoft Outlook and,
obtaining the components needed to connect the field offices to the wide
area network. Higgins added that the delays were compounded by the
FBI\'s own sloppy inventories of existing networks and its
underestimation of how taxing the network traffic would be once all
22,000 users came online using their new PCs.

While the FBI and SAIC waited for the networks to go live so they could
test the VCF on a real system, changes and fixes continued. Many of the
changes had to be to be made by all eight of SAIC\'s development teams.
Arnold Punaro, SAIC executive vice president and general manager,
admitted in a posting on the company\'s Web site that in the rush to get
the program finished by December, SAIC didn\'t ensure that all of its
programmers were making the changes the same way. That inconsistency
occasionally meant that different modules of the VCF handled data in
different ways. Consequently, when one module needed to communicate with
another, errors sometimes occurred.

\"This, however, did not compromise the system,\" according to Punaro.
The real killers, he said, were \"significant management turbulence\" at
the FBI, \"the ever-shifting nature of the requirements,\" and the
agents\' \"trial-and-error, \'We will know it when we see it\' approach
to development.\"

\[6\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

[September 2003: Testing Begins]{.underline}

SAIC began testing the program in the fall of 2003, and problems started
cropping up, some of which the agents had warned SAIC about over the
previous summer. SAIC officials complained to Higgins that Computer
Sciences Corp. didn\'t have its hardware and network in place, so SAIC
couldn\'t adequately test the VCF, crucial for a successful flash
cutover. They informed her that they would deliver a version of the VCF
to be in technical compliance with the terms of the contract and that
the FBI should feel free to make changes to it afterward.

[December 2003: VCF Delivery and Failure]{.underline}

On 13 December 2003, SAIC delivered the VCF to the FBI. Under the
direction of the FBI's new interim CIO, Zalmai Azmi, the FBI rejected
SAIC\'s delivery. The bureau found 17 \"functional deficiencies\" it
wanted SAIC to fix before the system was deployed. SAIC argued that at
least some of these deficiencies were changes in requirements. An
arbitrator was called in. The arbitrator found fault with both SAIC and
the FBI. Of the 59 issues and subissues derived from the original 17
deficiencies, the arbitrator found that 19 were requirements
changes---the FBI\'s fault; the other 40 were SAIC\'s errors.

While SAIC fixed bugs, Azmi, with the help of Depew\'s team, created
investigation scenarios that would take different cases from opening to
closing and tested them on the VCF. Those tests revealed an additional
400 deficiencies. \"We have requirements that are not in the final
product, yet we have capabilities in the final product that we don\'t
have requirements for,\" said Azmi.

At the same time Computer Sciences Corp was delivering the final pieces
of equipment to the FBI. By April, 22,251 computer workstations, 3408
printers, 1463 scanners, 475 servers, and new local and wide area
networks would all be up and running, 22 months later than the
accelerated schedule called for.

Azmi and SAIC had yet to agree on the VCF\'s ultimate fate, much less
when it would be deployed. Azmi rejected SAIC's proposal to take one
more year to make all the changes the FBI wanted at the cost of an
additional \$56 million.

[June 2003: Trying to Salvage Something from VCF:]{.underline}

Azmi asked SAIC to take the electronic workflow portion of the VCF, code
that was in relatively good shape, and turn that into what was
eventually called the Initial Operating Capability (IOC), at an
additional fixed price to the FBI of \$16.4 million. SAIC and the FBI
project team had six months to deliver a software package that would be
deployed to between 250 and 500 field personnel in Louisiana and at the
Hoover Building.

\[7\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

**VCF Program Acquisition Business Case Study (continued)**

The objectives for the new project were clear: test-drive the VCF\'s
electronic workflow; see how people reacted to the graphical user
interface; create a way to translate the output from the VCF forms, into
the ACS system; check out network performance; and develop a training
program. The IOC would use Azmi\'s rigorous approach to software
development and project management, which he called the Life Cycle
Management Directive.

[June 2004: SAIC Begins the IOC Project]{.underline}

With new management in place, about 120 SAIC engineers began work on the
IOC project. The FBI and SAIC agreed to keep to a strict development
schedule, define acceptance criteria, and institute a series of control
gates. Azmi\'s project manager worked closely with his SAIC counterpart
to adhere to the baseline requirements SAIC and the FBI had agreed on
for the IOC in July, thus avoiding a death spiral of change requests. In
January, the IOC was rolled out as a pilot right on schedule.

[March 2005: IOC Pilot Fails]{.underline}

The IOC pilot ended in March. The verdict: \"Although the IOC
application was an aid to task management, its use did not improve the
productivity of most users,\" according to an internal FBI assessment.
FBI employees, particularly agents, found that the IOC actually
increased their workload. Agents filled out forms electronically and
routed them to superiors for approval, after which the electronic form
was uploaded to the ACS, still in use, to be shared with the rest of the
FBI. But to comply with the FBI\'s paper-based records management
system, the form had to be printed out, routed, signed, and filed.

[May 2005: Sentinel Rises from the Ashes of VCF]{.underline}

In May 2005 Mueller announced Sentinel, a four-phase, four-year project
intended to do the VCF\'s job and provide the bureau with a Web-based
case- and records-management system that incorporates commercial
off-the-shelf software.

[March 2006: Sentinel goes to Contract]{.underline}

The FBI awarded a \$305M contract to Lockheed Martin for Sentinel. Total
program cost is estimated to be \$425M.

[June 2007: Sentinel's First Deployment]{.underline}

The FBI and Lockheed Martin rolled out Phase 1 Sentinel capabilities to
the FBI workforce. According to a press release from the FBI phase 1
includes a "Personal Workbox, which summarizes a user's cases and
leads....and user-friendly search capabilities." Azmi announces his
intent to use agile development to deliver new capability to the desktop
every six to nine months.

\[8\]

  -----------------------------------------------------------------------
  **Topic 5: Formal Source Selection (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on the formal source selection. After        |
| addressing any questions the participants may have, inform them that  |
| we will be moving on to the next topic: International Sourcing vs.    |
| Domestic Preferences.                                                 |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  the formal source selection?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image29.wmf){width="3.957638888888889in"   Slide \# 5-13
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Another element to developing the source selection criteria is to     |
| consider the type of sourcing that will be used to supply the         |
| acquisition. The program manager will have to determine whether       |
| international sourcing or domestic sourcing is preferred.             |
|                                                                       |
| These sourcing decisions will be guided by specific legislation and   |
| international agreements, some of which are discussed as part of this |
| topic.                                                                |
|                                                                       |
| -   Buy American Act of 1933                                          |
|                                                                       |
| -   FAR Part 25                                                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Trade Agreements                                                  |
|                                                                       |
| -   Prohibited Sources                                                |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| Ask the following question to discuss sourcing preferences. Use       |
| participant responses to facilitate a group discussion about when     |
| international sourcing might be used in place of domestic sources.    |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**When have you used international      |
| sourcing for an acquisition, and why?**                               |
|                                                                       |
| Discuss participant examples.                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| In 1933, the **Buy American Act** was passed (BAA - 41 U.S.C. §       |
| 10a--10d), thereby mandating preferences for the purchase of          |
| domestically produced goods in direct procurements by the United      |
| States government. Other pieces of Federal legislation extend similar |
| requirements to third-party purchases that utilize Federal funds,     |
| such as highway and transit programs.                                 |
|                                                                       |
| In certain government procurements, the requirement purchase may be   |
| waived if:                                                            |
|                                                                       |
| -   Purchasing the material domestically would burden the government  |
|     with an unreasonable cost (the price differential between the     |
|     domestic product and an identical foreign-sourced product exceeds |
|     a certain percentage of the price offered by the foreign          |
|     supplier)                                                         |
|                                                                       |
| -   The product is not available domestically in sufficient quantity  |
|     or quality                                                        |
|                                                                       |
| -   Doing so is in the public interest                                |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The President has the authority to waive the Buy American Act within  |
| the terms of a reciprocal agreement or otherwise in response to the   |
| provision of reciprocal treatment to U.S. producers (see Trade        |
| Agreements later in this topic).                                      |
|                                                                       |
| **FAR Part 25** prescribes the application of goods acquisition       |
| relative to the Buy American Act, Trade Agreements, Prohibited        |
| Sources and Solicitation Provisions and Contract Clauses in its       |
| subparts.                                                             |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Specifically, FAR Part 25.001 General states:                         |
|                                                                       |
| *(a) The Buy American Act---*                                         |
|                                                                       |
| > *(1) Restricts the purchase of supplies that are not domestic end   |
| > products, for use within the United States. A foreign end product   |
| > may be purchased if the contracting officer determines that the     |
| > price of the lowest domestic offer is unreasonable or if another    |
| > exception applies (see Subpart 25.1); and*                          |
| >                                                                     |
| > *(2) Requires, with some exceptions, the use of only domestic       |
| > construction materials in contracts for construction in the United  |
| > States (see Subpart 25.2).*                                         |
|                                                                       |
| *(b) The restrictions in the Buy American Act are not applicable in   |
| acquisitions subject to certain trade agreements (see Subpart 25.4).  |
| In these acquisitions, end products and construction materials from   |
| certain countries receive nondiscriminatory treatment in evaluation   |
| with domestic offers. Generally, the dollar value of the acquisition  |
| determines which of the trade agreements applies. Exceptions to the   |
| applicability of the trade agreements are described in Subpart 25.4.* |
|                                                                       |
| *(c) The test to determine the country of origin for an end product   |
| under the Buy American Act (see the various country "end product"     |
| definitions in 25.003) is different from the test to determine the    |
| country of origin for an end product under the trade agreements, or   |
| the criteria for the report on end products manufactured outside the  |
| United States (see 25.004).*                                          |
|                                                                       |
| > *(1) The Buy American Act uses a two-part test to define a          |
| > "domestic end product" (manufacture in the United States and a      |
| > formula based on cost of domestic components).*                     |
| >                                                                     |
| > *(2) Under the trade agreements, the test to determine country of   |
| > origin is "substantial transformation" (i.e., transforming an       |
| > article into a new and different article of commerce, with a name,  |
| > character, or use distinct from the original article).*             |
| >                                                                     |
| > *(3) For the reporting requirement at 25.004, the only criterion is |
| > whether the place of manufacture of an end product is in the United |
| > States or outside the United States, without regard to the origin   |
| > of the components.*                                                 |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| In certain procurements, **Trade Agreements** enable the use of       |
| international sources when reciprocal treatment exists for U.S.       |
| producers. For example, under the 1979 GATT Government Procurement    |
| Code, the U.S.-Israel Free Trade Agreement, the U.S.-Canada Free      |
| Trade Agreement, and the WTO 1996 Agreement on Government Procurement |
| (GPA), the United States provides access to the government            |
| procurement of certain U.S. agencies for goods from the other parties |
| to those agreements.                                                  |
|                                                                       |
| FAR Part 25, Subpart 25.4 Trade Agreements 25.402 General states:     |
|                                                                       |
| *(a) Trade Agreements Act (19 U.S.C. 2501, et seq.)*                  |
|                                                                       |
| > *(1) The Trade Agreements Act (19 U.S.C. 2501, et seq.) provides    |
| > the authority for the President to waive the Buy American Act and   |
| > other discriminatory provisions for eligible products from          |
| > countries that have signed an international trade agreement with    |
| > the United States, or that meet certain other criteria, such as     |
| > being a least developed country. The President has delegated this   |
| > waiver authority to the U.S. Trade Representative. In acquisitions  |
| > covered by the WTO GPA, Free Trade Agreements, or the Israeli Trade |
| > Act, the USTR has waived the Buy American Act and other             |
| > discriminatory provisions for eligible products. Offers of eligible |
| > products receive equal consideration with domestic offers.*         |
| >                                                                     |
| > *(2) The contracting officer shall determine the origin of services |
| > by the country in which the firm providing the services is          |
| > established. See Subpart 25.5 for evaluation procedures for supply  |
| > contracts covered by trade agreements.*                             |
|                                                                       |
| *(b) The value of the acquisition is a determining factor in the      |
| applicability of trade agreements. Most of these dollar thresholds    |
| are subject to revision by the U.S. Trade Representative              |
| approximately every 2 years.*                                         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| It is equally important to consider **prohibited sources**. FAR Part  |
| 25 provides specific guidance relative to prohibited sources.         |
|                                                                       |
| FAR Part 25, Subpart 25.7 Prohibited Sources 25.701 Restrictions      |
| states:                                                               |
|                                                                       |
| *(a) Except as authorized by the Office of Foreign Assets Control     |
| (OFAC) in the Department of the Treasury, agencies and their          |
| contractors and subcontractors must not acquire any supplies or       |
| services if any proclamation, Executive order, or statute             |
| administered by OFAC, or if OFAC's implementing regulations at 31 CFR |
| Chapter V, would prohibit such a transaction by a person subject to   |
| the jurisdiction of the United States.*                               |
|                                                                       |
| *(b) Except as authorized by OFAC, most transactions involving Cuba,  |
| Iran, and Sudan are prohibited, as are most imports from North Korea  |
| into the United States or its outlying areas. In addition, lists of   |
| entities and individuals subject to economic sanctions are included   |
| in OFAC's List of Specially Designated Nationals and Blocked Persons  |
| at http://www.treas.gov/offices/enforcement/ofac/sdn. More            |
| information about these restrictions, as well as updates, is          |
| available in OFAC's regulations at 31 CFR Chapter V and/or on OFAC's  |
| website at http://www.treas.gov/offices/enforcement/ofac.*            |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| Ask the following question to relate sourcing preferences             |
| considerations and corresponding decisions to the VCF case study used |
| in this module. Use participant responses to facilitate a group       |
| discussion about international sourcing vs. domestic sources.         |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**How does the specific legislation that
  applies to sourcing preferences relate to the VCF Acquisition Business
  Case Study, and how might it impact the program manager's decision
  making authority for this program?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 6: International Sourcing vs. Domestic Preferences
  (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on international sourcing vs. domestic       |
| preferences. After addressing any questions the participants may      |
| have, inform them that we will be moving on to the next topic: Price  |
| Reasonableness.                                                       |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  international sourcing vs. domestic preferences?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness**

  -----------------------------------------------------------------------

  ---------------------------------------------------- ----------------------
  ![](media/image30.wmf){width="3.957638888888889in"   Slide \# 5-14
  height="2.975in"}                                    

  ---------------------------------------------------- ----------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The concept of "price reasonableness" is to assure that no more than  |
| a commercially reasonable price (as determined, for example, by a     |
| comparison of price quotations and market prices) is paid to procure  |
| goods or services. To ensure compliance with this principle, the      |
| program manager and /or team should conduct a price-reasonableness    |
| analysis of the prices offered in a procurement process (both for     |
| competitive and single-source procurements).                          |
|                                                                       |
| The main objective of the price-reasonableness analysis is to ensure  |
| that the quoted prices of the selected contractor or supplier are     |
| neither unreasonably high (which could happen, for example, when      |
| there is a poor response to the Request for Proposals) nor            |
| unreasonably low (which could happen, for example, when there is      |
| severe price competition, or poorly understood description of the     |
| objective of the procurement).                                        |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| The concept of price reasonableness as it relates to the contracting  |
| officer is addressed in FAR Subpart 12.2 Special Requirements for the |
| Acquisition of Commercial Items, Section 12.209 Determination of      |
| Price Reasonableness:                                                 |
|                                                                       |
| *While the contracting officer must establish price reasonableness in |
| accordance with 13.106-3, 14.408-2, or Subpart 15.4, as applicable,   |
| the contracting officer should be aware of customary commercial terms |
| and conditions when pricing commercial items. Commercial item prices  |
| are affected by factors that include, but are not limited to, speed   |
| of delivery, length and extent of warranty, limitations of seller's   |
| liability, quantities ordered, length of the performance period, and  |
| specific performance requirements. The contracting officer must       |
| ensure that contract terms, conditions, and prices are commensurate   |
| with the Government's need.*                                          |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| FAR Subpart 15.3 Source Selection, Section 15.305 Proposal Evaluation |
| (a)(1) also refers to the cost or price evaluation and states:        |
|                                                                       |
| *(1) Cost or price evaluation. Normally, competition establishes      |
| price reasonableness. Therefore, when contracting on a                |
| firm-fixed-price or fixed-price with economic price adjustment basis, |
| comparison of the proposed prices will usually satisfy the            |
| requirement to perform a price analysis, and a cost analysis need not |
| be performed. In limited situations, a cost analysis (see             |
| 15.403-1(c)(1)(i)(B)) may be appropriate to establish reasonableness  |
| of the otherwise successful offeror's price. When contracting on a    |
| cost-reimbursement basis, evaluations shall include a cost realism    |
| analysis to determine what the Government should realistically expect |
| to pay for the proposed effort, the offeror's understanding of the    |
| work, and the offeror's ability to perform the contract. (See 37.115  |
| for uncompensated overtime evaluation.) The contracting officer shall |
| document the cost or price evaluation.*                               |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| Ask the following question to establish a common understanding of     |
| price reasonableness analysis.                                        |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**How would you define a price          |
| reasonableness analysis?**                                            |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| The price-reasonableness analysis is a review of the prices proposed  |
| by a contractor or supplier to ensure that the prices offered in the  |
| bids or proposals are fair to both parties (neither too high nor two  |
| low) considering the effort required to complete the task, the        |
| quality of the bid or proposal, and the comparability of the prices   |
| on similar projects in local and international markets.               |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| There are several ways a price-reasonableness analysis may be         |
| conducted without examining the individual components of the total    |
| price.                                                                |
|                                                                       |
| -   Competitive Prices: In an open competitive bidding process where  |
|     price is an evaluation factor and several acceptable and          |
|     qualified bids or proposals are received, it can be safely        |
|     assumed that the prices are reasonable as determined by the       |
|     market conditions, but only if there are minimal discrepancies    |
|     among the offered prices. If there are significant discrepancies  |
|     between any two bids or proposals, the program manager should     |
|     ensure that the description of the procurement is clear enough    |
|     not to be misunderstood, and that there are no calculation        |
|     errors.                                                           |
|                                                                       |
| -   Historical Prices: If the prices offered are comparable to the    |
|     prices offered in the past for similar contracts either by the    |
|     same or other contractors or suppliers used in prior              |
|     acquisitions, to other large government buyers, or to a donor     |
|     agency in the country of the project, then the prices can be      |
|     assumed to be reasonable. When comparing current prices with      |
|     historical prices, inflation and a reasonable margin for          |
|     increased profits should be taken into account.                   |
|                                                                       |
| -   Catalogue Prices: This method is suitable when purchasing         |
|     off-the-shelf commercial goods, where the supplier is likely to   |
|     have a catalogue, and the supplier is willing to provide a        |
|     current, dated page from the catalogue that lists the price of    |
|     the item being procured and any discounts that may be available   |
|     to large buyers. In this case, the supplier may be requested to   |
|     provide a reference of another large buyer, whether inside the    |
|     country of the project or outside, who can verify the prices that |
|     were paid by that buyer to this supplier.                         |
|                                                                       |
| > (Continued on the following page)                                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Other ways a price-reasonableness analysis may be conducted without   |
| examining the individual components of the total price include:       |
|                                                                       |
| -   Published Prices: Some commercially traded commodities such as    |
|     grains, petrol, or copper have international or national prices   |
|     that are published in popular press. If the prices offered are    |
|     the same or lower than these published prices, then it can be     |
|     assumed that the prices are reasonable.                           |
|                                                                       |
| -   Independent estimate or planned budget. If an independent         |
|     estimate has been prepared, such as a realistic and reasonable    |
|     budget during the acquisition strategy development process, this  |
|     estimate can be used as a reference for reasonable prices.        |
|                                                                       |
| -   Comparison with prices of a similar item: In the procurement of   |
|     goods, if a supplier can demonstrate the prices of a similar      |
|     item, and justify its price difference based on additional or     |
|     different features, the program manager can accept the proposed   |
|     prices based on these two factors. The differences between the    |
|     supplier's price and the price of a competitor should be detailed |
|     and justified, and the referenced prices should be available from |
|     verifiable independent sources (not only from the supplier        |
|     itself).                                                          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| Ask the following questions to facilitate a large group discussion of |
| price reasonableness analysis.                                        |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are some other types of price    |
| reasonableness analysis you have used?**                              |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use participant examples to further define the concept of price       |
| reasonableness analysis and the different types of analyses that may  |
| be conducted.                                                         |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="0.5006944444444444in"                    |
| height="0.4888888888888889in"}**What are some examples of how you     |
| have used a price reasonableness analysis?**                          |
|                                                                       |
| **What affect did the analysis have on your determination?**          |
|                                                                       |
| **Answer:**                                                           |
|                                                                       |
| Use participant examples to further illustrate the concept of price   |
| reasonableness and how it relates to the source selection process of  |
| program management.                                                   |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 7: Price Reasonableness (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes the topic on price reasonableness. After addressing    |
| any questions the participants may have, inform them that we will be  |
| moving on to the next topic: Review.                                  |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have regarding
  price reasonableness?**

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **Topic 8: Review**

  -----------------------------------------------------------------------

  ------------------------------------------------------ ----------------
  ![](media/image31.wmf){width="3.957638888888889in"     Slide \# 5-14
  height="2.975in"}                                      

  ------------------------------------------------------ ----------------

+-----------------------------------------------------------------------+
| ![](media/image5.png){width="0.75in" height="0.6868055555555556in"}   |
| **[CONTENT TO BE PRESENTED:]{.underline}**                            |
|                                                                       |
| **Key Points:**                                                       |
|                                                                       |
| Let's review the learning objectives for this module on Perform       |
| Source Selection.                                                     |
|                                                                       |
| -   Given an acquisition case study, apply appropriate risk analysis  |
|     methods as part of source selection criteria.                     |
|                                                                       |
| -   Relate the specific legislation that applies to sourcing          |
|     preferences to an acquisition and explain how it impacts the      |
|     program manager's decision making authority.                      |
|                                                                       |
| -   Apply concepts of sourcing preferences and price reasonableness   |
|     to allow for best value selection in program management.          |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  **Topic 8: Review (continued)**

  -----------------------------------------------------------------------

+-----------------------------------------------------------------------+
| ![](media/image3.png){width="0.4048611111111111in"                    |
| height="0.6354166666666666in"} **Instructor Notes:**                  |
|                                                                       |
| This concludes Module 5 -- Perform Source Selection.. Address any     |
| questions the participants may have before moving on to Module 6 --   |
| Contract Administration.                                              |
+-----------------------------------------------------------------------+

  -----------------------------------------------------------------------
  ![](media/image7.png){width="0.5006944444444444in"
  height="0.4888888888888889in"}**What questions do you have about Module
  5 -- Perform Source Selection?**

  -----------------------------------------------------------------------