# Presentation: 099009

## What Executives Need to KnowAbout Software Acquisitions

**ITS Professional Capacity Building Program**
**T3 Session**
- July 12, 2006
- 1:00 p.m. EDT

## TODAY’S PRESENTERS

**Your Instructor:**
**Art Salwin**
- Mitretek Systems
**Your Host:**
**Mac Lister**
- FHWA Resource Center

## Software Horrors

## Slide 4

**Notes:**
We begin with some real-world horror tales of actual software project experiences.
Self-explanatory
Example of on-ITS transportation projects in the news:
TMC #1
TMC #2
Accident Reporting System
Metro

## Slide 5

**Notes:**
More examples:
California Dept of Motor Vehicles (drivers license registration)
Denver Airport baggage handling—costs are not just those incurred by the software project
London ambulance system - deaths

## Software = Trouble

***Software is Different!***

**Notes:**
Main idea:  Software problems are widespread and large in magnitude.
What’s a significant overrun on construction projects--what’s tolerable?  What’s bad?  What’s really bad?
[Note:  Managers typically respond with numbers in the 5 to 15% range.]
[Summarize their answers.]  Well here’s the software experience.  The pie chart summarizes the experience across the board (not just ITS; not even just Government) from data collected by the Standish Group:
red zone:  1 project in 3 is canceled.  (80,000 per year, at a cost of $81Billion) to prevent further bleeding, delivering nothing
yellow zone:  a little over 1/2 deliver something but with average 89% over budget, 222% over schedule (i.e., 3X), and deliver only 61% of the originally proposed features and functions  (pay more, take longer, get less)
green zone:  only 1 project in 6 is generally okay
A more recent Standish group study shows a slight improvement in these numbers--the red and green zones are now equal in size.  [green has grown from about 1/6 to 1/4; red has shrunk from 1/3 to 1/4; exact figures are 28% green, 28% red, and 44% yellow]  This is largely attributed to a recommended practice that we’ll present in module 3 of this seminar.  [It’s bite-size pieces, but don’t tell the students that now.]
Our goal is to get you in the green zone, or at least a greenish shade of yellow.
Aside:  one study showed that even with calibrated models, whose use is beyond the state-of-the-practice in ITS, get 25% 50% of the time.

## Software is Different!

- Learn as you go

- Fundamental flaw of software engineering: assuming that *“one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it ... this assumption is fundamentally wrong.”*
- — Brooks, 1987

**Software is Different!**

**Notes:**
Differences are not just the overruns, but are fundamental to the way that software is developed.
Fred Brooks tells us that you can’t know everything up front.  Very different from traditional projects.
And if you don’t, know what a realistic cost and schedule is, how can you  expect to meet it?
Frequent iterations are essential.  “Design a little, build a little, test a little”; use what you get and then repeat.  Take 2 steps forward, one step back. 
The implication of all this is that you learn as you go.  Therefore you must structure the project to accommodate these realities instead of using a process that assumes you know what you want, which is implicit in the engineer/contractor/low bid approach.  

## Software is Different!

- The bad news is...

**Software**
**is**
**Different**

**Classic **
**Approaches**
**Don’t **
**Work**

**Need **
**Different**
**Approaches**

**Notes:**
So if software is different and the classic approaches don’t work then

We have to try something different. 

That’s the bad news...

## Software is Different!

- The good news is...

**Contracting**

**Maintenance**

**Staffing**

**Notes:**
The good news is that there are best practices/remedies (hold up Guidebook to Software Acquisition Best Practices) that really work. (Aside: 5% of the companies surveyed consistently deliver their projects in the green zone, within 5% of budget and 3% of schedule with no reported cancellations.)  In fact, in some cases multiple options are available.  

[no need to list all the bullets below, just highlight one or two of them as examples]
There are options in the types of contracts that you can use to procure the software. 
There are options regarding who will maintain the software.
There are a number staffing options for managing the project.

Many of the options deal with technical approaches.  In this seminar, we’ll explore some of the managerial options and decisions that are relevant to the executive.  

## What You Can Do To Help

**What You Can Do To Help**

**Notes:**
At this point, we’ve gone through all the theory by covering the problems you can encounter (and want to avoid) and how software must be developed iteratively.  We now turn to the real meat of the seminar and provide practical advice on what an executive can do to help achieve success on software initiatives. 
In this module we will go through a number of specific things that you, as an executive, can do to help make your software projects a success.  Do’s and Don’ts will be given where applicable.

Strategic things.

## How You Can Help

**1.**  **Provide adequate resources**

**Notes:**
The first way you can help is by providing adequate resources, thereby showing your commitment and support for the project.  When we say “resources” we’re not just referring to money...

## 1. Provide Adequate ResourcesProvide Adequate Staffing

- 1. Provide Adequate Resources**Provide Adequate Staffing**

- Assign a **full-time** project manager

**Notes:**
Main idea:  In fact, a most critical resource is to provide the necessary staffing.  
Now there will be a whole team of people who will come and go with various stages of the project.  In fact, in the full-length course we talk about teaming arrangements across a wide range of people in the agency and with the contractor.  
But within that cadre of people, there will be a smaller, core group, of full-time individuals who are dedicated to the project and live & die with it.
At a minimum, as part of that core, you need to assign a full-time project manager.
This manager will be very busy. In fact, on CHART the number of action items for Government side often exceeded number assigned to the contractor.  Since we stress an iterative approach, the manager needs to be fully engaged at all points.  The project manager role never goes away—not after requirements are developed, contract is issued, or development is completed.  There never comes a time where the manager can “throw the system over the fence” and let the contractor run with it.   Some sample roles:  

##     1. Provide Adequate ResourcesStaffing Options

**Hire in-house staff**

**Notes:**
Main idea:  How do you staff the team?
The ideal solution would be to assign in-house staff for the project manager and other core team members.  That way you can have your own people manage the project.  However, we recognize that for state agencies, hiring staff can be difficult--limited number of staff slots, hiring freezes, non-competitive salary scales, etc. 
So what do you do if you can’t hire? Here’s one of the places where we thankfully have a lot of options, and we’ve listed some of them here that have been successfully used on ITS projects.
Contract employees:  work on-site like in-house staff.  WSDOT has had success with this approach with independent consultants.  
University support can write specs, review products, etc.  They may have outside expertise and don’t have a commercial product bias.  VDOT has used this approach.  The project manager for CHART is actually hired through a contract with the University of Maryland.
V&V contractors:  On CHART they have 2 V&V contractors--one for process, the other for technical.  Not-for-profits can be used.
Hire a project manager through a contract:  CHART does this through the University of Maryland.

## 1. Provide Adequate ResourcesOther Essential Resources

- Non-core team members

- Staff training
- Contractor resources for management support
- Facilities
- Software maintenance

**Notes:**
Main idea:  Let’s talk about some other essential resources.  
Going beyond the project manager and the core staff, non-core team members are others who are involved on the project.  Unlike the core staff who are dedicated to the project, they will not be involved full time.  But they still have vital roles: 
Some sample non-core team members with their roles and activities are listed.  Most are self evident.  (“End users” include administrators, shift supervisors, etc.)
Not only do team members need a charge number, but their time must be freed up so they can attend meetings, participate in reviews, requirements interviews, testing, and prototyping.  This is where you can help:  Ensure these resources are freed up so the project can get access to them.  Shows your commitment to the project.
Staff training:  an educated customer is essential.  (Software is different, so civil engineering experience may not be relevant.)  CHART sent their officials to training on the contractor’s methodology.  A capable contractor is essential (as we shall see), but overall project capability is crucially dependent on customer as well as contractor capabilities.  Studies show customer experience having roughly equal impact on success rates as contractor expertise, with chance of success going up dramatically with more experience.
Management support:  Kansas found they needed to allocate roughly 20% to contractor for document prep, review, attend meetings, etc.  If contractor skimps on this you’re not getting a bargain.  (Danger of rewarding the wrong actions.)
Facilities:  Where is the software going to be developed?  Where will testing take place--can’t do it on the public roads.
Software maintenance...

## How You Can Help

**1.**  **Provide adequate resources**

**2.  Foster teaming**

**Notes:**
Providing adequate resources--money, staffing, and other--is the first way you can help.

The second way you can help is to foster teaming.

## How You Can Help 2. Foster Teaming

- Within your agency
- With other agencies
- With neighboring jurisdictions

**Notes:**
Main idea:  This slide presents some examples of the types of people you can help to get on your team.  Encourage your people to form teaming relationships.

Within your agency:  you can help free up resources and make sure the people, even those not directly involved on the project, are made available.  This is “provide adequate resources” revisited.

With other agencies:  your backing will carry more weight in bringing these people to the table.

With neighboring jurisdictions:  On ITS, we’re pushing regional integration so the neighbors need to be brought to the table.  Again your backing can help make this happen.

## DO

- Coordinate with other agenciesand jurisdictions

**DO**

- 2. Foster Teaming

**Notes:**
Main idea:  What can you do to encourage the teaming relationships?

Make people available by freeing them up and giving them a budget number.  Your project manager needs access to them.  Let other managers know that the software is a priority.  It also means not just freeing up the “losers”.

How do you encourage outside agencies to come to the table, make them team if they don’t want to:  By virtue of your position, your requests for assistance may carry a lot more weight than those of your project manager.

On so many of the project failures, an antagonistic relationship develops between the agency and the software contractor.  But in fact perhaps the most important member of the team is the software contractor, so...

## 2. Foster Teaming Non-Core Team Members

**Notes:**
Main idea:  Let’s talk about some other essential resources.  
Going beyond the project manager and the core staff, non-core team members are others who are involved on the project.  Unlike the core staff who are dedicated to the project, they will not be involved full time.  But they still have vital roles: 
Some sample non-core team members with their roles and activities are listed.  Most are self evident.  (“End users” include administrators, shift supervisors, etc.)
Not only do team members need a charge number, but their time must be freed up so they can attend meetings, participate in reviews, requirements interviews, testing, and prototyping.  This is where you can help:  Ensure these resources are freed up so the project can get access to them.  Shows your commitment to the project.
Staff training:  an educated customer is essential.  (Software is different, so civil engineering experience may not be relevant.)  CHART sent their officials to training on the contractor’s methodology.  A capable contractor is essential (as we shall see), but overall project capability is crucially dependent on customer as well as contractor capabilities.  Studies show customer experience having roughly equal impact on success rates as contractor expertise, with chance of success going up dramatically with more experience.
Management support:  Kansas found they needed to allocate roughly 20% to contractor for document prep, review, attend meetings, etc.  If contractor skimps on this you’re not getting a bargain.  (Danger of rewarding the wrong actions.)
Facilities:  Where is the software going to be developed?  Where will testing take place--can’t do it on the public roads.
Software maintenance...

## DO

*... **even if it’s a subcontractor*

**DO**

- 2. Foster Teaming

**Notes:**
Main idea:  We want to emphasize the importance of fostering teaming with the software contractor, which is often difficult to do.
Need to establish working relationships based on open communications and teaming that lead to working towards common goals. 
Teaming with the contractor and avoiding adversarial relationships is difficult even in the best of circumstances. (The two sides are wary of one another.)  But a particular challenge is what to do when the software is developed by a subcontractor.  Cannot let the prime get in the way.  May even need explicit contracting language to allow direct access.  (Later on we’ll talk about ways to avoid this dilemma altogether by not having software subcontracts.)
Has this been a problem on other projects in your agency?  If so, what did you do about it? 
ITS war story:  Software subcontractor frustrated by inability to gain access to customer for necessary project input.  Prime was a major obstacle, requiring formal meetings, etc.  Several years later, they learned that customer had also been diligently trying to get access to the contractor.  In our section on contracting, we’ll discuss some ways to avoid this situation.
Because the relationship with the software contractor is so important, a little later on we will discuss picking the right contractor to team with.
Aside:  Long-term working relationships are good.  In some cases, even 10 years.  Concerns about getting locked in are legitimate, but they shouldn’t be the driving force--too many sins committed in the name of not getting locked in.
ADDRESS VISIBILITY PROBLEM

## How You Can Help

**1.  Provide adequate resources**
**2.  Foster teaming**

**3.  See if you can buy instead of build**

**Notes:**
So the second way you can help is by fostering teaming, especially with the contractor.  A little later on, we’ll talk about how to ensure that you pick the right contractor to team with.

## ... If You Can Buy

- 3. Buy Instead of Build**Don’t Build ...**

**Notes:**
One of the many options that are available to you is the decision whether to build the software from scratch (i.e., to develop it) or buy an existing off-the-shelf product.
Our answer is 
Don’t Build
If you can buy it.
“It is no longer acceptable for the government to specify, build, and maintain a comparable product when a commercial solution is available.”
— Carney and Oberndorf, 1997
Clearly buying software is not always an option, but when it is, buying offers a number of advantages.

## 3. Buy Instead of BuildPotential Advantages of Buying The Software

- Lower risk
- Shortened schedule
- Reduced costs
- Enhanced quality
- Stay current with technology

**Notes:**
Main idea:  There are a number of potential advantages associated with buying the software instead of building it from scratch.
Reduce risk:  It’s already there.
Speed up acquisition time:  No need to develop the software.
Lower development cost:  amortize across multiple customers
Enhance quality:  If you buy, bugs have been ironed out elsewhere.  Feature enhancements requested by customers have already been incorporated.  Also the overall quality of ITS software state-of-the-art is enhanced since vendors aren't continually re-inventing the wheel.
Reduce maintenance costs:  costs are amortized over all the customers.
Caution:  Avoid buying version 1.00 of a software product.  (But isn't that what you're really doing when you build from scratch?)

## 3. Buy Instead of BuildBuying Software Requires Flexibility

- You may not get everything you want

***80% may be good enough!***

- You may even have to change some operational procedures

**Notes:**
An off-the-shelf product probably won’t meet every last requirement.  
But 80% is often good enough, especially when you consider that in the yellow zone, 50% delivery is more typical.  Desirements vs. requirements.  Bottom line:  it requires tradeoffs. 
The software will change you anyway.  Here’s a place where an executive can help.  Example:  
Company #1 had so many absence codes that they had to use a home-grown payroll system.  Very cumbersome to maintain, filled with bugs, not current with the state-of-the-art.
Company #2 changed the accumulated leave policy from 2X annual accrual to everyone getting 8 weeks to accommodate this feature of a commercial package.
Aside:  Potential disadvantages of buying software include:
unneeded functions
stovepiping
glueware
Aside:  A complaint we’ve heard from ITS managers is that their managers become very rigid and dogmatic about not allowing the system to change anything.

## DO

- Provide the needed flexibility

**DO**

- 3. Buy Instead of Build

**Notes:**
Before a development contract is issued, make sure that the buy option was considered and rejected for valid reasons.
The buy decision “should be the product of analysis, reasoning, and engineering decisions, not the desire to jump on the latest bandwagon.”  — Carney and Oberndorf, 1997)

Provide the necessary flexibility in mindset and requirements to not unnecessarily preclude the buy option.

## How You Can Help

**1.  Provide adequate resources**
**2.  Foster teaming**
**3.  See if you can buy instead of build**

**4.  Break the project into pieces**

## Project

- 4.  Break the Project into Pieces

**Notes:**
Main idea:  We start with the overall project.  There are 2 aspects of splitting it into pieces.  
The first aspect is separating software from construction, using separate contracts.  Another quote from the Tony Kane memo:  “Often, ITS work is included within a larger construction project.  Experience has shown that this may be inadvisable, and that the ITS work may be better accomplished as a separate procurement.”  The biggest piece in terms of dollars will be the construction contract.  By making this a separate contract, it avoids any problems with ensuring that at least 50% of the work is done by the prime.  (Typically in terms of dollars--but not in terms of headaches--the software is only a small part of an overall transportation project.)
By making the software contract separate, it allows you to pick the right software contractor--one who is software knowledgeable.  Explore the options we discussed previously such as non-professional services.  
If you can’t split the contracts, then make sure that you can have direct, open communications with the software subcontractor.
Example of splitting:  
Putting up DMSs.  The supports are part of the construction contract.  The signs themselves and interfacing them to traffic management software is a separate item. 
The second aspect of splitting the project into pieces, and the one we want to emphasize here, it splitting the software development up into a series of bite-size pieces.  We talked about how large projects fail much more often than smaller ones.  In effect, we’re reducing complexity by breaking the project into a series of smaller projects.  Deploy a core capability and add functionality incrementally.  Also gives you early winners, an opportunity to kick the tires, and builds political support.  Allows you to learn as you go.  Each phase ends with a plan for the next one.  Lowers customer risk (only at risk for current phase) and provides contractor incentive to perform (to get next phase).
This is premised on having the right contract such as a task-ordered arrangement.

## 4.  Break the Project into Pieces Examples of Bite-Size Pieces

- Functions
- Geographic scope
- Interfaces
- Number of devices
- Operational positions

## 4.  Break the Project into Pieces Take Bite-Size Pieces

- 4.  Break the Project into Pieces** ****Take Bite-Size Pieces**

- Gain benefits and operational experience sooner

- Fewer headaches

- Tangible progress builds political support

**Notes:**
Main idea:  Don’t do everything in one big step.  No big bang.  “Take baby steps”   “Big steps are killers”.   “Have an early success.”  “Get early winners.” 
Especially the first chunk should be as small as possible.  Start by building only what you absolutely have to have.  To frills.  Start with core functions that you can build upon later.
What are some examples of “bite-size” pieces?  Possible answers:
don’t do all the functions, just some of them
don’t implement all the interfaces
restrict the number of vehicles/sensors/field devices/intersections
manually carry out an operation that will ultimately be automated
numbers and types of operational positions and capabilities available to them
degree of integration
SIMPLIFY! SIMPLIFY! SIMPLIFY! the requirements and then develop
Mouse click #1:  Ask the class “Why gain operational experience sooner?”.  Some reasons are:
1) Technical reasons:  anticipated wants aren't what you will necessarily need. "The customer doesn't generally know what is needed and neither does anyone else."  [Watts Humphrey]   Get the system to the field and see how it's used.  “Kick the tires.”  Many anticipated features go unused.  [note:  this is very different from combative relationship where contractor builds what was asked for and then blames the customer for making changes].  Also, if software acquisition is new to you, cut your teeth on something manageable (big acquisitions = big problems; little acquisitions = little problems).  Opportunity for contractor and customer to learn to work together.
2)  Political reasons:  Chance to show progress and get management support.  Build upon your successes.  Get them to ask for more.  Improves staff morale--they see something tangible resulting from their efforts.
3) Realize the benefits earlier on.
4)  Learn from your mistakes.
5) Better accommodates new technology as it becomes available--not forever chasing advances.
Mouse click #2:  Fewer headaches:
The y-axis represents any of the “bad things” that can happen.  (It’s really multiple axes in one.)  Chance of cancellation:  <1% very small projects cancelled; 65% of very large projects cancelled.  [Jones, Patterns of System Failures, 1996]  (Productivity goes the other way from the curve.)
Read the x-axis backward:  as you decrease size you get a disproportionate benefit.  “Because the effort required to build software increases disproportionately faster than the size of the software, a reduction in size will improve development speed disproportionately.”  Cutting size by 1/2 typically cuts the overall effort by 2/3.   [McConnell, 1996]  Note the personnel turnover:  you lose buy-in when people leave.  The new staff want things differently than the original ones.  Old agreements no longer hold.

## DO

- Separate software from construction

**DO**

- 4.  Break the Project into Pieces** **

**Notes:**
Successful projects are never scheduled longer than 9 months.  Six or less  is more typical.  Internet time is under 90 days.
More Standish statistics:
Projects over $10M, none in the green zone.
Projects under $750K, 50% in the green zone.
More recent Standish study shows larger green and smaller red than before
(approximately 1/4 green,  1/4 red, 1/2 yellow) 
Main reason:  bite-size pieces.
The various “do’s” are inter-related.  Breaking the software into pieces means having a task-ordered contract or something similar to accommodate this.
Final bullet item: reinforcing that software as a whole is broken off from construction.

## How You Can Help

**1.  Provide adequate resources**
**2.  Foster teaming**
**See if you can buy instead of build**
**4.  Break the project into pieces**

**5.  Pick the right contract**

**Notes:**
Pick the contracting vehicle that’s most appropriate for each piece.  We’ll focus on the software contract.
The choice of contracting mechanism will set the stage for the success or failure of the project.
The contracting mechanism must allow the project to take advantage of the open communications that are used in teaming.  It defines all the activities, roles, responsibilities, and privileges on the project.  It takes cognizance of the fact that Software is Different and we need a different type of contracting mechanism than the familiar ones, which just don’t work for software. 
Again, software is different, so different contracting mechanisms are needed rather than the old familiar ones.

## 

**Memorandum**

- U.S. Departmentof Transportation
**Federal Highway****Administration**

- We continue to hear many comments from both the public and private sectors on the difficulty in procuring ITS projects under Federal-aid procedures.  These comments, and the recent article “Death by Low Bid” (attachment 1), are the direct result of the perception that ITS projects must always use the same procurement approach
- Often, ITS work is included within a larger construction project.  Experience has shown that this may be inadvisable, and that the ITS work may be better accomplished
- Many stand-alone ITS projects do not have to be procured under the rules for construction projects, since they do not meet the FHWA definition of construction.  If field devices or communications infrastructure is being development of software to integrate does **not** meet the definition of construction.  Software should never be procured using the “normal” low bid process, rather it should be treated as engineering or design services to be provided only by qualified and experienced
- Another option available is the use of a State or local agency’s own procurement process,  which is permitted uder 49 CFR 18.  While some states have procurement

- Reference: *Action: Procuring ITS Projects, *FHWA Memo from Anthony R. Kane, Executive Director, dated October 6, 1999.[**emphasis** and color added]

**Software**
****

- 5.  Pick the Right Contract

**Notes:**
In fact, there’s FHWA guidance in this area.  In 1999 Tony Kane, who at the time was the Executive Director of FHWA, issued a memo stating in part that “development of software...does not meet the definition of construction.”  In particular, he advises that “Software should never [emphasis added] be procured using the “normal” low bid process...”.
Hold up copy of memo:  We have a copy of the memo as part of your handouts.  When the memo was issued, it was accompanied by an article entitled “Death by low bids?” which describes what happens when this advice is not heeded.
Aside:  The reason low bid process is so dangerous is that it is anathema to the teaming and open communications we want to foster.  It doesn’t allow iterative development; it’s more in the flavor of “Here’s what we want, now go do it.”  It assumes you know what you want up front (not the case in software).
Aside:  Low bid is more than just taking the lowest price.  “Bid” means that we’re not using a negotiated process.  It would be appropriate, for example, if you wanted to buy a hundred copies of a word processor where there’s nothing to negotiate, just get the lowest price. 
Aside:  Other key words from the memo include “this perception [must use same procurement approach required for construction projection] has resulted in numerous procurement disasters [emphasis added] where the “normal” low-bid procurement process has been inappropriately used for ITS projects.”

## 5.  Pick the Right Contract

**Design/Build**
**D/B/O/M**

**Task Order**

**T&M**

**Indefinite Quantities**

**Design Competition**

**Non-Professional Services**

**Cost-Plus**

**Notes:**
Main idea:  Fortunately there’s a wealth of contracting mechanism options available to you.
T&M or cost-plus makes the contractor less resistant to working with you since they are not being squeezed on a fixed price contract.
It’s important to phase the activities (we’ll get into this shortly) so several of the options listed (indefinite quantity, multi-phase, task order) address this.
Other possibilities:
Non professional services used by VDOT.  This is an example of the “common rule” whereby states can use their own mechanisms if these are more favorable for high tech projects.
Design competition phase means paying two or more contractors to work in parallel to reduce risk and then pick the appropriate one at some point.  This was done on the National ITS Architecture.

## Use low bid for software

**Put final requirements in contract**
**Use engineer/contractor****(two-step) process**

**DON’T**

- 5.  Pick the Right Contract

**DON’T**

**DON’T**

**Notes:**
So to recap,

Don’t use low bid.  The whole basis of low bid is that you know the cost, schedule, and specs up front.  None of that applies to software.

An implication of  not going low bid is that you don’t include the final requirements in the contract.  Contractor needs to be involved in the process of developing requirements.  If you put them in the contract and you have a flexible contract, then every time the requirements change you’ll have to change the contract.  Furthermore, changing requirements in this fashion after contract award could lead to protests by the losing offerors.

Similarly don’t use the familiar two-step process in which an engineering firm writes the requirements and then they’re “thrown over the fence” at the software contractor.

## DO

- Use an incremental approach
- Call out software activities and products
- Gain full access to subcontractor(s)
- Separate software from construction

**DO**

- 5.  Pick the Right Contract

**Notes:**
Instead, what you want to do in picking a contract is
Use flexible contracting (i.e., T&M or cost-plus) that accommodates the uncertainties inherent in the software process, allows for an iterative process, and utilizes what is learned through open communications.
Use an incremental approach (task-ordered, multi-phase contracts, etc).  Examples of activities are rapid prototyping, requirements walkthroughs, revisions to requirements, relaxing difficult to meet requirements.  Attach draft requirements to the RFP with the idea that they will be fleshed out after contract award.
Call out software activities and products in the RFP
Gain full access to subcontractor(s) by putting the appropriate language in the contract so you can team with them as well 
Separate software from construction, which allows software contractor to be prime.  This is a lead-in to the next topic...

## How You Can Help

**1.  Provide adequate resources**
**2.  Foster teaming**
**3.  See if you can buy instead of build**
**4.  Break the project into pieces**
**5.  Pick the right contract**

**6.  Pick the right contractor**

**Notes:**
Once we’ve picked the right contract vehicle, how do we go about picking the right contractor to carry it out?

## DO

- Use software-related source selection criteria

- Select contractor with strong software capabilities

- Describe how you will control software requirements growth and volatility.
- Describe how you will trace requirements to ...
- How did you estimate the software cost and size?
- How will you identify defects in the ...
- Describe how you track defects.
- How will you track interfaces?
- Describe task exit criteria and how you ensure they are satisfied.
- How do you train staff on your procedures and ensure they will follow them?
- Reference:  *The Guidebook of Software Acquisition Questions, *Software Program Managers Network, www.spmn.com

- 6.  Pick the Right Contractor

**Notes:**
Usually software is after-thought.  Software contractor may not even be on-board when contract is awarded (prime picks someone later).  
Main idea:  Picking the right contractor is essential.  
Now that sounds like motherhood, so how do we make it happen?
Make software a main focus of the RFP with software-related tasks and deliverables, as well as the usual material on transportation-related experience, staffing plans, progress reports, etc.  Focus on the software engineering processes used to build operational software as well as products.  Contractor internal development  processes are important criteria for getting products.
RFP:  Hold up the SPMN brochure
Here’s some example items.
In addition to the software tasks, can put into the RFP your expectations about teaming, etc. and how the project will be managed.  (Aside:  this in itself is part of the open communications.)
ITS example:  Louisiana SOW
use of software acquisition principles in addition to
software tasks:
design including database, interfaces, security
risk mitigation
software testing (factory through acceptance)
use of 3rd party software
integration
configuration management
When scoring the proposals, weigh the software technical criteria.

## 6.  Pick the right contractor

- Level 1
- Initial

- Level 2
- Repeatable

**Software Capability Maturity Model (SW-CMM)**

- Continuous Process

- Improvement

**Level**

**Level**

**Key Process Areas**

**Key Process Areas**

**Focus**

**Focus**

- 5

- Optimizing

- Product and

- Process Quality

- 4

- Managed

- Engineering

- Process

- 3

- Defined

- Project

- Management

- 2

- Repeatable

- Competent People and Heroics

- 1

- Initial

- Level 3
- Defined

- Level 4
- Managed

- Level 5
- Optimizing

- Software Project Planning
- Software Project Tracking and Oversight
- Software Subcontract Management
- Software Quality Assurance
- Software Configuration Management
- Requirements Management

- Software Process Definition
- Organization Process Focus
- Peer Reviews
- Training Program
- Intergroup Coordination
- Software Product Engineering
- Integrated Software Management

- Process Measurement and Analysis
- Quality Management

- Defect Prevention
- Technology Innovation
- Process Change Management

**Notes:**
Main idea:  Introduction to Software Capability Maturity Model (CMM) as a possible tool for ascertaining contractor software credentials and picking the right contractor.
An option that can assist you in picking the right contractor is something called the Software CMM developed by the Software Engineering Institute.  Contractors are scored on the processes they use, with level 1 being the lowest level of software maturity (and the most common) and level 5 the highest.  Consider putting a requirement in the RFP that the contractor be at a certain level--2 or 3 (or equivalent) would be appropriate for ITS.  The downside is that in the transportation community there is a dearth of certified contractors.  So another option is to apply the criteria more informally, perhaps picking out a couple of the key process areas from the CMM and using that in the RFP.

## How You Can Help

**1.  Provide adequate resources**
**2.  Foster teaming**
**3.  See if you can buy instead of build**
**4.  Break the project into pieces**
**5.  Pick the right contract**
**6.  Pick the right contractor**

**7.  Resist schedule pressure**

**Notes:**
Once you’ve selected the right contract vehicle and got the right contractor on board, what can you do to help manage the project.  Here’s an important way:  resist schedule pressure.

## 7.  Resist Schedule Pressure#1 Cause of Failure =Excessive Schedule Pressure

**Notes:**
Project retrospectives show that the best way to ensure project failure is to squeeze the schedule.  By squeezing the schedule you won’t get the product any sooner, you’ll just ensure that it’ll be late.

## 7.  Resist Schedule Pressure#1 Cause of Failure =Excessive Schedule Pressure

**Time Allowed on Schedule**

**$**

**Notes:**
This is a plot of how much a project will cost vs. how much time was allowed on the schedule.

If the instructor desires, they can mark a point on the chart and ask the students what they think would happen if you allowed more time.  Less time.  Would the point go up or down?

Studies and models show the curve looks like this.

The good, somewhat unintuitive news, is that you can save money by stretching the schedule.  (Software is different!)

More good news:  Not only that, but because processes are carried out in a more orderly fashion, quality (in terms of fewer defects) also goes up.

The bad news is that by squeezing the schedule, you pay more.  Lots more.  It’s not linear, and some models even show a 4th power exponent relationship.
Aside:  Can explain the reasons why.  Doing lots of work in parallel that should take place in series.  Loss of productivity as there’s a lot more communication between parties that needs to take place with overlapping activities.  Basically you can squeeze the schedule to a limited extent, but with a tremendous loss in productivity.

Even worse, there’s a limit beyond which you can’t go, now matter how hard you squeeze.  The impossible-to-do zone.  (Retrospectives show no project anywhere at anytime of a particular size and complexity has succeeded in the range.)  Yet, estimates are that maybe as much as 70% of all projects are set in the impossible-to-do zone!  In other words you are doomed to failure from the outset, no matter how effective you are in managing the project.  And as people scurry about trying to meet impossible dictates they know they can’t meet, morale suffers, and they take shortcuts resulting in poorer quality (right axis) and more re-work that ultimately leads to more cost and schedule overruns.
“The tried and true ways of blasting through a schedule barrier--more workers, money, overtime, computer time--don’t seem to work for software.”
Putnam and Myers, 1992
Another example of how software is different.

##   Dictate schedules

			Play catch up

- 			Play catch up

**DON’T**

- 7.  Resist Schedule Pressure

**DON’T**

**Notes:**
We get into the mess when schedules are dictated and bear little resemblance to the requirements to be developed.  So
Don’t dictate schedules.  Very often schedules are set for political reasons, bear no resemblance to the work to be performed, and end up in the impossible-to-do zone.
And even if you don’t make impossible schedules, things still slip.  If they do...
Don’t play catch up (remember Brook’s Law).  After all if you can’t meet the schedule you set, how can you hope to do even more than the schedule in the remaining time? Robert L. Glass, on page 74 of his book “Software Runaways” quotes the program manager for IBM on the FAA’s AAS project “We’re confident that if we encounter any early delays, that those would be overcome by progress made later.”  [AAS turned out to be one of the costliest software disasters in history.] 

## DO

- Build planning and reviews into the schedule
- Revise schedule with project
- Relax or eliminate requirements

**DO**

- 7.  Resist Schedule Pressure

**Notes:**
Main idea:  Tough questions you can ask to make sure they are done.
Start out by setting the schedule using technical criteria.  (There are models and techniques for doing this.  Perhaps ask the contractor in the RFP how they go about it.)  “How did you set the schedule; what was the basis for it?”
Allow for reviews instead of building in schedule overruns.  “Does the schedule have review periods built into it”?
If in spite of best efforts to set realistic schedules, project experience shows that contractor productivity cannot keep up with the planned schedule, or if things change (e.g., more requirements have been identified), 
either
Change the schedule to accommodate changing circumstances  “Does the schedule reflect current realities”?
or
Relax/eliminate/substitute requirements so you will stay on schedule.  Some requirements changes are inevitable; the key is controlling the process.
(counterpart to the Don’t continually run to try and catch up).
Either way, continue to base the schedule on technical criteria so that the schedule accurately reflects the product you’re trying to produce (project scope = product scope).
If things change and you don’t adjust the schedule accordingly:  This is an implicit way of squeezing the schedule.  Even if you start with a reasonable schedule, if you keep adding functions without giving schedule relief you’re in effect squeezing.
Aside:  how do you know if you’re running behind schedule?  That’s a whole story in itself, but there are formal techniques.  Collecting data on status of the project (“metrics”) is part of the story.  Again, in selecting the contractor, one of the questions that can be asked is how they go about doing this.

## How You Can Help

**1.  Provide adequate resources**
**2.  Foster teaming**
**3.  See if you can buy instead of build**
**4.  Break the project into pieces**
**5.  Pick the right contract**
**6.  Pick the right contractor**
**7.  Resist schedule pressure**

**8.  Emphasize quality**

**Notes:**
If you’re not going to apply schedule pressure, then what can you do?  Answer is to emphasize quality.  Now that may sound like motherhood, so let’s discuss what we mean by that.

## $

- 8.  Emphasize Quality

**Notes:**
Main idea:  The other key item to consider is quality.

We’ve seen this slide before, showing that managers are concerned with budget and schedule.

Now we’ll add one more thing that should be of prime concern:  quality.
“Ideal [software] managers ask about quality every time they ask about schedule.” -- SEI Symposium, Summer 2000

## Emphasize Quality Cost to Fix a Problem

***“******Measure twice, cut once.”***

- Time

***“******Design twice, code once.”***

**Notes:**
Main idea:  The longer you wait to address problems, the more it will cost.

As you proceed from requirements through design and integration to operations, the longer you wait to fix a problem, the most it will cost.
Most errors are in the requirements and the design, not in the coding.  So put the effort there.  Catching them earlier means less re-work, which means less cost, shorter schedule.  Need to allow time on schedule to do this, which can give the appearance that no progress is being made.

The graphic is the answer to “But we can’t afford to build in all the quality”. Studies show that organizations who emphasize quality actually produce software for less money and time.  Productivity goes up with quality.  

The reason:  The cost to fix a problem goes up rapidly the longer you wait to fix it.  Depending on the study, the numbers vary somewhat, but a factor of something like 100:1 from top to bottom is typical.  Less re-work is needed if quality is built in from the start.

There’s an old expression that tailors use:  “Measure twice, cut once.”  

The software equivalent is “Design twice, cut once.” -- www.spmn.com
In particular, it’s important to spend a significant amount of time on requirements.  (Related to our previous item, of not squeezing the schedule.)  Important to get them right.  Allow for this in the schedule.  Once the requirements are “done”, sign them, and then re-cost and re-schedule the rest of the project.

## You Can’t Test-In Quality

- 	*“**Testing shows the presence, **not the absence, of bugs.”*

- The bad news is...

- 8.  Emphasize Quality

**Notes:**
Main idea:  Testing won’t ensure quality.
Unfortunately, testing takes out problems; it doesn’t insert quality.  At best, testing catches 1/2 the bugs.  
And quality is not something, like a function, that can be added later. 
Therefore...

## Emphasize Quality Build in Quality From the Start

**Notes:**
Main idea:  Build in quality from the start rather than trying to test it in later.

...It’s more effective to reduce the error insertion rate. 

The good news is that once again...
there are many techniques for doing just that.
And we’ve listed some of them here, (we’re not going to discuss them--just noting that they do indeed exist) including a host of acronyms.
formal completion criteria for completing one step and going onto the next (e.g, all modules have to have a certain level of testing before they can be integrated), metrics, etc.  Part of the way to get better insight as to true status of project schedule and budget.
[the ones on the right are deliberately chosen to be a bunch of buzz words; in case someone asks, they stand for “software quality assurance”, “verification and validation”, “personal software practice”, “team software practice”.  Some of these terms may be trademarked.] 

The customer typically doesn’t get that involved in these, but it’s a case of picking the right contractor who does these as a matter of course (e.g., they train their staff on the techniques) and making sure they follow their procedures.  (CHART uses a V&V contractor for process.)

Aside:  Consider having your staff trained on the techniques by the contractor.  Witness various QA functions by the contractor when they perform them.

## Emphasize Quality Quality-Related QuestionsBefore Contract Award

- How do you find defects...
- in requirements?
- in architecture?
- in design?
- in code?
- How do you track defects?
- What tools do you use?
- Describe your software QA organization, processes, and products.

**Notes:**
Another aspect of picking the right contractor.

## Emphasize Quality Quality-Related Questions After Contract Award

- How many defects
- have been found?
- have been removed?
- remain?
- At historical defect removal rates, how long will it take to achieve acceptable quality?

**Notes:**
Staff become cynical because they say that although management plays lip service to quality, when push comes to shove, they forgo quality in favor of schedule.
If you always ask about schedule and budget, they’ll get the message.  By all means, continue to be concerned about those items, but supplement it with questions about quality.  “Ideal TSP project managers ask about quality every time they ask about schedule.”--Software Engineering Institute, Introduction to the TSP and PSP, SEI Symposium, Summer 2000.
Here’s another place where you can read the warning signs incorrectly:  A contractor is behind schedule and suddenly catches up.  Is that a good sign?  Probably not.  Usually means they’re sacrificing quality (forgoing quality assurance steps).
The last question shows quality assurance driving schedule and not the schedule determining how much quality assurance will be done.

## Rely only on testingto ensure quality

**DON’T**

- 8.  Emphasize Quality

**Notes:**
Have to build in the quality, can’t test it in.
Although we are promoting an emphasis on quality, we emphatically do not do so to the exclusion of schedule and budget considerations.  All three must be managed, with tradeoffs among them.

## DO

**Ask quality-related questions**
**Allow sufficient time for planning and requirements development**
**Test**

**DO**

- 8.  Emphasize Quality

**Notes:**
Encourage the use of quality by emphasizing 
Ask quality-related questions:
Before contract award to select the right contractor.  
After contract award to focus your managers towards quality and not just cost and schedule pressures.
Build in the quality by allowing sufficient time for requirements development (related to previous “what you can do item”).  Once the requirements are developed, use them to re-cost and re-schedule the development aspects of the project.  This may lead to an iterative process (wringing out the overruns before they happen).  When this stabilizes, sign the requirements.

## DON’T

**DON’T**

- 8.  Emphasize Quality

**Notes:**
Have to build in the quality, can’t test it in.
Although we are promoting an emphasis on quality, we emphatically do not do so to the exclusion of schedule and budget considerations.  All three must be managed, with tradeoffs among them.

## Summary

**Summary**

## Contractor at risk
Hold contractor’s feet to the fire
Full functionality

Known results
Familiar contract mechanisms
Budget & schedule pressure

- Hold contractor’s feet to the fire
- Full functionality
- Known results
- Familiar contract mechanisms
- Budget & schedule pressure

- Shared risk
- Teaming & open communication
- 80% is good enough
- Learn as you go
- Unfamiliar contract mechanisms
- Emphasize quality along with budget & schedule

**The Better Road to Software Acquisition**

**Notes:**
Main idea:  You have to make a choice as to how to approach your software projects.

Clearly the goal is to have successful projects, delivering fully functionality, high quality products. To some extent, the path you take is not the important thing in the long run, it’s the results you achieve.  So we shouldn’t be wedded to any particular practice to the detriment of achieving our goals.  So the question is, “What is the best way to get there”? 

 On the surface, it would appear that the scenario we’ve recommended (bullet items on the right side of the chart) is a fairly unattractive one.  Why would you subject yourself to this, when what you really want are the items listed on the left side.
Why assume the risk of a T&M contract, when contractor assumes all the risk on a low bid?  Why accept 80% when what you really want is full functionality, especially since you get the functionality incrementally and not with the first delivery?  Why fund staff positions and tie up your people when it would be nice to “throw it over the fence” at the contractor?  Why go with unfamiliar contracting mechanisms instead of those you are comfortable with?  Why do maintenance to fix buggy software when the contractor should warranty their work?

Certainly no one would go with the angst of the right side if the left side, easy way out, were available. However we would offer that the left side won’t lead to the success you desire.  Instead...

## Recap8 Things You Can Do To Help

**1.  Provide adequate resources**
**2.  Foster teaming**
**3.  See if you can buy instead of build**
**4.  Break the project into pieces**
**5.  Pick the right contract**
**6.  Pick the right contractor**
**7.  Resist schedule pressure**
**8.  Emphasize quality**

**Notes:**
Again, the 8 ways you can help.  They also appear in your notebook along with a list of all the DO’s and DON’Ts.

## Additional Resources

**Courses**
**NHI:*** *_www.nhi.fhwa.dot.gov_
*ITS Software Acquisition *#137019
*Intelligent Transportation System (ITS) Procurement *#137020
**CITE:*** *_www.citeconsortium.org_
*Managing High-Technology Projects in Transportation*

**Notes:**
Hand outs:
ITS Software Acquisition Course flyer; it’s the companion course to this seminar
The Road to Successful Software Acquisition:  Executive Summary
Display:
The Road to Successful Software Acquisition checklists
ITS Software Effective Acquisition Practices (costs $80; $67 for AASHTO members; somewhat less in bulk quantities)
SPMN’s Best Practices brochure
Guide to Software Acquisition Best Practices:  other SPMN guidebooks include Guidebook of Software Acquisition Questions, can be used for structuring an RFP
Peer-to-peer program:  software is one of the topic areas.  Way to access free help from those who have been there before.

## Additional Resources (cont’d.)

**Reference Materials**
***The Road to Successful ITS Software Acquisition***
- 	EDL documents 4130, 4131, 4132 at _http://www.its.dot.gov/library.htm_
- 	USDOT Publications FHWA-JPO-98-035, -036, and -037
***ITS Software: Effective Acquisition Practices***
- AASHTO Publications at (800) 231-3475
***Guide to Software Acquisition Best Practices***
- Software Program Managers Network _www.spmn.com_

## Additional Help (cont’d.)

**The ITS Peer-to-Peer Program**
-   (888) 700-7337
-   _P2P@volpe.dot.gov_
-   _http://www.pcb.its.dot.gov/p2p_2.asp_

**Notes:**
ITS Software Effective Acquisition Practices (as of June 2006, costs $85; $68 for AASHTO members; somewhat less in bulk quantities)
Peer-to-peer program:  software is one of the topic areas.  Way to access free help from those who have been there before.

## Contact Information

**Mac Lister**

**Art Salwin**

**arthur.salwin@mitretek.org**

**Mac.Lister@fhwa.dot.gov**