**AMS**

**Best Practices and Learned Lessons**

+----+---------+-------------+----------------------------------------+
| ** | **Act   | **Best      | **Service Team Testimony**             |
| It | ivity** | Practice or |                                        |
| em |         | Learned     |                                        |
| ** | **Or**  | Lesson**    |                                        |
|    |         |             |                                        |
|    | **Disci |             |                                        |
|    | pline** |             |                                        |
+----+---------+-------------+----------------------------------------+
| *  | **JRC   | Teams that  | The acquisition executive and other    |
| *1 | re      | coordinate  | JRC members expect full compliance     |
| ** | adiness | early and   | with AMS. Programs can not present to  |
|    | pr      | c           | the JRC without meeting AMS            |
|    | ocess** | ontinuously | requirements.                          |
|    |         | with        |                                        |
|    |         | process     | Using the full AMS process ensures:    |
|    |         | stakeholder |                                        |
|    |         | or          | 1\) The JRC has all the information    |
|    |         | ganizations | needed to make timely, informed, and   |
|    |         | have a      | effective decisions                    |
|    |         | smoother    |                                        |
|    |         | path to     | 2\) Agency compliance with public law  |
|    |         | successful  |                                        |
|    |         | investment  | The JRC readiness process helps        |
|    |         | decisions   | communicate AMS requirements to        |
|    |         |             | service teams and improves team        |
|    |         |             | preparation for obtaining JRC          |
|    |         |             | decisions. All key players attend      |
|    |         |             | (ATO-F, ATO-A, ATO-P) and immediate    |
|    |         |             | feedback is obtained.                  |
+----+---------+-------------+----------------------------------------+
| *  | *       | Market      |                                        |
| *2 | *Market | analysis    |                                        |
| ** | Ana     | yields a    |                                        |
|    | lysis** | larger      |                                        |
|    |         | competitive |                                        |
|    |         | pool and    |                                        |
|    |         | better view |                                        |
|    |         | of          |                                        |
|    |         | contractor  |                                        |
|    |         | capability  |                                        |
+----+---------+-------------+----------------------------------------+
| *  | **Risk  | Identify    | Assessing the risks of each            |
| *3 | manag   | and assess  | alternative during concept and         |
| ** | ement** | risk from   | requirements definition helps identify |
|    |         | the         | the best approaches to evaluate during |
|    |         | earliest    | initial investment analysis.           |
|    |         | lifecycle   |                                        |
|    |         | management  | Identifying and assessing risk items   |
|    |         | phases      | during final investment analysis       |
|    |         |             | enables adequate resources to be       |
|    |         |             | available at the right time for        |
|    |         |             | program needs.                         |
|    |         |             |                                        |
|    |         |             | Early identification and               |
|    |         |             | characterization of risk facilitates   |
|    |         |             | the implementation of mitigation       |
|    |         |             | strategies before they become too      |
|    |         |             | expensive.                             |
|    |         |             |                                        |
|    |         |             | Monthly risk management meetings in    |
|    |         |             | which risks are evaluated, solutions   |
|    |         |             | developed, and responsibilities        |
|    |         |             | assigned enable program milestones to  |
|    |         |             | be preserved.                          |
+----+---------+-------------+----------------------------------------+
| *  | **      | High-level  | Investment analysis should be          |
| *4 | Require | r           | performed using a specification        |
| ** | ments** | equirements | derived from the program requirements  |
|    |         | in the      | document. An investment analysis team  |
|    |         | program     | cannot derive any valid estimate based |
|    |         | r           | on high-level requirements in the      |
|    |         | equirements | program requirements document.         |
|    |         | document    |                                        |
|    |         | are not     |                                        |
|    |         | sufficient  |                                        |
|    |         | for costing |                                        |
|    |         | and         |                                        |
|    |         | evaluating  |                                        |
|    |         | a           |                                        |
|    |         | lternatives |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Describe    |                                        |
|    |         | r           |                                        |
|    |         | equirements |                                        |
|    |         | in          |                                        |
|    |         | commercial  |                                        |
|    |         | terms       |                                        |
|    |         | rather than |                                        |
|    |         | government  |                                        |
|    |         | jargon      |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Tailor your | The ATCBI-6 product team made a        |
|    |         | data        | conscious effort to acquire/procure    |
|    |         | r           | only those drawings required to define |
|    |         | equirements | the product baseline. This was done in |
|    |         | carefully   | lieu of ordering a reprocurement data  |
|    |         | to specify  | package. Separate engineering data     |
|    |         | only what   | packages were ordered with the         |
|    |         | you truly   | second-level engineering and organic   |
|    |         | need        | depot repair packages. These were an   |
|    |         |             | appropriate match to the system        |
|    |         |             | requirement. The contractor is         |
|    |         |             | required to maintain these data        |
|    |         |             | packages.                              |
|    |         |             |                                        |
|    |         |             | We also tailored our software licenses |
|    |         |             | to allow the FAA to receive licenses   |
|    |         |             | to maintain and operate the system. By |
|    |         |             | not purchasing a reprocurement data    |
|    |         |             | package, which may have otherwise      |
|    |         |             | inhibited the contractor from          |
|    |         |             | revealing data thought to be           |
|    |         |             | proprietary, the FAA received the      |
|    |         |             | contractor\'s best product.            |
+----+---------+-------------+----------------------------------------+
|    |         | Use         | The team analyzed each program         |
|    |         | automated   | requirement during development of the  |
|    |         | tools to    | product specification to ensure it     |
|    |         | trace user  | accurately reflected user needs.       |
|    |         | r           | Results of this review were documented |
|    |         | equirements | in a requirements analysis report. The |
|    |         | from the    | user was required to certify that the  |
|    |         | program     | report accurately reflected their      |
|    |         | r           | need. This made development of a       |
|    |         | equirements | specification more efficient and       |
|    |         | document to | established the groundwork for         |
|    |         | the product | traceability of requirements           |
|    |         | sp          | throughout the procurement process.    |
|    |         | ecification | This traceability extended from the    |
|    |         | through     | program requirements document through  |
|    |         | design and  | testing enabling us to validate that   |
|    |         | testing to  | each requirement was achieved.         |
|    |         | ensure all  |                                        |
|    |         | r           |                                        |
|    |         | equirements |                                        |
|    |         | are         |                                        |
|    |         | achieved in |                                        |
|    |         | the end     |                                        |
|    |         | product     |                                        |
+----+---------+-------------+----------------------------------------+
| *  | **Inv   | Assemble a  | Our investment analysis never did jell |
| *5 | estment | team with   | right and we spent a lot of time       |
| ** | a       | clearly     | getting people to do something. The    |
|    | nalysis | defined     | benefits folks would not complete      |
|    | team    | duties and  | their part until the cost part was     |
|    | form    | working     | done. The cost folks would not do      |
|    | ation** | re          | their part until the requirements      |
|    |         | lationships | folks were done. The requirements      |
|    |         | from the    | folks were waiting on the OMB-300 and  |
|    |         | start of    | BCAR to be done.                       |
|    |         | investment  |                                        |
|    |         | analysis    | The bottom line is that the ATO-P and  |
|    |         |             | ATO-F organizations don't have an      |
|    |         |             | efficient process for completing       |
|    |         |             | end-product documents like the OMB-300 |
|    |         |             | and the BCAR. We finally agreed on an  |
|    |         |             | order and schedule for completing the  |
|    |         |             | subtasks leading to the initial        |
|    |         |             | investment decision. We recommend      |
|    |         |             | subsequent teams sort out roles and    |
|    |         |             | responsibilities and develop a process |
|    |         |             | timeline right from the start.         |
+----+---------+-------------+----------------------------------------+
| *  | *       | The 5%      | ATO-F-endorses 5% FTE inflation        |
| *6 | *Afford | inflation   | factor, whereas contract costs are     |
| ** | ability | factor for  | inflation-based, around 3%. It does    |
|    | asses   | FTEs seems  | not take long for FTE cost to grow     |
|    | sment** | wrong.      | much more than contractor cost. This   |
|    |         |             | does not seem right and could lead to  |
|    |         |             | a distorted cost analysis. For this    |
|    |         |             | reason, programs with a 10-year        |
|    |         |             | lifecycle need a different             |
|    |         |             | calculation.                           |
+----+---------+-------------+----------------------------------------+
| *  | *       | Competition | A debate existed at the beginning of   |
| *7 | *Compet | can reduce  | procurement planning between the       |
| ** | ition** | costs       | benefits of competition and the        |
|    |         | drastically | efficiencies of contracting with the   |
|    |         |             | incumbent at the projected price of    |
|    |         | Market      | \$1 million per system. Competition    |
|    |         | analysis    | allowed the government the benefit of  |
|    |         | can         | buying our requirement at a production |
|    |         | discover    | price of \$450K per system, a savings  |
|    |         | unknown     | of \$82.5 million for 150 systems.     |
|    |         | sources     |                                        |
|    |         |             | In a departure from past practice of   |
|    |         |             | purchasing production systems on a     |
|    |         |             | ID/IQ basis, purchases were based upon |
|    |         |             | economic quantity taking advantage of  |
|    |         |             | planned production capacity and        |
|    |         |             | material and part discounts. This      |
|    |         |             | produced savings of \$15 million. By   |
|    |         |             | buying off an existing production      |
|    |         |             | line, we were able to save \$4 million |
|    |         |             | in startup cost. A shorter program     |
|    |         |             | schedule saved an additional \$7       |
|    |         |             | million.                               |
+----+---------+-------------+----------------------------------------+
| *  | **Qu    | The ability | AMS enables us to pre-qualify vendors, |
| *8 | alified | to          | thereby enabling solicitations to be   |
| ** | vendor  | pre-qualify | sent to 3 qualified contractors rather |
|    | lists** | vendors     | than 50 to 60 solicitations we would   |
|    |         | offers      | have sent out pre-AMS. This enabled us |
|    |         | s           | to streamline the whole procurement    |
|    |         | treamlining | process. The time from advertising to  |
|    |         | and         | award took only 2 months. Contrast     |
|    |         | time-saving | this with pre-AMS procedures, where it |
|    |         | advantages  | would have taken as long as a year.    |
|    |         |             |                                        |
|    |         |             | Eliminated vendors were pleased with   |
|    |         |             | the quick responses and the fact that  |
|    |         |             | they didn\'t have to spend long days   |
|    |         |             | reworking proposals when they didn\'t  |
|    |         |             | have a chance. Vendors appreciated our |
|    |         |             | announcing what we planned to do and   |
|    |         |             | then following through with that plan. |
+----+---------+-------------+----------------------------------------+
|    |         | Use of a    | Award of contracts since establishment |
|    |         | qualified   | of the QVL is two weeks or less.       |
|    |         | vendors     | Quality firms selected for various     |
|    |         | list can    | environmental requirements frequently  |
|    |         | accelerate  | used in the region have been screened  |
|    |         | contract    | and identified. When an emergency      |
|    |         | award to 2  | environmental project is identified,   |
|    |         | weeks or    | we have firms available to respond     |
|    |         | less        | quickly. Cost savings on a recent      |
|    |         |             | award was 113% lower price than        |
|    |         |             | estimated by the FAA based on previous |
|    |         |             | prices.                                |
+----+---------+-------------+----------------------------------------+
| *  | **GSA   | Some useful | 1\. Make sure the vendors are in the   |
| *9 | sch     | best        | business of performing the services    |
| ** | edule** | practices   | you require                            |
|    |         | when        |                                        |
|    |         | soliciting  | 2\. Make sure they are interested in   |
|    |         | vendors     | your procurement                       |
|    |         | from the    |                                        |
|    |         | GSA         | 3\. Allow a reasonable time for them   |
|    |         | schedule    | to prepare the information required by |
|    |         |             | the SIR                                |
|    |         |             |                                        |
|    |         |             | Only 2 of 10 vendors responded to our  |
|    |         |             | SIR. Neither could meet the            |
|    |         |             | requirements. We lost 3 weeks          |
|    |         |             | evaluating responses, notifying        |
|    |         |             | vendors, and advertising the           |
|    |         |             | requirements on the internet and       |
|    |         |             | issuing another SIR.                   |
+----+---------+-------------+----------------------------------------+
| ** | **Per   | Perfor      | An acquisition for telecommunication   |
| 10 | formanc | mance-based | services was initially turned over to  |
| ** | e-based | contracting | another government entity to develop a |
|    | contra  | greatly     | contract for the FAA. They used a      |
|    | cting** | simplifies  | detailed \"how to\" statement of work, |
|    |         | contract    | which had a negative impact causing    |
|    |         | award and   | the acquisition to become mired in     |
|    |         | adm         | "how to" details that caused numerous  |
|    |         | inistration | time extensions and amendments.        |
|    |         |             |                                        |
|    |         |             | About 9 months later, the FAA took     |
|    |         |             | back the procurement and changed the   |
|    |         |             | approach completely. First, we         |
|    |         |             | conducted a market survey that         |
|    |         |             | revealed the availability of           |
|    |         |             | competition. This shifted the          |
|    |         |             | procurement from sole source to        |
|    |         |             | competition. Second, the statement of  |
|    |         |             | work was restructured as               |
|    |         |             | performance-based. As a result, we     |
|    |         |             | issued a SIR one month after taking    |
|    |         |             | back the procurement, closed it 9 days |
|    |         |             | later, and awarded a contract within 5 |
|    |         |             | weeks.                                 |
+----+---------+-------------+----------------------------------------+
| ** | **IDIQ  | Indefinite  | An IDIQ contract was awarded for BUSS  |
| 11 | cont    | Delivery    | systems used within many regions. With |
| ** | racts** | Indefinite  | all initial contracting work done,     |
|    |         | Quantify    | regions just issue delivery orders     |
|    |         | contracts   | against prices already obtained. This  |
|    |         | can satisfy | saves a lot of time and money since    |
|    |         | nat         | the initial contracting work is done   |
|    |         | ional-level | only one time.                         |
|    |         | r           |                                        |
|    |         | equirements |                                        |
|    |         | efficiently |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **In    | Allow       | 1\) (Positive)Performance-Based Award  |
| 12 | centive | offerors to | Fee                                    |
| ** | cont    | propose     |                                        |
|    | racts** | both        | a\. Positive incentive - Base fee,     |
|    |         | positive    | plus award fee amount                  |
|    |         | and         |                                        |
|    |         | negative    | b\. Quarterly decision based on how    |
|    |         | incentives  | effectively the contractor achieved    |
|    |         | for         | specific negotiated measures of        |
|    |         | perfor      | effectiveness tied to program          |
|    |         | mance-based | objectives                             |
|    |         | contracts   |                                        |
|    |         |             | 2\) (Negative) Credits owed for not    |
|    |         |             | meeting service requirements. The      |
|    |         |             | following must be clearly stated:      |
|    |         |             |                                        |
|    |         |             | a\. The requirement                    |
|    |         |             |                                        |
|    |         |             | b\. The criterion that will trigger    |
|    |         |             | the credit                             |
|    |         |             |                                        |
|    |         |             | c\. The measurement method             |
|    |         |             |                                        |
|    |         |             | d\. The credit amount                  |
+----+---------+-------------+----------------------------------------+
|    |         | Fixed-price | Incentives built into our contract     |
|    |         | incentive   | resulted in a 10-month early           |
|    |         | contracts   | acceptance of the capability we        |
|    |         | offer       | procured.                              |
|    |         | rewards to  |                                        |
|    |         | both        |                                        |
|    |         | contractors |                                        |
|    |         | and the FAA |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Comme | When        | Our schedule estimate proved to be     |
| 13 | rcial-o | planning a  | unrealistic because the program could  |
| ** | ff-the- | COTS        | not be satisfied by a true off the     |
|    | shelf** | p           | shelf solution. Moreover, the          |
|    |         | rocurement, | requirements document was too vague,   |
|    |         | make sure   | which caused considerable angst during |
|    |         | your        | the evaluation. There were too many    |
|    |         | r           | interpretations of what the            |
|    |         | equirements | requirements really meant.             |
|    |         | are sound   |                                        |
|    |         | and the     |                                        |
|    |         | solution is |                                        |
|    |         | truly       |                                        |
|    |         | of          |                                        |
|    |         | f-the-shelf |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Procurement |                                        |
|    |         | of COTS     |                                        |
|    |         | must focus  |                                        |
|    |         | on          |                                        |
|    |         | measurement |                                        |
|    |         | of          |                                        |
|    |         | d           |                                        |
|    |         | emonstrated |                                        |
|    |         | performance |                                        |
|    |         | as opposed  |                                        |
|    |         | to          |                                        |
|    |         | measurement |                                        |
|    |         | of paper    |                                        |
|    |         | proposals   |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | When buying | The FAA team never considered we would |
|    |         | commercial  | have to sign up to the vendor\'s terms |
|    |         | software,   | and conditions, never asked to see the |
|    |         | be prepared | vendor\'s license agreements, and      |
|    |         | to          | never discussed licensing terms. As a  |
|    |         | negotiate   | result, we spent a lot of time under   |
|    |         | license     | pressure negotiating software          |
|    |         | agreements  | licensing terms after we thought a     |
|    |         |             | deal had been made. Make sure the      |
|    |         |             | license agreement terms and conditions |
|    |         |             | do not waive or conflict with FAA      |
|    |         |             | contract terms and conditions.         |
+----+---------+-------------+----------------------------------------+
| ** | **Other | The "Other  | "Other Transactions" are not subject   |
| 14 | Transac | Tr          | to procurement statutes and            |
| ** | tions** | ansactions" | regulations. They are subject to       |
|    |         | authority   | fiscal laws and statutes of general    |
|    |         | in PL 49    | applicability. An OT is not a standard |
|    |         | USC 106     | procurement contract, grant, or        |
|    |         | offers      | cooperative agreement and protest      |
|    |         | great       | rules do not apply. It allows greater  |
|    |         | flexibility | flexibility and permits use of best    |
|    |         | in certain  | business practices. OTs can be used on |
|    |         | situations  | research and development and prototype |
|    |         |             | requirements, but may not be used to   |
|    |         |             | acquire goods and services for the     |
|    |         |             | direct benefit or use of the           |
|    |         |             | government.                            |
|    |         |             |                                        |
|    |         |             | The LAAS program needed to proceed,    |
|    |         |             | but program funds were not available   |
|    |         |             | in the FY budget. Without OT           |
|    |         |             | authority, this program would have to  |
|    |         |             | wait 5 years for budgeted funds. LAAS  |
|    |         |             | is using the OT to establish a         |
|    |         |             | government and industry partnership    |
|    |         |             | that will develop, test, and certify a |
|    |         |             | LAAS system. Once systems are          |
|    |         |             | certified, FAA may choose to purchase  |
|    |         |             | or not. The industry will have a FAA   |
|    |         |             | certified system to market both        |
|    |         |             | domestically and internationally.      |
|    |         |             |                                        |
|    |         |             | During the first stage, FAA will       |
|    |         |             | provide only in-kind resources. In the |
|    |         |             | second stage, some funding will be     |
|    |         |             | provided but not full funding. All     |
|    |         |             | data and rights will be retained by    |
|    |         |             | industry participants. Documentation   |
|    |         |             | is not be subject to FOIA procedures.  |
|    |         |             |                                        |
|    |         |             | Industry participants are not required |
|    |         |             | to produce anything (except high-level |
|    |         |             | and certification documentation).      |
|    |         |             | Either party can cancel the OT. The    |
|    |         |             | FAA is not required to purchase the    |
|    |         |             | developed system. The FAA will ensure  |
|    |         |             | developed systems meet FAA needs       |
|    |         |             | should we decide to purchase any. The  |
|    |         |             | developer gets the benefit of in-kind  |
|    |         |             | resources and an FAA-certified system  |
|    |         |             | for commercial marketing. The FAA gets |
|    |         |             | the results of the research and the    |
|    |         |             | potential for a developed system       |
|    |         |             | without the traditional cost and       |
|    |         |             | contract restraints.                   |
+----+---------+-------------+----------------------------------------+
| ** | **OMB   | Review OMB  | The ICE-MAN SIR and RFP were           |
| 15 | C       | Circular    | structured to allow government         |
| ** | ircular | A-76        | entities to compete with commercial    |
|    | A-76**  | carefully   | vendors. The question of the           |
|    |         | and obtain  | applicability of OMB Circular A-76     |
|    |         | legal       | arose. The FAA decided to not apply or |
|    |         | counsel     | adhere to A-76 procedures. The initial |
|    |         | when        | award was made based on best value.    |
|    |         | seeking     | Since the award selection was a        |
|    |         | offers from | government entity, it was unclear      |
|    |         | both        | whether A-76 procedures should have    |
|    |         | government  | been applied. To address this concern, |
|    |         | and         | the award was suspended, and the       |
|    |         | commercial  | awardee was requested to submit their  |
|    |         | offerors    | proposal (unchanged technically) in    |
|    |         |             | accordance with A-76 procedures. This  |
|    |         |             | proposal and the next best             |
|    |         |             | non-government proposal were evaluated |
|    |         |             | using the A-76 mechanism. This         |
|    |         |             | evaluation showed that the original    |
|    |         |             | selection was still the best value for |
|    |         |             | the FAA. Results were published and    |
|    |         |             | all competing offerors were given an   |
|    |         |             | opportunity to appeal the decision. No |
|    |         |             | appeals were filed and no protests     |
|    |         |             | were received. The award was           |
|    |         |             | reinstated to the original awardee.    |
|    |         |             | Under FAR procedures, this probably    |
|    |         |             | would have resulted in cancellation of |
|    |         |             | the procurement and starting over      |
|    |         |             | using A-76 procedures. The open        |
|    |         |             | communication with offerors helped     |
|    |         |             | them to understand the FAA\'s          |
|    |         |             | position, the fairness of our process, |
|    |         |             | and the desire to select the \"best    |
|    |         |             | value\". If the FAA had applied or     |
|    |         |             | adhered to the A-76 process from the   |
|    |         |             | start, the solicitation process time   |
|    |         |             | would have increased greatly.          |
|    |         |             | Application of A-76 procedures \"after |
|    |         |             | the fact\" was adequate to ensure the  |
|    |         |             | FAA created a level playing field for  |
|    |         |             | comparing government offers with       |
|    |         |             | commercial offers.                     |
+----+---------+-------------+----------------------------------------+
| ** | **M     | Using       | We used an initial screening SIR to    |
| 16 | ultiple | multiple    | reduce the number of qualified vendors |
| ** | SIRs**  | SIRs to     | from 17 to 7. We saved cost and time   |
|    |         | pr          | in the evaluation process since the    |
|    |         | ogressively | team knew exactly how many proposals   |
|    |         | down-select | to prepare for. In the final           |
|    |         | vendors can | screening, we required oral            |
|    |         | reduce      | presentations and capability           |
|    |         | evaluation  | demonstrations at vendor sites. We     |
|    |         | time and    | required key personnel to give the     |
|    |         | cost        | oral presentations. This allowed       |
|    |         |             | evaluators to ask questions and        |
|    |         |             | observe responses and it eliminated    |
|    |         |             | consultants preparing proposals. This  |
|    |         |             | gave us a first-hand indication of the |
|    |         |             | capabilities of the key person         |
|    |         |             | assigned to the project. We evaluated  |
|    |         |             | past performance in the final SIR      |
|    |         |             | using a best-value subjective          |
|    |         |             | determination. Each vendor was fully   |
|    |         |             | evaluated and documented before moving |
|    |         |             | to the next one.                       |
+----+---------+-------------+----------------------------------------+
|    |         | Send the    | Competing vendors were identified from |
|    |         | draft       | responses to our initial SIR. We sent  |
|    |         | sp          | our draft specifications to those      |
|    |         | ecification | vendors for comment while we worked on |
|    |         | to screened | the RFP. This improved the quality of  |
|    |         | vendors for | our specification and enabled vendors  |
|    |         | comment     | to prepare quality proposals more      |
|    |         | when using  | quickly.                               |
|    |         | multiple    |                                        |
|    |         | SIRs        |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Some        | Explicitly state how information       |
|    |         | valuable    | received in response to a previous SIR |
|    |         | tips on the | will be used during evaluation of      |
|    |         | use of      | subsequent SIRs.                       |
|    |         | multiple    |                                        |
|    |         | SIRs        | State whether vendors should respond   |
|    |         |             | to the follow-on SIR with only a delta |
|    |         |             | document or whether vendors need to    |
|    |         |             | resubmit information from previous     |
|    |         |             | submittals. State also what the        |
|    |         |             | procedure will be if information is    |
|    |         |             | ambiguous relative to information      |
|    |         |             | received previously.                   |
|    |         |             |                                        |
|    |         |             | Require vendors to identify all areas  |
|    |         |             | of a technical proposal that have      |
|    |         |             | changed from one SIR to the next. Be   |
|    |         |             | sure the evaluation team is fully      |
|    |         |             | aware of what they can consider during |
|    |         |             | an evaluation. We recommend using no   |
|    |         |             | more than two SIRs.                    |
|    |         |             |                                        |
|    |         |             | A competitive range of 3-5 vendors is  |
|    |         |             | recommended. This creates an           |
|    |         |             | environment for thorough and timely    |
|    |         |             | evaluation, minimizes cost to the      |
|    |         |             | agency and competitors, and provides   |
|    |         |             | fair competition. Too many vendors add |
|    |         |             | to the time and cost of evaluation and |
|    |         |             | little, if any value to the final      |
|    |         |             | selection.                             |
+----+---------+-------------+----------------------------------------+
|    |         | Screening   | Thirteen potential offers responded to |
|    |         | for         | our first SIR for capability           |
|    |         | capability  | statements. After we reviewed the      |
|    |         | can achieve | capability statements, we invited all  |
|    |         | true        | respondents to compete by submitting   |
|    |         | competition | proposals. Under FAR full and open     |
|    |         | without     | competition procedures, we would have  |
|    |         | wasting     | been required to permit other          |
|    |         | contractor  | competitors to join the competition.   |
|    |         | or          | This may have resulted in offers being |
|    |         | government  | received from unacceptable sources,    |
|    |         | resources   | thereby increasing the cost of issuing |
|    |         |             | the solicitation, evaluation time, and |
|    |         |             | the risk of protest. The team would    |
|    |         |             | have invested time dealing with        |
|    |         |             | proposals that had no reasonable       |
|    |         |             | chance of award and those offerors     |
|    |         |             | would have incurred proposal expenses  |
|    |         |             | when they had no reasonable chance of  |
|    |         |             | meeting the requirement.               |
+----+---------+-------------+----------------------------------------+
| ** | **Proc  | The quality | 1\) Set the stage for quality early.   |
| 17 | urement | of the      | Never show industry anything that is   |
| ** | Qu      | procurement | sloppy                                 |
|    | ality** | is largely  |                                        |
|    |         | dependent   | 2\) Preparation of a high quality SIR  |
|    |         | on the      | takes more time than most expect       |
|    |         | quality of  |                                        |
|    |         | the SIR     | 3\) Have discipline in the preparation |
|    |         |             | of every part of the SIR               |
|    |         |             |                                        |
|    |         |             | a\. Establish a configuration          |
|    |         |             | management process                     |
|    |         |             |                                        |
|    |         |             | b\. Have one person in charge of       |
|    |         |             | managing changes                       |
|    |         |             |                                        |
|    |         |             | 4\) Make sure the SIR reads well and   |
|    |         |             | is consistent throughout               |
|    |         |             |                                        |
|    |         |             | 5\) Make the size of the proposal      |
|    |         |             | commensurate with the complexity of    |
|    |         |             | the acquisition. More pages = higher   |
|    |         |             | cost to industry and more time for the |
|    |         |             | FAA review.                            |
|    |         |             |                                        |
|    |         |             | 6\) State each requirement in only one |
|    |         |             | place within the SIR                   |
|    |         |             |                                        |
|    |         |             | 7\) Allow offerors to know your budget |
|    |         |             | constraints and propose accordingly    |
|    |         |             |                                        |
|    |         |             | 8\) Create flexibility where practical |
|    |         |             |                                        |
|    |         |             | a\. Allow offerors to price their      |
|    |         |             | solution while ensuring fair           |
|    |         |             | comparison                             |
|    |         |             |                                        |
|    |         |             | b\. Welcome tailored terms and         |
|    |         |             | conditions                             |
|    |         |             |                                        |
|    |         |             | c\. Allow offerors to take exception   |
|    |         |             | to requirements                        |
+----+---------+-------------+----------------------------------------+
|    |         | Personnel   | Our procurement was slated to be on a  |
|    |         | commitment  | fast track, but the procurement was    |
|    |         | is          | slowed by lack of qualified people to  |
|    |         | essential   | keep the project on track during the   |
|    |         | when        | pre-release and evaluation stages.     |
|    |         | planning a  |                                        |
|    |         | compressed  |                                        |
|    |         | award       |                                        |
|    |         | schedule    |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Proc  | Make sure   | You don't an unfair competitive        |
| 18 | urement | spaces      | advantage going to a snooping          |
| ** | Sec     | around      | contractor.                            |
|    | urity** | s           |                                        |
|    |         | olicitation | You don't want protests either.        |
|    |         | records are |                                        |
|    |         | off limits  |                                        |
|    |         | to offeror  |                                        |
|    |         | personnel   |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **SOW   | State your  | Eastern region awarded a contract to   |
| 19 | prepar  | r           | provide 24-hour weather observation    |
| ** | ation** | equirements | services at Baltimore, Maryland. The   |
|    |         | clearly and | statement of work included a           |
|    |         | s           | requirement for a supervisor, but did  |
|    |         | pecifically | not specify the supervisor was to be   |
|    |         | in the      | full-time and on-site. The contractor  |
|    |         | statement   | is requesting we renegotiate pricing,  |
|    |         | of work     | claiming our specification was not     |
|    |         |             | clear. By not being specific enough in |
|    |         |             | the SOW, we have experienced a lot of  |
|    |         |             | aggravation and wasted energy.         |
+----+---------+-------------+----------------------------------------+
|    |         | Use the     |                                        |
|    |         | procurement |                                        |
|    |         | toolkit in  |                                        |
|    |         | FAST to     |                                        |
|    |         | download    |                                        |
|    |         | and         |                                        |
|    |         | manipulate  |                                        |
|    |         | contract    |                                        |
|    |         | clauses and |                                        |
|    |         | statements  |                                        |
|    |         | of work     |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Co    | Constant    | Constant communications and daily      |
| 20 | mmunica | com         | contact among team members proved to   |
| ** | tions** | munications | be extremely beneficial. The service   |
|    |         | among team  | team structure also facilitated        |
|    |         | members is  | brainstorming. Daily contact between   |
|    |         | extremely   | the contracting officer and program    |
|    |         | beneficial, | manager achieved mutual understanding  |
|    |         | p           | and the prevention of unauthorized     |
|    |         | articularly | procurements. Video conferencing       |
|    |         | between the | facilitated communications with remote |
|    |         | contracting | team members and saved travel money.   |
|    |         | officer and |                                        |
|    |         | program     |                                        |
|    |         | manager     |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | You can     | AMS offers great flexibility in        |
|    |         | save time   | communications with offerors. You can  |
|    |         | and energy  | focus discussions on offerors          |
|    |         | by          | determined most qualified to receive   |
|    |         | tailoring   | an award and limit discussions with    |
|    |         | your        | those determined to be less qualified. |
|    |         | com         |                                        |
|    |         | munications |                                        |
|    |         | with offers |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Early       | Early and direct communications allow  |
|    |         | com         | FAA to learn what was available from   |
|    |         | munications | industry and to develop better         |
|    |         | with        | requirements. Freedom in discussions   |
|    |         | industry    | was valuable throughout the process.   |
|    |         | helps to    | It was an advantage to be able to have |
|    |         | understand  | discussions with only those vendors    |
|    |         | what is     | with whom we had a need. Issuance of a |
|    |         | available   | draft description allowed for industry |
|    |         | and to      | input into a final document.           |
|    |         | better      |                                        |
|    |         | define      |                                        |
|    |         | r           |                                        |
|    |         | equirements |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | A robust    | We established a robust FTI web-site   |
|    |         | s           | that consisted of:                     |
|    |         | olicitation |                                        |
|    |         | web-site    | -   Information documents; e.g.,       |
|    |         | can greatly |     Exhibit 300 program baseline       |
|    |         | facilitate  |                                        |
|    |         | com         | -   Frequently asked questions and     |
|    |         | munications |     answers                            |
|    |         | with        |                                        |
|    |         | vendors     | -   SIR, including all amendments      |
+----+---------+-------------+----------------------------------------+
|    |         | Be careful! | Be careful and alert to what is being  |
|    |         | Open        | said to potential offerors.            |
|    |         | com         | Industry\'s motivation is dollars; the |
|    |         | munications | FAA\'s is to be fair and reasonable.   |
|    |         | provides    | Be smart. Communicate, but protect the |
|    |         | op          | taxpayers and the integrity of the     |
|    |         | portunities | procurement.                           |
|    |         | to          |                                        |
|    |         | prejudice   |                                        |
|    |         | the         |                                        |
|    |         | competition |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Open        |                                        |
|    |         | discussion  |                                        |
|    |         | regarding   |                                        |
|    |         | technical   |                                        |
|    |         | and cost    |                                        |
|    |         | proposals   |                                        |
|    |         | is          |                                        |
|    |         | beneficial  |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Com         | By working issues with contractors     |
|    |         | munications | before they submit their cost          |
|    |         | with        | proposals, contractors are able to     |
|    |         | contractors | propose their fair competitive         |
|    |         | during      | advantages. This yielded major cost    |
|    |         | source      | and schedule savings in the areas of   |
|    |         | selection   | development, testing, logistics, and   |
|    |         | can save    | training.                              |
|    |         | substantial |                                        |
|    |         | time        | The volume and complexity of issues    |
|    |         |             | added 4 weeks to discussions with      |
|    |         |             | contractor. However, the understanding |
|    |         |             | they gained improved their proposals   |
|    |         |             | to the point we were able to complete  |
|    |         |             | our evaluation 3 months ahead of       |
|    |         |             | schedule and award without             |
|    |         |             | negotiations, saving at least one      |
|    |         |             | month on a very ambitious award        |
|    |         |             | schedule.                              |
+----+---------+-------------+----------------------------------------+
|    |         | Open        | Open discussion with the contractor    |
|    |         | com         | resulted in a better understanding of  |
|    |         | munications | what FAA needed and the contractor     |
|    |         | during      | position. Things that are normally     |
|    |         | source      | resolved after the proposal is         |
|    |         | selection   | submitted (i.e., cost proposal format) |
|    |         | facilitate  | were settled before the proposal was   |
|    |         | mutual      | submitted. This helped reduce the      |
|    |         | un          | number of changes and resulted in a    |
|    |         | derstanding | \$1.1M reduction of the contractor\'s  |
|    |         | and result  | original estimate.                     |
|    |         | in better   |                                        |
|    |         | contracts   |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Involve end | The ASDE-X service team developed an   |
|    |         | users from  | early and continuing relationship with |
|    |         | the         | the end-user. They reviewed every key  |
|    |         | earliest    | program planning element like the      |
|    |         | stages of   | program requirements document, product |
|    |         | product     | specification, and draft request for   |
|    |         | planning    | offer. During contract performance,    |
|    |         | and         | they participated in the system        |
|    |         | development | requirements review, the preliminary   |
|    |         |             | design review, and the critical design |
|    |         |             | review. We also involved the National  |
|    |         |             | Air Traffic Controllers Association    |
|    |         |             | and Airway Facilities Unions early in  |
|    |         |             | the process to ensure the end product  |
|    |         |             | would be acceptable to them.           |
+----+---------+-------------+----------------------------------------+
|    |         | Oral        |                                        |
|    |         | discussions |                                        |
|    |         | with        |                                        |
|    |         | contractors |                                        |
|    |         | provided a  |                                        |
|    |         | clearer     |                                        |
|    |         | un          |                                        |
|    |         | derstanding |                                        |
|    |         | of the      |                                        |
|    |         | requirement |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Eva   | Keep your   | Keep your evaluation factors simple,   |
| 21 | luation | evaluation  | explain them clearly in the SIR, and   |
| ** | fa      | factors     | test against your performance goals    |
|    | ctors** | simple and  | before posting. Make sure there is a   |
|    |         | make sure   | total understanding of the factors on  |
|    |         | they are    | your evaluation team.                  |
|    |         | relevant to |                                        |
|    |         | the         | Test completeness of the SOW and       |
|    |         | intended    | sections L and M. If you want to make  |
|    |         | outcome of  | sure offerors propose on everything in |
|    |         | your        | the SOW, cover it in L and M.          |
|    |         | s           |                                        |
|    |         | olicitation | If you are going to use a contractor   |
|    |         |             | for the evaluation, make sure it is    |
|    |         |             | announced in the SIR and make them     |
|    |         |             | sign non-disclosure agreements and     |
|    |         |             | ensure there is no conflict of         |
|    |         |             | interest.                              |
+----+---------+-------------+----------------------------------------+
|    |         | Use best    | It enables you to consider factors     |
|    |         | value as    | other than price when making an award. |
|    |         | the #1      |                                        |
|    |         | decision    |                                        |
|    |         | criteria in |                                        |
|    |         | most cases  |                                        |
|    |         | rather than |                                        |
|    |         | price       |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Use past    |                                        |
|    |         | performance |                                        |
|    |         | as an       |                                        |
|    |         | evaluation  |                                        |
|    |         | factor      |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Use of key  |                                        |
|    |         | dis         |                                        |
|    |         | criminators |                                        |
|    |         | and past    |                                        |
|    |         | performance |                                        |
|    |         | yield       |                                        |
|    |         | quality     |                                        |
|    |         | contractors |                                        |
|    |         | for         |                                        |
|    |         | competition |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Some best   | 1\) Don't wait until most of the SIR   |
|    |         | practices   | is complete - good criteria take time  |
|    |         | for         | to develop!                            |
|    |         | developing  |                                        |
|    |         | good        | 2\) Stay away from evaluation criteria |
|    |         | evaluation  | that include absolutes; i.e.,          |
|    |         | criteria    | pass/fail. Use criteria that allow     |
|    |         |             | judgment.                              |
|    |         |             |                                        |
|    |         |             | 3\) Use adjectival rather than         |
|    |         |             | numerical scoring. Adjectival offers   |
|    |         |             | more flexibility.                      |
|    |         |             |                                        |
|    |         |             | 4\) Work hard on the adjectival        |
|    |         |             | definitions and ensure they are clear. |
|    |         |             |                                        |
|    |         |             | 5\) Limit the number of evaluation     |
|    |         |             | factors. Enumerate specific points in  |
|    |         |             | elements that are considered but not   |
|    |         |             | scored.                                |
|    |         |             |                                        |
|    |         |             | 6\) Assess risk as an integrated part  |
|    |         |             | of your evaluation criteria, not as a  |
|    |         |             | stand-alone criterion.                 |
+----+---------+-------------+----------------------------------------+
| ** | *       | Ensure all  | Schedule a question and answer period  |
| 22 | *Source | evaluators  | to clarify any ambiguities and ensure  |
| ** | evalu   | read and    | there are no misunderstandings.        |
|    | ation** | understand  | Prepare one or two questions that you  |
|    |         | the SOW,    | will ask each contractor at the end of |
|    |         | items to be | their presentation. This is a good way |
|    |         | scored, and | to hear their unprepared responses to  |
|    |         | the scoring | hypothetical situations or to          |
|    |         | criteria    | procedural questions.                  |
|    |         | before      |                                        |
|    |         | beginning   |                                        |
|    |         | the         |                                        |
|    |         | evaluation  |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Keep        | Make sure evaluators capture and       |
|    |         | evaluators  | analyze oral information provided by   |
|    |         | in place    | presenters in personal notes and close |
|    |         | long enough | to the actual presentation time.       |
|    |         | to edit,    | Otherwise, the large amount of         |
|    |         | analyze,    | information will fade from memory as   |
|    |         | and clean   | time goes on. Our sessions were        |
|    |         | up          | videotaped to spot-check items; but,   |
|    |         | evaluation  | the idea was to get the evaluators     |
|    |         | findings    | thoughts down as quickly as possible,  |
|    |         |             | with videotape as a back-up.           |
|    |         |             | Evaluators were brought in from the    |
|    |         |             | regions for short evaluation periods,  |
|    |         |             | and on hindsight, not enough time was  |
|    |         |             | allowed for them to review the         |
|    |         |             | information from oral presentations.   |
|    |         |             | Evaluators should have been asked to   |
|    |         |             | stay an additional week after oral     |
|    |         |             | presentations to edit, analyze, and    |
|    |         |             | clean up findings. As a result,        |
|    |         |             | considerable time was spent            |
|    |         |             | communicating back and forth with      |
|    |         |             | evaluators for clarifications,         |
|    |         |             | explanations, etc., when writing the   |
|    |         |             | evaluation report.                     |
+----+---------+-------------+----------------------------------------+
|    |         | A small     | Using a small team of 6 evaluators     |
|    |         | team of     | saved time and cost. Evaluators were   |
|    |         | kn          | part of the original team of program   |
|    |         | owledgeable | and customer representatives that      |
|    |         | evaluators  | prepared the statement of work and     |
|    |         | is both     | specification and participated in the  |
|    |         | cos         | bidder\'s conference. They were        |
|    |         | t-effective | already familiar with the requirements |
|    |         | and prudent | when it came time to evaluate          |
|    |         | in most     | proposals.                             |
|    |         | cases       |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Orals       | NISC-II used a smaller technical       |
|    |         | pr          | evaluation team (6 members working     |
|    |         | esentations | short durations) as compared to 10 or  |
|    |         | and page    | more evaluators on NISC-I over the     |
|    |         | limits on   | course of many months. Evaluations     |
|    |         | written     | were completed in about 5 weeks as     |
|    |         | material    | compared with 10 or more on NISC-I.    |
|    |         | force       |                                        |
|    |         | vendors and | Most technical evaluators were members |
|    |         | evaluators  | of the planning workgroup that wrote   |
|    |         | to focus    | the statement of work. They were       |
|    |         | their       | familiar with NISC and its purposes,   |
|    |         | efforts     | and were also the actual customers of  |
|    |         |             | the program. Since these participants  |
|    |         |             | were well along the learning curve,    |
|    |         |             | review of proposals was quick and      |
|    |         |             | efficient.                             |
|    |         |             |                                        |
|    |         |             | Oral presentations were videotaped for |
|    |         |             | a major part of the work effort and    |
|    |         |             | supplementary written technical detail |
|    |         |             | was limited to 60 pages per section.   |
|    |         |             | This concentrated the evaluation time  |
|    |         |             | and shortened the whole process        |
|    |         |             | without any apparent loss in quality.  |
|    |         |             |                                        |
|    |         |             | We also conducted a bidders\'          |
|    |         |             | conference and numerous internet       |
|    |         |             | updates/exchanges before releasing the |
|    |         |             | RFO. This enabled us to tell industry  |
|    |         |             | what we were planning, solicit input,  |
|    |         |             | answer questions, and refine the RFO   |
|    |         |             | before release.                        |
+----+---------+-------------+----------------------------------------+
| ** | **Oral  | Some tips   | Having an oral presentation is         |
| 23 | p       | for         | absolutely essential for services      |
| ** | resenta | achieving   | contracts. Require the program manager |
|    | tions** | effective   | to make the presentation. Avoid        |
|    |         | oral        | getting a PR person. Use the           |
|    |         | pr          | opportunity to hear from your program  |
|    |         | esentations | manager and all key personnel. They    |
|    |         |             | are the ones to manage the project and |
|    |         |             | if they can not cover it adequately,   |
|    |         |             | no one else can.                       |
|    |         |             |                                        |
|    |         |             | Make audio recordings. They help if an |
|    |         |             | evaluator misses part of the           |
|    |         |             | presentation. It is also useful for    |
|    |         |             | reviewing and verifying what was said  |
|    |         |             | if there is any doubt about the points |
|    |         |             | covered or how they were addressed.    |
|    |         |             |                                        |
|    |         |             | Ensure any audio-visual equipment you  |
|    |         |             | plan to use is available and in        |
|    |         |             | working order, such as an audio        |
|    |         |             | recorder, cassette tapes, timer, and   |
|    |         |             | overhead projector.                    |
|    |         |             |                                        |
|    |         |             | Allow sufficient time between oral     |
|    |         |             | presentations to record your           |
|    |         |             | evaluation scores. If you wait and     |
|    |         |             | hear more than one presentation before |
|    |         |             | doing your evaluation, you may not     |
|    |         |             | remember exactly what was said or      |
|    |         |             | which contractor said it. Take notes.  |
|    |         |             | This will help when doing your         |
|    |         |             | evaluations. Also, one evaluator may   |
|    |         |             | note a point that was covered which    |
|    |         |             | another evaluator may have missed.     |
+----+---------+-------------+----------------------------------------+
|    |         | Oral        | Oral presentations help the evaluation |
|    |         | pr          | team and SSO observe key personnel as  |
|    |         | esentations | they present their capability and plan |
|    |         | are a plus, | of attack. These sessions were video   |
|    |         | but some    | taped which provided the SSO an        |
|    |         | precautions | opportunity to participate in the      |
|    |         | are         | presentation process. The tapes also   |
|    |         | important   | provided documentation for the team's  |
|    |         |             | evaluation and recommendations.        |
|    |         |             |                                        |
|    |         |             | Vendors were required to conduct their |
|    |         |             | own taping sessions for the oral       |
|    |         |             | presentations. Some became very        |
|    |         |             | elaborate. Once the SIR has been       |
|    |         |             | issued, oral presentations should be   |
|    |         |             | conducted within 3-6 weeks. Longer     |
|    |         |             | time causes contractors to keep        |
|    |         |             | practicing, refining, and enhancing.   |
|    |         |             | This ultimately increases their costs. |
|    |         |             | We recommend you specify limitations   |
|    |         |             | on taping sessions. With no time       |
|    |         |             | limit, the presentations just get more |
|    |         |             | and more elaborate with little derived |
|    |         |             | benefit                                |
+----+---------+-------------+----------------------------------------+
|    |         | Use the     | Oral proposals avoided the             |
|    |         | oral        | presentation of a huge amount of       |
|    |         | proposal    | written material from vendors and we   |
|    |         | technique;  | did not have to \"wade\" through the   |
|    |         | it is very  | submissions. The oral proposal         |
|    |         | clear and   | technique was very effective in        |
|    |         | concise,    | two-way communication and quickly      |
|    |         | and it lets | identified issues for clarification.   |
|    |         | you see     | The process allowed a rapid technical  |
|    |         | first hand  | evaluation from notes taken at the     |
|    |         | how vendors | presentations.                         |
|    |         | respond to  |                                        |
|    |         | technical   |                                        |
|    |         | evaluation  |                                        |
|    |         | factors     |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Oper  | Conduct the |                                        |
| 24 | ational | OCD early   |                                        |
| ** | Cap     | during      |                                        |
|    | ability | source      |                                        |
|    | d       | selection   |                                        |
|    | emonstr | to          |                                        |
|    | ation** | determine   |                                        |
|    |         | who has     |                                        |
|    |         | viable      |                                        |
|    |         | solutions   |                                        |
|    |         |             |                                        |
|    |         | Make sure   |                                        |
|    |         | vendors     |                                        |
|    |         | know what   |                                        |
|    |         | you're      |                                        |
|    |         | looking for |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | An          | Our original plan was to pay two       |
|    |         | operational | contractors \$2.5 million each to test |
|    |         | capability  | their systems in an operational        |
|    |         | test        | capability test. Our original plan     |
|    |         | running in  | also envisioned a 3-phase procurement: |
|    |         | parallel    |                                        |
|    |         | with the    | 1\) Issue a SIR for down selection;    |
|    |         | RFO can     |                                        |
|    |         | save time   | 2\) Conduct an OCT; and,               |
|    |         | and prove   |                                        |
|    |         | suitability | 3\) Issue a RFO.                       |
|    |         |             |                                        |
|    |         |             | During our market survey, we convinced |
|    |         |             | potential offerors to perform the      |
|    |         |             | operational capability test without    |
|    |         |             | payment. They agreed to run the OCT in |
|    |         |             | parallel with the RFO, which gave the  |
|    |         |             | team high confidence a low-price       |
|    |         |             | system could meet our requirements.    |
|    |         |             | Running it in parallel with the RFO as |
|    |         |             | an element of source selection saved   |
|    |         |             | an estimated \$5 million and shortened |
|    |         |             | award schedule by 4 months.            |
+----+---------+-------------+----------------------------------------+
| ** | *       | AMS allows  | This substantially reduced the number  |
| 25 | *Source | competition | of proposals and evaluation time and   |
| ** | sele    | to be       | resources. It also reduced industry's  |
|    | ction** | sought      | bid and proposal costs since companies |
|    |         | among the   | with little or no chance for award did |
|    |         | most        | not make the down selection.           |
|    |         | qualified   |                                        |
|    |         | vendors     |                                        |
|    |         | rather than |                                        |
|    |         | full and    |                                        |
|    |         | open as     |                                        |
|    |         | required    |                                        |
|    |         | under the   |                                        |
|    |         | FAR         |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | We were     | 1\) SIR down selection                 |
|    |         | able to     |                                        |
|    |         | complete a  | 2\) Small teams of knowledgeable       |
|    |         | complex     | technical and business evaluators      |
|    |         | source      |                                        |
|    |         | section in  | 3\) Standard template to write the     |
|    |         | 4 ½ months  | evaluation report                      |
|    |         | using a few |                                        |
|    |         | simple      | 4\) Adjectival ratings verses point    |
|    |         | techniques  | scoring                                |
|    |         |             |                                        |
|    |         |             | 5\) Open discussions                   |
|    |         |             |                                        |
|    |         |             | 6\) Tailored requirements              |
|    |         |             |                                        |
|    |         |             | 7\) Elimination of best and final      |
|    |         |             | offers                                 |
+----+---------+-------------+----------------------------------------+
|    |         | Don't go to | We extended the time for submission of |
|    |         | extremes to | proposals sections for a vendor who    |
|    |         | keep a      | was late and who made errors           |
|    |         | vendor in   | (including Section B). The team had to |
|    |         | the         | guide vendor in completing Section B   |
|    |         | competition | and in determining how its products    |
|    |         |             | fit FAA requirements. In the end,      |
|    |         |             | vendor products and prices were        |
|    |         |             | attractive, but it was not responsive  |
|    |         |             | to FAA requirements and schedule.      |
+----+---------+-------------+----------------------------------------+
|    |         | Don\'t      | Our first selected vendor never        |
|    |         | notify a    | returned the signed model contract as  |
|    |         | vendor it   | required by the letter requesting a    |
|    |         | has been    | \"best and final\" price. In effect,   |
|    |         | selected    | the vendor had never agreed to FAA     |
|    |         | for award   | terms and conditions. This was         |
|    |         | until you   | revealed in the following 4 days of    |
|    |         | have the    | negotiations, until the vendor was     |
|    |         | required    | finally eliminated as non-responsive   |
|    |         | paperwork   | to FAA contract terms and conditions.  |
|    |         | signed      |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Cost        | Our initial SIR went out without       |
|    |         | estimates   | definitive size/scope. As a result,    |
|    |         | should be   | proposals far exceeded our resources.  |
|    |         | published   |                                        |
|    |         | during      |                                        |
|    |         | planning    |                                        |
|    |         | stages to   |                                        |
|    |         | help        |                                        |
|    |         | vendors     |                                        |
|    |         | size the    |                                        |
|    |         | anticipated |                                        |
|    |         | work        |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Use         |                                        |
|    |         | auctioning  |                                        |
|    |         | techniques  |                                        |
|    |         | to obtain a |                                        |
|    |         | lower quote |                                        |
|    |         | from        |                                        |
|    |         | vendors     |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Debri | C           |                                        |
| 26 | efing** | onsiderable |                                        |
| ** |         | time and    |                                        |
|    |         | resource    |                                        |
|    |         | can be      |                                        |
|    |         | saved using |                                        |
|    |         | telec       |                                        |
|    |         | onferencing |                                        |
|    |         | to debrief  |                                        |
|    |         | losing      |                                        |
|    |         | vendors     |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Pro   | Be wary and | Some of what we said was during the    |
| 27 | tests** | get written | debriefing of a vendor was twisted and |
| ** |         | questions   | used against the government during a   |
|    |         | in advance  | protest. Since a Judge will determine  |
|    |         | when        | whether the government made rational   |
|    |         | debriefings | decisions based on testimony and       |
|    |         | vendors to  | written documentation from the         |
|    |         | avoid       | evaluation, evaluators should          |
|    |         | having your | completely document the rationale for  |
|    |         | statements  | their positions even during            |
|    |         | used        | preliminary phases of the evaluation.  |
|    |         | against you |                                        |
|    |         | in a        |                                        |
|    |         | protest     |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Mark        | Ensure sensitive documents are         |
|    |         | sensitive   | adequately marked when they are        |
|    |         | documents   | prepared. During a protest, the        |
|    |         | adequately  | contracting officer submitted the      |
|    |         | to ensure   | documents prepared by the service team |
|    |         | proper use  | to the Disputes Resolution Officer     |
|    |         | during      | (DRO) as requested. Among those        |
|    |         | disputes    | documents was the Award Decision       |
|    |         |             | Document (ADD) with the initial and    |
|    |         |             | final cost/price evaluation reports    |
|    |         |             | attached. The initial cost/price       |
|    |         |             | evaluation report included sensitive   |
|    |         |             | and inflammatory information from the  |
|    |         |             | pricing evaluation team regarding the  |
|    |         |             | successful offeror\'s financial        |
|    |         |             | posture. The cost/price evaluation     |
|    |         |             | reports were intended to be used as    |
|    |         |             | internal government memos only. The    |
|    |         |             | FAA\'s DRO provided this sensitive     |
|    |         |             | information to the protestor who in    |
|    |         |             | turn used it to enhance and supplement |
|    |         |             | the protest. The protest was prolonged |
|    |         |             | by the release of sensitive,           |
|    |         |             | confidential, and proprietary          |
|    |         |             | documentation without proper           |
|    |         |             | protection.                            |
+----+---------+-------------+----------------------------------------+
|    |         | Some useful | 1\) Treat all competitors fairly       |
|    |         | best        |                                        |
|    |         | practices   | 2\) Establish a source selection       |
|    |         | for         | process before proposals are received  |
|    |         | avoiding a  | and follow it                          |
|    |         | protest     |                                        |
|    |         |             | 3\) Conduct a thorough evaluation and  |
|    |         |             | be reasonable in your conclusion       |
|    |         |             |                                        |
|    |         |             | 4\) Document the rationale basis for   |
|    |         |             | your decision thoroughly               |
+----+---------+-------------+----------------------------------------+
| ** | **C     | Using a     |                                        |
| 28 | ontract | spreadsheet |                                        |
| ** | ad      | (Excel) to  |                                        |
|    | ministr | track       |                                        |
|    | ation** | invoices is |                                        |
|    |         | a great way |                                        |
|    |         | of          |                                        |
|    |         | maintaining |                                        |
|    |         | the status  |                                        |
|    |         | of contract |                                        |
|    |         | costs and   |                                        |
|    |         | funding     |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Open        |                                        |
|    |         | exchange of |                                        |
|    |         | information |                                        |
|    |         | minimizes   |                                        |
|    |         | change      |                                        |
|    |         | orders      |                                        |
|    |         | during      |                                        |
|    |         | performance |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **      | Use open    |                                        |
| 29 | Constru | com         |                                        |
| ** | ction** | munications |                                        |
|    |         | to discuss  |                                        |
|    |         | small       |                                        |
|    |         | c           |                                        |
|    |         | onstruction |                                        |
|    |         | job         |                                        |
|    |         | r           |                                        |
|    |         | equirements |                                        |
|    |         | and price   |                                        |
|    |         | with        |                                        |
|    |         | contractors |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Consider    |                                        |
|    |         | using       |                                        |
|    |         | government  |                                        |
|    |         | credit      |                                        |
|    |         | cards to    |                                        |
|    |         | pay for     |                                        |
|    |         | c           |                                        |
|    |         | onstruction |                                        |
|    |         | jobs under  |                                        |
|    |         | \$10,000    |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Some good   | When using a list of pre-qualified     |
|    |         | ideas that  | vendors each with multiple             |
|    |         | reduce the  | subcontractors on a construction       |
|    |         | cost of     | contract where there are a large       |
|    |         | d           | number of drawings (over 300 in our    |
|    |         | istributing | case), we recommend the procurement    |
|    |         | the         | office provide one or two copies to    |
|    |         | s           | each qualified vendor without charge   |
|    |         | olicitation | and offer, for the price of            |
|    |         | package and | reproduction, the solicitation to any  |
|    |         | amendments  | interested sub. General contractors    |
|    |         | on complex  | could purchase additional sets as      |
|    |         | c           | well.                                  |
|    |         | onstruction |                                        |
|    |         | p           | A plan holders list should be kept by  |
|    |         | rocurements | the procurement branch and amendments  |
|    |         |             | to the solicitation distributed to     |
|    |         |             | everyone on the list. It seems central |
|    |         |             | distribution is an unavoidable         |
|    |         |             | procedure, although costly.            |
|    |         |             |                                        |
|    |         |             | The use of a printing company to make  |
|    |         |             | distribution is another option, but    |
|    |         |             | this creates the problem of how to     |
|    |         |             | ensure amendments reach vendors.       |
|    |         |             |                                        |
|    |         |             | We recommend announcing amendments on  |
|    |         |             | the Internet.                          |
+----+---------+-------------+----------------------------------------+
|    |         | Some        | 1\) An incentive provision in the      |
|    |         | creative    | contract for early completion, in      |
|    |         | ways to     | effect creates a Fixed Price Incentive |
|    |         | save money  | Contract                               |
|    |         | on          |                                        |
|    |         | c           | 2\) Use of "best and final offer"      |
|    |         | onstruction | saved \$248,000 from an low initial    |
|    |         | contracts   | offer                                  |
|    |         |             |                                        |
|    |         |             | 3\) Extensive use of the Internet when |
|    |         |             | announcing the solicitation and        |
|    |         |             | posting the SIR saved a lot of time in |
|    |         |             | the contracting cycle                  |
+----+---------+-------------+----------------------------------------+
|    |         | Rather than | In our construction solicitation, we   |
|    |         | reviewing   | asked each general contractor to       |
|    |         | su          | submit qualifications on its proposed  |
|    |         | bcontractor | electrical and mechanical              |
|    |         | proposals,  | subcontractors. If they did not know   |
|    |         | hold the    | which subs they were going to use,     |
|    |         | general     | they were to submit at least two       |
|    |         | contractor  | electrical and mechanical              |
|    |         | responsible | subcontractor proposals. This resulted |
|    |         |             | in more time spent reviewing subs than |
|    |         |             | general contractors. Furthermore, we   |
|    |         |             | found they often could not provide     |
|    |         |             | details regarding past performance.    |
|    |         |             | Thus, we found that evaluating         |
|    |         |             | subcontractors added considerable time |
|    |         |             | to the process without much benefit.   |
+----+---------+-------------+----------------------------------------+
|    |         | Requiring a | One factor contributing to significant |
|    |         | pre-s       | time saved from solicitation to award  |
|    |         | olicitation | was the requirement that no one would  |
|    |         | site visit  | receive the solicitation unless        |
|    |         | by all      | present at a pre-solicitation          |
|    |         | prospective | site-visit where the solicitation      |
|    |         | vendors     | packages was distributed. The site     |
|    |         | offers real | visit also served as a pre-bid         |
|    |         | benefit     | conference. It afforded interested     |
|    |         |             | offerors an opportunity to view the    |
|    |         |             | site and to learn from the agency what |
|    |         |             | was expected. Agency personnel were    |
|    |         |             | able to size up the vendors. Also, the |
|    |         |             | requirement to be at the site to       |
|    |         |             | receive the solicitation eliminated    |
|    |         |             | the need to mail the solicitation      |
|    |         |             | package to a great number of firms. As |
|    |         |             | it developed, only a relatively few    |
|    |         |             | firms showed enough interest to attend |
|    |         |             | the site visit. There was no mailing   |
|    |         |             | of amendments to specs and drawings    |
|    |         |             | except to those who attended the       |
|    |         |             | pre-bid site visit.                    |
+----+---------+-------------+----------------------------------------+
|    |         | Limiting    |                                        |
|    |         | the number  |                                        |
|    |         | of pages    |                                        |
|    |         | and font    |                                        |
|    |         | size in bid |                                        |
|    |         | packages    |                                        |
|    |         | encourages  |                                        |
|    |         | vendors to  |                                        |
|    |         | stress      |                                        |
|    |         | quality     |                                        |
|    |         | over        |                                        |
|    |         | quantity    |                                        |
|    |         | and it      |                                        |
|    |         | lowers      |                                        |
|    |         | s           |                                        |
|    |         | olicitation |                                        |
|    |         | costs for   |                                        |
|    |         | them and    |                                        |
|    |         | the         |                                        |
|    |         | government  |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Real  | Define your | Real property and facilities can only  |
| 30 | p       | real        | be procured when ratified in an        |
| ** | roperty | property /  | approved program requirements          |
|    | and     | facility    | document.                              |
|    | facil   | (RP/FAC)    |                                        |
|    | ities** | requirement |                                        |
|    |         | in the      |                                        |
|    |         | program     |                                        |
|    |         | r           |                                        |
|    |         | equirements |                                        |
|    |         | document    |                                        |
|    |         | before      |                                        |
|    |         | coming to   |                                        |
|    |         | the         |                                        |
|    |         | h           |                                        |
|    |         | eadquarters |                                        |
|    |         | RP/FAC      |                                        |
|    |         | o           |                                        |
|    |         | rganization |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Involve the | Acquiring real property and building   |
|    |         | RP/FAC      | facilities is a very lengthy process   |
|    |         | o           | owing to environmental laws and        |
|    |         | rganization | regulations. So involve the RP/FAC     |
|    |         | from the    | organization during final investment   |
|    |         | very        | analysis as the requirements are first |
|    |         | earliest    | defined.                               |
|    |         | stages of   |                                        |
|    |         | planning    |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Follow-up   | This eliminates the prospect of        |
|    |         | all verbal  | miscommunication. The written          |
|    |         | com         | follow-up can be an e-mail, letter, or |
|    |         | munications | memo.                                  |
|    |         | between the |                                        |
|    |         | client and  |                                        |
|    |         | RP/FAC      |                                        |
|    |         | o           |                                        |
|    |         | rganization |                                        |
|    |         | with        |                                        |
|    |         | written     |                                        |
|    |         | statements  |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Establish a | Retain this team throughout the        |
|    |         | 2-person    | procurement process.                   |
|    |         | team, one   |                                        |
|    |         | from the    |                                        |
|    |         | program     |                                        |
|    |         | office and  |                                        |
|    |         | one from    |                                        |
|    |         | the RP/FAC  |                                        |
|    |         | or          |                                        |
|    |         | ganization, |                                        |
|    |         | to satisfy  |                                        |
|    |         | client      |                                        |
|    |         | r           |                                        |
|    |         | equirements |                                        |
+----+---------+-------------+----------------------------------------+
|    |         | Use         | The contracts are already in place.    |
|    |         | qualified   | Simply define and fund your            |
|    |         | vendor      | requirements and seek competitive bids |
|    |         | lists to    | using qualified vendors.               |
|    |         | speed to    |                                        |
|    |         | real        |                                        |
|    |         | property /  |                                        |
|    |         | facility    |                                        |
|    |         | development |                                        |
|    |         | and         |                                        |
|    |         | imp         |                                        |
|    |         | lementation |                                        |
|    |         | processes   |                                        |
+----+---------+-------------+----------------------------------------+
| ** | **Human | Some        | Clearly define human performance in    |
| 31 | fa      | valuable    | the program requirements document.     |
| ** | ctors** | tips for    |                                        |
|    |         | optimizing  | Changing or adding human performance   |
|    |         | human       | requirements downstream almost always  |
|    |         | performance | adversely impacts cost and schedule.   |
|    |         | during      |                                        |
|    |         | product     | A COTS or NDI procurement does not     |
|    |         | development | eliminate the need to define human     |
|    |         |             | requirements.                          |
|    |         |             |                                        |
|    |         |             | Never suggest human interface design   |
|    |         |             | changes or modifications outside       |
|    |         |             | official contractual channels. Changes |
|    |         |             | may have legal or programmatic         |
|    |         |             | ramifications when dealing with        |
|    |         |             | contractors; others may impact union   |
|    |         |             | bargaining options; and all have       |
|    |         |             | human-system performance implications. |
|    |         |             |                                        |
|    |         |             | System performance criteria should     |
|    |         |             | consider the human component to the    |
|    |         |             | degree necessary to enable data        |
|    |         |             | collection and testing of human-system |
|    |         |             | performance.                           |
|    |         |             |                                        |
|    |         |             | While human performance constraints    |
|    |         |             | may not be the limiting factor in      |
|    |         |             | achieving desired product performance, |
|    |         |             | they will almost always impose         |
|    |         |             | constraints on program cost and        |
|    |         |             | schedule.                              |
|    |         |             |                                        |
|    |         |             | System design and development can      |
|    |         |             | easily lose sight of human-system      |
|    |         |             | performance impacts on the end user    |
|    |         |             | (operator, maintainer, and support     |
|    |         |             | personnel) especially those related to |
|    |         |             | cognitive tasks. These elements must   |
|    |         |             | be investigated during detailed design |
|    |         |             | and prototyping.                       |
|    |         |             |                                        |
|    |         |             | Human factors risk and performance     |
|    |         |             | analysis needs to occur during concept |
|    |         |             | and requirements definition and        |
|    |         |             | investment analysis to help estimate   |
|    |         |             | resource expenditures associated with  |
|    |         |             | human performance aspects of the       |
|    |         |             | product.                               |
|    |         |             |                                        |
|    |         |             | Human factors roles and                |
|    |         |             | responsibilities must be clearly       |
|    |         |             | defined for those individuals and      |
|    |         |             | organizations supporting product       |
|    |         |             | acquisition. Lack of clear designated  |
|    |         |             | responsibility often leads to          |
|    |         |             | expensive ambiguity about who is to do |
|    |         |             | what and how much human factors is     |
|    |         |             | enough                                 |
|    |         |             |                                        |
|    |         |             | Human factors efforts must be          |
|    |         |             | integrated with other program          |
|    |         |             | activities and not conducted as a      |
|    |         |             | separate design activity or            |
|    |         |             | end-of-development assessment.         |
|    |         |             |                                        |
|    |         |             | Human factors specialists must provide |
|    |         |             | realistic cost and schedule estimates  |
|    |         |             | for human factors activities for       |
|    |         |             | inclusion in the implementation        |
|    |         |             | strategy and planning document. Human  |
|    |         |             | factors specialists need to be fully   |
|    |         |             | acquainted with the program in order   |
|    |         |             | to provide this input.                 |
+----+---------+-------------+----------------------------------------+