INTERIM REPORT ON AWARD-FEE CRITERIA 

FOR THE SYSTEM ENGINEERING AND 

TECHNICAL ASSISTANCE II CONTRACT 

Federal Aviation Administration 

Report Number: FI-2009-002 

Date Issued: October 7, 2008 
U.S. Department of 

Memorandum

Transportation 

Office of the Secretary 

of Transportation 

Office of Inspector General 

Subject: ACTION: Interim Report on Award-Fee 

Criteria for the System Engineering and 

Technical Assistance II Contract 

Federal Aviation Administration 

Report Number: FI-2009-002 

 

Date: October 7, 2008 

From: Mark H. Zabarsky 

Assistant Inspector General for 

 Acquisition and Procurement Audits 

Reply to 

Attn. of: JA-60 

To: Vice President of Acquisition and Business Services 

 Federal Aviation Administration 

As part of our ongoing audit of the Use of Cost-Plus-Award-Fee (CPAF) contracts 

within the Department, we are issuing this interim report regarding the award of 

the Federal Aviation Administration’s (FAA) System Engineering and Technical 

Assistance (SETA II) contract to support the National Airspace System (NAS) 

Architecture and Capital Investment Plan. The objectives of our audit were to 

determine whether: (1) award-fee plans established adequate criteria for evaluating 

contractor performance, and (2) the amount of award fees paid to contractors was 

adequately supported. 

On August 16, 2000, FAA awarded a 10-year CPAF contract for SETA II to BAE 

Systems Applied Technologies1

 for approximately $455 million. The SETA II 

contract is broad in scope and covers facilities and equipment; operations and 

maintenance; and research, engineering, and development requirements. BAE 

Systems provides assistance to a wide range of FAA organizations, such as the 

Office of Research and Acquisitions and other FAA lines of business. FAA 

established an award-fee pool totaling approximately $36 million. The contractor 

is currently in the sixteenth award-fee performance period, which ended on 

September 30, 2008.2

 Approximately $8 million remains available for the 

seventeenth and subsequent performance periods. The SETA II contract states 

 

1

 BAE Systems was formed in November 1999 when British Aerospace merged with Marconi Electronics Systems. 2

 A performance period is a 6-month timeframe. 
2

that FAA may unilaterally change the performance evaluation plan provided the 

contractor receives notice of the changes, in writing, within the first 30 days after 

commencement of the award-fee evaluation period. 

We performed this performance audit in accordance with Generally Accepted 

Government Auditing Standards as prescribed by the Comptroller General of the 

United States, and included such tests as we considered necessary to detect fraud, 

waste, and abuse. Exhibit A describes the scope of our audit and the methodology 

we used to achieve our objectives. The information in this interim report will be 

included in a later report addressing Department-wide CPAF contracting issues. 

FINDINGS 

We found that the evaluation criteria in the performance evaluation plan3 for the 

SETA II contract did not include measurable criteria needed to adequately 

evaluate contractor performance. Also, descriptions defining scores to rate 

contractor performance were vague and performance evaluators4

 were not 

required to document the rationale for the performance ratings. Evaluation criteria 

without clearly defined metrics, and vague scoring descriptions, could result in 

inflated contractor performance evaluations and, consequently, inappropriately 

approved award fees. 

 

Additionally, FAA no longer requires its performance evaluators to provide 

succinct narrative comments, including the identification of specific strengths, 

weaknesses, and deficiencies, when rating a contractor. Removing such a 

requirement allows for and relies on unsupported personal opinions and individual 

judgments, and raises questions as to what was actually the basis for judging 

contractor performance and awarding fees. 

Further, we found that contracting officials did not justify the cost-effectiveness of 

selecting a CPAF-type contract by evaluating administrative costs versus expected 

benefits to the Government. Without this evaluation, FAA had no assurance that a 

CPAF-type contract was appropriate. The award-fee guidelines used by FAA5

 and 

other Federal agencies require costs and benefits consideration prior to choosing a 

CPAF contract. 

3

 The performance evaluation plan is the basis for determining the amount of award fee and includes the award-fee 

criteria to be considered under each area evaluated; the percentage of award fee, if any, available for each area; and 

the frequency of evaluation periods. 4

 Performance evaluators are FAA staff who track and assess contractor performance daily. 

5

 FAA included this best practice in its Award-Fee Contracting Guide, issued September 2007; however, it was 7 years 

after the contract was awarded. 
3

Award-Fee Plan Criteria Were Too General 

The performance evaluation plan did not include clear and measurable award-fee 

criteria needed to adequately evaluate contractor performance. For example, the 

Technical Achievement factor used undefined terms such as “the quality and 

accuracy of deliverables” or “the degree of flexibility” as the standards of review. 

The Task Order Management factor used general terms such as “understands task 

order requirements and deliverables.” Additionally, the Contract Management 

factor used general terms such as “submits documents requiring FAA review and 

approval in a timely manner.” 

FAA’s guidance states that outcome-based criteria are the least administratively 

burdensome type of performance evaluation criteria, and should provide the best 

indicator of overall success. Guidance from other Federal Government agencies, 

such as the National Aeronautics and Space Administration (NASA) and the 

Departments of the Air Force, Army, and Navy, also prefer using measurable 

criteria to evaluate contractor performance. Additionally, guidance from these 

agencies, including FAA, states that using evaluation criteria that are too broad 

can result in evaluators not being able to provide meaningful comments to support 

ratings. 

Establishing specific criteria based on performance objectives would reduce the 

risk of unwarranted or subjective performance evaluations and ratings. Evaluation 

criteria must emphasize the most important aspects of the program to motivate the 

contractor to improve performance for each performance period. Performance 

monitors cannot provide meaningful comments and evaluations using vague, 

general, or too broadly defined criteria. 

The effect of having evaluation criteria without establishing clearly defined 

metrics for assessing performance could result in inflated evaluations and, 

consequently, inappropriately approved award fees. For the first fifteen award-fee 

periods of the SETA II contract, FAA paid the contractor approximately $13 

million (90 percent of the available award-fee pool) without assurances of whether 

the acquisition outcomes fell short of, met, or exceeded expectations. Without 

indicating areas of emphasis or desired outcomes, FAA does not have assurances 

that contract objectives are being met, nor does the contractor have motivation to 

perform the best possible job in those areas deemed critical. 

Revising the SETA II performance evaluation plan to include measurable awardfee criteria will put approximately $8 million in expected award fees for the 

SETA II contract to better use by ensuring FAA’s contract objectives are being 

met, and motivating the contractor to perform the best job in those areas deemed 

critical by the FAA. 
4

Evaluator Scoring Descriptions Were Too Vague for Rating 

Performance 

The chart in the performance evaluation plan (Exhibit B) used to compute the 

amount of award fee contains a payment structure that associates a range of award 

fees (between 0 and 100 percent) with four adjectival ratings—Excellent, Very 

Good, Satisfactory, and Poor. The descriptions defining the adjectival ratings, 

however, did not clearly define the basis for assigning such a rating. For example, 

a rating of Excellent is defined as: 

“Contractor's performance significantly exceeds the general level of 

achievement of a qualified Contractor in this field with all objectives 

exceeded.” 

Terms such as “significantly exceeds” and “general level of achievement of a 

qualified Contractor in this field” are undefined. FAA must clearly describe its 

adjectival ratings so there will be a basis for performance monitors to use in 

assessing contractor performance. 

Performance Evaluation Plan No Longer Requires Evaluators to 

Justify Scores 

FAA no longer requires performance evaluators to provide succinct narrative 

comments, including the identification of specific strengths, weaknesses, and 

deficiencies, when rating a contractor. Instead, FAA only encourages its 

evaluators to provide such comments. The SETA II Director of Systems 

Engineering said the removal of the documentation requirement was intended to 

increase the number of evaluation responses submitted by performance evaluators. 

The director said the response rate has since increased; we did not verify if the rate 

increased. 

Guidance from other Federal Government agencies such as NASA, and 

Departments of the Air Force, Army, and Navy, states that performance monitors 

should observe contractor performance based on the criteria specified in the 

performance evaluation plan, and document results by giving specific examples to 

support their conclusions. FAA consistently provided the contractor with high 

ratings and award fees—approximately $13 million, or 90 percent of the available 

award-fee pool. Without sufficiently documenting contractor evaluations, we 

could not determine if FAA properly awarded fees. Such a process allows for and 

relies on unsupported personal opinions and individual judgments. 
5

FAA Has No Assurance the Contract Type Is Appropriate for the 

SETA II Contract 

FAA contracting officials did not justify the cost effectiveness of selecting a 

CPAF-type contract. Performance evaluation on an award-fee contract requires 

greater effort and more resources than other types of contracts because oversight is 

required to monitor and document contractor performance. FAR, FAA, and 

award-fee guides used by other Federal agencies require agencies to consider costs 

and benefits before choosing a CPAF-type contract. For example, NASA 

guidance states that before a CPAF contract is selected, a contracting officer 

should perform a cost/benefit analysis of the expected benefits versus the added 

administrative costs. 

RECOMMENDATIONS 

We recommend that the Vice President of Acquisition and Business Services, 

FAA: 

1. Develop measurable award-fee criteria for assessing contractor performance 

and link the criteria to the acquisition outcomes for the work to be 

accomplished. 

2. Describe adjectival ratings clearly so there will be a basis for assessing 

performance. 

3. Require performance evaluators to provide succinct narrative comments, 

including the identification of specific strengths, weaknesses, and deficiencies 

to support assigned ratings. 

4. Require the SETA II contracting officer to reevaluate the contract type for 

future SETA procurement contracts and justify the use of an award-fee 

contract by performing a cost/benefit analysis. 

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL’S 

RESPONSE 

We discussed the problems cited in this report with FAA officials including the 

Acting Program Director for the Contract Operations and Oversight Group, the 

Manager of the Contracting Services Group, and the SETA II Program Manager 
6

on August 26, 2008. As a result, FAA officials agreed to take the following 

actions to address the problems cited in this report.6

 

• The SETA Team intends to revise its practices to more closely align with 

the goal of performance-based contracting. 

• The SETA Team will define contract goals and objectives that are 

performance-based. This includes requiring the SETA-II contractor to 

provide a plan to accomplish the goals and objectives. 

• The SETA Team will include performance metrics in all applicable task 

orders for incorporation into the award-fee process. 

• The Contracting Officer recommends the revision of the award fee plan for 

SETA II for all subsequent Award Fee Periods to more accurately identify 

measurable award fee criteria for determining contractor performance and 

link the criteria to the acquisition outcomes for the required work. 

As FAA management’s actions are ongoing at the time of this report, we cannot 

evaluate the efficiency of these actions; however, we believe the results of these 

actions will correct the problems identified in this report. Implementing the 

planned corrective actions will put approximately $8 million in expected award 

fees to better use by revising the performance evaluation plan and ensuring FAA’s 

acquisition objectives are being met. 

ACTIONS REQUIRED 

In accordance with Department of Transportation Order 8000.1C, we would 

appreciate receiving your written comments within 30 calendar days. If you 

concur with the recommendations, please indicate the specific action taken or 

planned and provide the target date for completion. If you do not concur with the 

findings or recommendations, please provide your rationale. You may provide 

alternative courses of action that you believe would resolve the issues presented in 

this report. Also, please comment whether you agree that the estimated $8 million 

remaining in award fees for the SETA II contract could be put to better use by 

revising the performance evaluation plan. 

We appreciate the courtesies and cooperation of FAA representatives during this 

audit. If you have any questions concerning this report, please call me at (202) 

366-5225 or Terrence Letko, Program Director, at (202) 366-9917. 

 

6

 FAA’s planned corrective actions were provided in a memorandum, September 23, 2008. 
7

# 

cc: Acting Federal Aviation Administrator 

 Senior Procurement Executive 

 Martin Gertel, M-1 

 Anthony Williams, ABU-100 
8

EXHIBIT A. SCOPE AND METHODOLOGY 

This audit is associated with our Department-wide Audit of the Use of Cost-PlusAward-Fee Contracts, Project Number 07F3011F000. We conducted this 

performance audit in accordance with Generally Accepted Government Auditing 

Standards and included such tests as we considered necessary to detect fraud, 

waste, and abuse. 

To determine whether the SETA II contract award-fee plans were adequately 

designed we reviewed: 

• FAA’s guidance for award-fee contracts; 

• Award-fee plans and criteria; 

• Statements of work and deliverables for the contract and selected task 

orders; 

• Two performance evaluation reports used to measure the contractor’s 

performance against award-fee criteria; and 

• Best Practices for award-fee contracts (Departments of the Air Force, 

Army, and Navy, and NASA Award-Fee Guides). 

We reviewed the acquisition plan for the contract to determine whether 

improvements could be made in the methods used for obtaining the SETA II 

contract. We also interviewed FAA acquisition and program officials regarding 

the performance evaluation plan and processes and discussed the results of our 

findings with senior FAA acquisition officials. 

Exhibit A. Scope and Methodology 
9

 EXHIBIT B. EVALUATOR SCORING CHART 

DESCRIPTORS DESCRIPTIONS

EXCELLENT 

Rating 90 – 100

Contractor’s performance significantly 

exceeds the general level of 

achievement of a qualified Contractor in 

this field with all objectives exceeded. 

VERY GOOD 

Rating 80 – 89

Contractor’s performance has achieved 

highly effective results that are fully 

responsive and compliant with all 

requirements contained within the 

contract and task orders. 

SATISFACTORY 

Rating 50 – 79

Contractor’s performance is fully 

responsive to SETA-II Contract and 

task order requirements; adequate 

results. 

POOR 

Rating 0 – 49

While Contractor’s performance may be 

responsive to SETA-II Contract and 

task order requirements, results are 

barely adequate. Performance that 

doesn’t meet expectations is noted with 

a clear negative impact on overall 

contract performance. 

 

 

 

 

Exhibit B. Evaluator Scoring Chart 
EXHIBIT C. Major Contributors to this Report 

10

EXHIBIT C. MAJOR CONTRIBUTORS TO THIS REPORT 

Name Title 

Terrence Letko Program Director 

Dormayne Dillard-Christian Project Manager 

Jelilat Ojodu Auditor 

Jean Diaz Writer/Editor 
11

The following page is a textual version of an exhibit found in this document that 

was not in the original document; it has been added here to accommodate assistive 

technology. 
12

Interim Report on Award-Fee Criteria for the System 

Engineering and Technical Assistance II Contract 

Section 508 Compliant Presentation 

Exhibit B. Evaluator Scoring Chart

A descriptor rating of excellent, 90 to 100 percent, is described as a contractor’s 

performance significantly exceeds the general level of achievement of a qualified 

Contractor in this field with all objectives exceeded. 

A descriptor rating of very good, 80 to 89 percent, is described as a contractor’s 

performance has achieved highly effective results that are fully responsive and 

compliant with all requirements contained within the contract and task orders. 

A descriptor rating of satisfactory, 50 to 79 percent, is described as a contractor’s 

performance is fully responsive to SETA II contract and task order requirements; 

adequate results. 

A descriptor rating of poor, 0 to 49 percent, is described as while a contractor’s 

performance may be responsive to SETA II contract and task order requirements, 

results are barely adequate. Performance that doesn’t meet expectations is noted 

with a clear negative impact on overall contract performance.