NASA Procurement: Use of Award Fees for Achieving Program Outcomes Should Be Improved
Published: 2007-01-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Cost-plus-award-fee contracts accounted for almost half of the National Aeronautic and Space Administration's (NASA) obligated contract dollars for fiscal years 2002-2004. Since 1990, we have identified NASA's contract management as a high-risk area--in part because of a lack of emphasis on end results. Congress asked us to examine (1) the extent NASA's guidance on award fees addresses problems previously identified with the use of award-fee contracts and (2) whether NASA follows its guidance in using award fees to achieve desired outcomes. We reviewed the top 10 dollar value award-fee contracts active from fiscal years 2002 through 2004.
section_title: What GAO Found
paragraphs: NASA guidance on the use of cost-plus-award-fee (CPAF) contracts provides criteria to improve the effectiveness of award fees. For example, the guidance emphasizes outcome factors that are good indicators of success in achieving desired results, cautions against using numerous evaluation factors, prohibits rollover of unearned fee, and encourages evaluating the costs and benefits of such contracts before using this contract type. However, NASA does notalways follow the preferred approach laid out in its guidance. For example, some evaluation criteria contained input or process factors, such as program planning and organizational management. Moreover, some contracts included numerous supporting subfactors that may dilute emphasis on any specific criteria. Although the Federal Acquisition Regulation and NASA guidance require considering the costs and benefits of choosing a CPAF contract, NASA did not perform such analyses. In some cases there appears to be a significant disconnect between program results and fees paid. For example, NASA paid the contractor for the Earth Observing System Data and Information System Core System 97 percent of the available award fee despite a delay in the completion of the contract by over 2 years and an increase in the cost of the contract of more than 50 percent. NASA officials expressed satisfaction with the results of the contracts we reviewed, and this was further evidenced by the extent of fee paid. NASA's satisfaction was based on its evaluations of contractor performance against criteria established in the award-fee plan. While NASA's evaluations would indicate generally good contractor performance, that performance did not always translate into desired program outcomes. That disconnect raises questions as to the extent NASA is achieving the effectiveness it sought through the establishment of guidance on the use of award fees. NASA has not evaluated the overall effectiveness of award fees and does not have metrics in place for conducting such evaluations.

section_title: Letter
paragraphs: The Federal Acquisition Regulation (FAR) states that cost-plus-award-fee  (CPAF) contracts are intended to motivate excellent contractor  performance in areas such as quality, timeliness, technical ingenuity, and  cost-effective management. During the early 1990s, the National  Aeronautics and Space Administration (NASA) Inspector General and  NASA internal studies raised concerns about NASA’s use of CPAF  contracts. As a result, NASA developed specific guidance to improve the  effectiveness of award fees. The CPAF contract type continues to be used  extensively by NASA for obtaining both goods and services, accounting for  almost half of NASA contract dollars for fiscal years 2002 through 2004.  Given this, you requested that we examine NASA’s use of award-fee  contracts and determine (1) the extent NASA’s guidance addresses the  problems previously identified with the use of award-fee contracts and (2)  whether NASA follows its guidance in using award fees to achieve desired  outcomes.
To address these objectives, we reviewed a sample of NASA CPAF  contracts that were active from fiscal years 2002 through 2004. We  reviewed contract files, obtained information from program and  contracting officials through the use of a structured questionnaire, and  discussed the application of award-fee criteria with NASA officials  involved in the award-fee process. Our work was conducted between  August 2005 and October 2006 in accordance with generally accepted  government auditing standards. For a complete description of our scope  and methodology, see appendix I.
subsections: section_title: Background
paragraphs: In January 2004, the President announced a new “Vision for Space  Exploration” calling for human and robotic missions to the Moon, Mars,  and beyond. Over the next two decades, NASA plans to spend over $100  billion to develop a number of new capabilities, supporting technologies,  and facilities that are critical to enabling space exploration missions.  Development of the critical capabilities and technologies will be largely  dependent on NASA contractors, who constitute more than two-thirds of  NASA’s workforce. According to NASA officials, 87 percent of NASA’s  $16.6 billion budget for fiscal year 2006 was spent on work performed by  its contractors.
Since 1990, we have designated NASA’s contract management as a high- risk area. This is based primarily on NASA’s lack of a modern integrated  financial management system that can provide reliable information on  contract spending and performance as well as NASA’s lack of emphasis on  end results, product performance, and cost control. For example, our most  recent high-risk report stated that while NASA has taken actions to  improve its contract management function, it continues to face  considerable challenges in implementing its contracts effectively.
NASA is organized under four mission directorates—Aeronautics  Research, Exploration, Science, and Space Operations—each of which  covers a major area of the agency’s research and development efforts. The  agency is composed of NASA headquarters, 10 field centers, and the  contractor-operated Jet Propulsion Laboratory.
subsections: section_title: Cost-Plus-Award-Fee Contracts
paragraphs: NASA and other federal agencies can choose among numerous contract  types for acquiring goods and services that can differ in part according to  the nature of the fee that agencies offer to the contractor for achieving or  exceeding specified objectives or goals. According to the FAR, a CPAF  contract is appropriate to use when it is difficult to measure key elements  of performance. It is widely used to procure nonroutine services such as  the development of new systems. Typically, award-fee contracts  emphasize several aspects of contractor performance, such as schedule  performance, technical performance, and cost control. Because  development and administration of award-fee contracts involve  substantially more effort over the life of a contract than other types of  contracts, the FAR and NASA’s Award Fee Contracting Guide specify that  the expected benefits of using an award-fee contract must exceed the  additional administrative effort and cost involved.
The theory behind CPAF contracts is that although the government  assumes most of the cost risk, it retains control over most or all of the  contractor’s potential profit as leverage. On CPAF contracts, the award fee  is often the only source of potential fee for the contractor. According to  the NASA FAR Supplement and NASA’s Award Fee Contracting Guide,  these contracts can include a base fee of anywhere from 0 to 3 percent of  the estimated value of a nonservice contract. However, NASA’s regulations  and guide do not allow the use of a base fee on service contracts. Table 1  shows the percentage of award fee available on the contracts we  examined. (See app. II for a description of these contracts.)
section_title: NASA’s Use of Cost-Plus- Award-Fee Contracts
paragraphs: NASA relies heavily on CPAF contracts. This contract type accounted for  48 percent of obligated contract dollars and 7.7 percent of contract actions  from fiscal years 2002 through 2004. By comparison, between fiscal years  1999 and 2003, award-fee contracts accounted for 13 percent of the  contract dollars and 3.4 percent of contract actions at the Department of  Defense (DOD). A CPAF contract includes an estimate of the total cost of  what is being contracted for, may include a fee with a possible base  amount fixed at the inception of the contract, and includes an award  amount that is intended to motivate excellence in contract performance.  The award fee is paid based upon the government’s periodic judgmental  evaluations of contractor performance.
section_title: Award-Fee Criteria
paragraphs: When developing evaluation plans, NASA’s award-fee guide indicates that  evaluation plans may include outcomes, outputs, inputs, or a combination  of these elements. NASA’s guide expresses a preference for outcome  factors. It notes that while it is sometimes valuable to consider input and  output factors when evaluating contractor performance, outcome factors  are better indicators of success relative to the desired result.
An outcome factor is an assessment of the results of an activity  compared to its intended purpose. Outcome-based factors are the least  administratively burdensome type of performance evaluation factor,  and should provide the best indicator of overall success. Outcome- based factors should therefore be the first type of evaluation factor  considered for use, and are often ideal for nonroutine efforts.
An output factor is the tabulation, calculation, or recording of activity  or effort and can be expressed in a quantitative or qualitative manner.  Output factors may be more desirable for routine efforts, but are  administratively more burdensome than outcome factors due to the  tabulation, calculation, or recording requirements. When output factors  are used, care should be taken to ensure that there is a logical  connection between the reported measures and the program’s mission,  goals, and objectives.
Input factors refer to intermediate processes, procedures, actions, or  techniques that are key elements influencing successful contract  performance. These may include testing and other engineering  processes and techniques; quality assurance and maintenance  procedures; subcontracting plans; purchasing department  management; and inventory, work assignment, and budgetary controls.
section_title: Evaluation of Award-Fee Contracts
paragraphs: For CPAF contracts, NASA personnel conduct periodic, typically  semiannual evaluations of contractor’s performance against the criteria  specified in a performance evaluation plan. During the course of the  evaluation period, performance monitors track contractor performance,  and once the period is over they assess the performance and report to the  performance evaluation board (PEB). The PEB considers the reports as  well as any other pertinent information and prepares a report for the fee  determination official (FDO) with findings and recommendations. The  contractor is given an opportunity to provide a self-assessment of its  performance during the evaluation period, which is often a written report.  The FDO may meet with the PEB to discuss the report, after which a final  determination is made in writing as to the amount of fee to be paid. The  FDO provides the determination to the contracting officer and a copy of  the related document to the contractor.
section_title: Acquisition Environment Can Affect Acquisition Outcomes
paragraphs: When discussing award-fee contracts, it is important to acknowledge the  acquisition environment in which they are used. Award fees are intended  to motivate excellent contractor performance, which should result in  excellent program outcomes. However, award fees should not be used to  make up for factors internal or external to the acquisition environment  that hinder the success of acquisition outcomes. These factors may  include inadequate resources and financial management systems, lack of  knowledge prior to starting the acquisition, or unsound acquisition  practices. We have reported that in some cases, NASA’s failure to define  requirements adequately and develop realistic cost estimates resulted in  projects costing more, taking longer, and achieving less than originally  planned. The persistence of these problems in NASA contract  management is not only indicative of undisciplined processes or practices  such as these, but may also reflect the fact that the design, development,  and production of major space systems are extremely complex technical  processes that must operate within complex budget and political  processes. Even properly run programs can experience problems that may  arise from unknowns, such as technical obstacles and changes in  circumstances. Only a few things need to go wrong to cause major  problems, and many things must go right for a program to be successful.
section_title: NASA’s Award-Fee Policy Addresses Many Cost-Plus- Award-Fee Contracting Issues Identified as Problematic
paragraphs: The NASA FAR Supplement and NASA’s Award Fee Contracting Guide  address many of the issues and problems identified by NASA on the use of  award-fee contracts and provide criteria for appropriately using such  contracts. Much of the guidance on award-fee contracting was issued in  response to weaknesses in CPAF contracting practices identified by NASA  internal reviews and NASA’s Office of Inspector General in the early  1990s. Those weaknesses included the awarding of excessive fees with  limited emphasis on acquisition outcomes (end results, product  performance, and cost control); rollover of unearned fee; use of base fee;  and the failure to use both positive and negative incentives. NASA updated  its award-fee guide in 1994, 1997, and 2001 to explain and elaborate on its  award-fee policy. The 2001 revision also reflects the FAR’s additional  emphasis on using performance-based contracts.
NASA’s award-fee guide emphasizes tying fees to outcome factors. The  guide states that outcome-based factors are the least administratively  burdensome type of evaluation factor and should provide the best  indicator of overall success. The award-fee guide warns against  micromanaging performance and diluting the emphasis of criteria by  spreading the potential award fee over a large number of performance  evaluation factors. Instead, the guide recommends selecting broad  performance evaluation factors, such as technical factors, project  management, and cost control supplemented by a limited number of  subfactors under these factors.
Cost control is required to be a key performance evaluation factor in  award-fee performance evaluation plans, largely because of past  performance issues in which contractors were paid millions of dollars in  fees on contracts that were experiencing hundreds of millions of dollars in  cost overruns. The NASA FAR Supplement states that cost control shall  be no less than 25 percent of the total weighted evaluation factors when  explicit evaluation factor weightings are used. The NASA FAR Supplement  states that emphasis on cost control should be balanced against other  performance requirement objectives, and the contractor should not be  incentivized to pursue cost control to the point that overall performance is  significantly degraded.
NASA’s regulations prohibit rolling over unearned fee to subsequent  evaluation periods for service contracts. For such contracts, each interim  evaluation and the last evaluation are final. Another key element of the  current award-fee regulations is an increased emphasis on overall  contractor performance and the end product, rather than on incremental  progress. NASA requires conducting interim evaluations on end item  contracts until final product delivery to monitor performance prior to  contract completion and establish the basis for making interim payments.  At the end of the contract, a final evaluation is conducted and the  contractor’s total performance is evaluated against the award-fee plan to  determine total earned award fee. For example, the contractor may be  evaluated and paid an interim fee once every 6 months until the product is  delivered. During the final evaluation, the contractor’s performance is  evaluated to determine total earned award fee. The final evaluation may  result in the contractor retaining the fee previously awarded or receiving  additional or less fee than previously awarded and thus refunding a  portion of the fee to the government. The final evaluation provides NASA  the opportunity to make an award-fee decision based on actual quality,  total cost, and ability to meet the contract schedule at the point the final  product is delivered.
Further, under the award-fee policy in effect prior to the 1994 and  subsequent revisions to the guidance, base fee was allowed on all CPAF  contracts. NASA’s current regulations prohibit the use of base fee on  service contracts and restrict the use of base fee on end item contracts,  such as for hardware. When base fee is used, it is not to exceed 3 percent  of estimated contract costs and it should only be paid if the final award-fee  evaluation is satisfactory or better. We note that base fee, which was paid  on two of the three end item contracts we reviewed, did not exceed 3  percent, and none of the seven service contracts included base fee.
Another issue addressed by NASA’s regulations is the use of both positive  and negative performance incentives in its CPAF contracts. The NASA  FAR Supplement provides that award-fee contracts with primary  deliverables of hardware and with a total estimated cost and fee of greater  than $25 million require both kinds of incentives based on measurements  of hardware performance against objective criteria. Performance  incentives are separate and distinct from award fee and measure  contractor performance up to delivery and acceptance. Performance  incentives are designed to reward contractors when performance of  delivered hardware is above minimum contract requirements. For  example, if the government establishes a specified level of objective  performance for a product that the contractor exceeds, the contractor can  be paid a performance incentive in addition to the award fee already paid.  If the contractor just meets this measure, it cannot receive an additional  performance incentive and keeps the award fee already paid. If the  contractor fails to meet the measure, however, it must pay a negative  performance incentive fee that reduces or eliminates the entire award fee.
To address inconsistencies among NASA centers in how they evaluate  contractor performance, the current award-fee regulations also provide a  uniform rating system to be used for all NASA award-fee contracts. It  includes adjectival ratings as well as a numerical scoring system of 0-100.  Scores of 61-70 percent are considered satisfactory, and the regulations  specify that contractors receiving a rating of less than 61 percent will not  receive any fee. A contractor is not to be paid any base fee or award fee  for less than satisfactory overall performance.
NASA’s award-fee guide encourages the use of performance-based  contracts for the procurement of services and supplies. The guide states  that constructing performance-based contracts that clearly define  performance requirements, include easily understood performance  standards, and have an objective incentive mechanism will result in  contractors having a clearer understanding of the government’s  expectations and will ultimately facilitate enhanced contractor  performance.
Finally, because of the cost and administrative burden associated with  administering award-fee contracts, the FAR and NASA’s award-fee guide  specify consideration of the costs and benefits of using a CPAF contract  before committing to this contract type. Through an evaluation of the  administrative costs versus the expected benefits, the contracting officer  should be able to assess whether the benefits the government gains  through a CPAF contract will outweigh the additional costs of overseeing  and administering the contract. The award-fee guide provides an example  of how to calculate the administrative cost and states that benefits could  be measured in dollars saved through cost control or enhanced technical  capability.
section_title: NASA Has Not Always Followed the Preferred Approach Laid Out in Its Award- Fee Guidance
paragraphs: Although the revisions in NASA’s regulations and guidance on award-fee  contracts address many weaknesses previously identified, the contracts  that we reviewed did not always demonstrate use of award fees by the  centers in the way that NASA prefers as outlined in its guidance. Some  performance evaluation plans or reports included input evaluation factors,  which are not the best indicators of success relative to the desired result,  although they are allowed by the guidance. Other contracts included  numerous subcategories for evaluating the contractor that can lessen the  importance of any particular subcategory and reduce the leverage of the  award fee on any particular criterion. Also, although the FAR and NASA’s  award-fee guide calls for a consideration of the costs and benefits of using  cost-plus-award-fee contracts because of the cost and administrative  burden involved, we found no examples of a documented analysis of costs  and benefits. Finally, NASA officials expressed satisfaction with the  results of the contracts based on their evaluations of contractor  performance against criteria established in the award-fee plan. Those  evaluations would indicate generally good performance. However, that  performance did not always translate into desired program outcomes.  NASA paid a majority of the available award fee on all of the contracts we  reviewed, including those end item contracts that did not deliver a  capability within initial cost, schedule, and performance parameters. That  disconnect raises questions as to the extent NASA is achieving the  effectiveness it sought through the establishment of guidance on the use of  award fees. Further, NASA has not evaluated the overall effectiveness of  award fees in promoting program outcomes and does not have metrics in  place for measuring their effectiveness in achieving program outcomes.
subsections: section_title: Some Evaluation Factors Were Not Outcome Factors as Preferred by NASA Guidance
paragraphs: Some performance evaluation subfactors included in performance  evaluation plans or reports were not outcome oriented. NASA’s award-fee  guide states that while it is sometimes valuable to consider input and  output factors when evaluating contractor performance, it is NASA’s  preference when feasible to tie fees to evaluation factors that are based on  outcomes because outcome-based factors provide the best indicator of  overall success. The award-fee guide recommends selecting broad  performance evaluation factors, such as technical factors, project  management, and cost control, and cautions that factors related to  intermediate processes, procedures, and actions may cause the contractor  to divert its attention from the overall desired outcome. The guide states  that those types of factors, while allowed, are not always true indicators of  the contractor’s performance and should be relied on with caution.  Further, with service contracts, input factors may be of little or no value as  a basis for evaluation. While the contracts we reviewed generally used  outcome factors as part of the evaluation of performance, some  supporting subfactors that formed the basis of the ratings for performance  measured compliance with process or input factors that may not provide  the best indicators of success relative to the desired results.
For example, a part of the award fee on the Mechanical System  Engineering Services (MSES) contract was to be awarded for program and  business management performance. There were five subfactors under this  primary performance factor. Two of these subfactors, program planning  and organizational management and business management were input  subfactors. These two input subfactors measure contractor processes or  inputs, but do not focus on final results. Subfactors in the Landsat-7  contract included input subfactors such as responsiveness of the  contractor’s corporate management, quality and effectiveness of the  contractor’s scheduling system, and prudent utilization of manpower and  timely removal of manpower upon completion of tasks.
section_title: Some Contracts Did Not Limit the Number of Subfactors for Evaluating Contractor Performance as NASA Guidance Recommends
paragraphs: The NASA award-fee guide cautions that spreading the potential award fee  over a large number of performance evaluation factors dilutes emphasis  on any particular performance evaluation criterion, increases the prospect  of any one item being too small and thus overlooked, and increases the  administrative burden. It encourages broad performance evaluation  factors such as technical factors, project management, and cost control,  which should be supplemented by only a limited number of subfactors  describing significant evaluation elements over which the contractor has  effective management control. Our analysis showed that a large number of  subfactors were used to evaluate contractor performance for some  contracts.
For example, the Jet Propulsion Laboratory (JPL) contract, which includes  both service and product deliverables defined in task orders under the  contract, uses three primary performance evaluation factors for measuring  contractor performance—programmatic, scientific, and engineering;  institutional management; and support to outreach initiative programs.  Although the JPL performance evaluation plan characterizes award-fee  subfactors as representing major areas of emphasis during the  performance period, the award-fee subfactors used to support the broad  performance evaluation factors were numerous—96 subfactors were used  to evaluate the contractor’s performance in fiscal year 2004, and 108  subfactors were used in fiscal year 2005. The Engineering and Technical  Support for Life Sciences contract used three broad performance  evaluation factors also—technical performance, schedule performance  and contract management, and cost control—but evaluated the contractor  on numerous supporting subfactors identified as tasks or subtasks in the  contractor performance evaluation reports. For example, on one task  order under this contract, performance evaluation reports for various  evaluation periods showed as many as 50 different subtasks used to  evaluate the contractor’s performance for the primary evaluation criteria:  (1) technical performance and (2) schedule performance and contract  management.
The Landsat-7 contract also included a number of subfactors. Contractor  performance under this contract was evaluated in several different areas  each time the performance evaluation board met. Technical performance  and program management were grouped together in one primary  performance evaluation factor, and business management and cost  performance were grouped together in the other primary performance  evaluation factor. There were 9 subfactors under technical performance  and 12 subfactors under program management, including quality and  effectiveness of the contractor’s scheduling system. Under business  management and cost performance, 17 evaluation subfactors and elements  were to be considered, including compliance with general contract  provisions and clauses and weekly scheduling of teleconferences to  determine schedule status. In addition to the number of subfactors that fell  under the two primary performance evaluation factors, there were nine  additional evaluation criteria, including resourcefulness, communication,  and responsiveness.
section_title: NASA Did Not Perform Cost-Benefit Analyses
paragraphs: Although the FAR and NASA’s award-fee guide require consideration of  the costs and benefits of using a CPAF contract before committing to this  contract type to determine whether the benefits outweigh the additional  cost and administrative burden of managing the contract, we found no  instances where a documented cost-benefit analysis had been done for any  of the contracts under our review. According to the guidance, since award- fee contracts require additional administrative effort, they should be used  only when the contract values, performance period, and expected benefits  are sufficient to warrant that additional management effort. Careful  selection of the most appropriate contract type and careful tailoring  should prevent a situation in which the burden of administering the award  fee is out of proportion to the improvements expected in the quality of the  contractor’s performance and in overall management. In addition, CPAF  contracts can be particularly costly and burdensome for NASA to  administer because of contract reporting and review requirements. Major  cost drivers include the number of award-fee periods, performance  monitors, and performance evaluation board members necessary for  implementing the award-fee process. For example, according to NASA’s  Award Fee Contracting Guide’s conservative estimate, it would cost  about $387,000 to administer the award-fee process over the life of a 5-year  contract. The guide notes that the estimate does not represent all  associated administrative cost that might arise. Although NASA  procurement officials acknowledged that formal cost-benefit analyses  were not prepared, some officials referred to determination and findings  statements or acquisition strategy meeting documents associated with  specific contracts as providing some evidence of consideration given to  whether or not CPAF contracts should be used.
section_title: Award-Fee Payments at Times Did Not Reflect Program Outcomes
paragraphs: While NASA officials expressed satisfaction with the results of the  contracts, in some cases there appeared to be a disconnect between the  fee paid and program results. NASA paid most of the available fee on all of  the contracts we reviewed—including on projects that showed cost  increases, schedule delays, and technical problems. The total estimated  value of the 10 contracts we reviewed was more than $31 billion. NASA  paid between 80 and 99 percent of the maximum award fee possible on  these contracts. The average was 90 percent, which equated to almost a  billion dollars in total award fees paid under the 10 contracts. Table 2  shows the percentage of award fee paid for each of the 10 contracts we  reviewed.
NASA officials expressed satisfaction with contract results, which was  further evidenced by its evaluations of contractor performance against  criteria established in the award-fee plan. While NASA’s evaluations would  indicate generally good performance, such performance did not always  translate into desired program outcomes. That disconnect raises questions  as to the extent NASA is achieving the effectiveness it sought through the  establishment of guidance on the use of award-fees. On the end item  contracts we reviewed, although there were some periods in which NASA  paid a lesser percentage of the available fee, NASA ultimately paid more  than 90 percent of the available fee based on its evaluation of contractor  performance against criteria in the award-fee plan even when those  contracts did not deliver capability within initial cost, schedule, and  performance parameters. For example:    The prime contractor for the International Space Station (ISS) has  received 92 percent of the total award fee available—$425.3 million— although the cost increased by 131 percent, from $5.6 billion to $13  billion, in part due to increased contract scope and delays caused by  the Columbia accident, but also contractor cost overruns. In addition,  the contractor estimates that it will incur an additional $76 million in  overruns by the time the contract is completed. Further, the  completion date for space station assembly under the prime contract  was delayed by 8 years. In some cases these delays were caused by  actions not within the control of the contractor, such as problems with  the shuttle program and actions by the international partners.
The contractor for the Earth Observing System Data and Information  System (EOSDIS) Core System (ECS) was paid 97 percent of the  available award fee—$103.2 million—despite a delay in the completion  of the contract by more than 2 years and an increase in the cost of the  contract from $766 million to $1.2 billion. Technical problems, schedule  delays, and cost control problems led to a major restructuring of the  contract.
The Landsat-7 contractor was paid 99 percent of the available award  fee or more than $17 million. The original contract was managed by  the Air Force but was subsequently transferred to NASA and  rebaselined. The cost of the contract when transferred to NASA and  rebaselined was $342.7 million. The Landsat-7 launch was delayed by 9  months and although the original scope of the work under the contract  was significantly reduced, the cost of the contract increased. By the  time the contract was complete, costs had risen 20 percent to $409.6  million.
While some NASA officials pointed out that problems encountered on  these contracts were at times outside the control of the contractor,  difficulties such as these with achieving program results have resulted in  NASA contract management being considered a high-risk area by GAO. We  did not review these contracts to determine responsibility for undesirable  results and therefore make no conclusion as to whether the fee paid was  appropriate on each particular contract. However, the high fees paid on  contracts where programs experienced disappointing results raise  questions as to the effectiveness of award fees as a tool for obtaining  desired program outcomes.
For the service contracts we reviewed, NASA officials reported that they  were satisfied with the results and quality of services provided. While we  could not assess these contracts against cost, schedule, and performance  outcomes as we could with the end item contracts, we did assess the  award-fee criteria used in these contracts against NASA guidance. Here we  found instances of process and input-oriented subfactors and the inclusion  of numerous subfactors in evaluating performance. Further, we found no  evidence that a cost-benefit analysis had been performed prior to choosing  the contract type. Taken together, this is not the preferred approach  according to NASA guidance, which raises questions as to the degree to  which performance outcomes—getting the quality of service desired—was  actually the basis for judging contractor performance and awarding fee.
section_title: NASA Has Not Assessed the Effectiveness of Award Fees in Achieving Program Outcomes or Developed Metrics for Conducting Such Evaluations
paragraphs: NASA views CPAF contracts as a viable and often preferred mechanism  for acquiring the types of goods and services that the agency procures.  NASA’s satisfaction with the results of these contracts is evidenced by the  level of fee paid on all of the contracts we reviewed and is based on  NASA’s evaluation of compliance with criteria contained in its award-fee  plans. However, the agency has not evaluated the overall effectiveness of  award fees in promoting desired outcomes. As noted, NASA developed its  new policies on award-fee contracts because the agency and its Office of  Inspector General found that it was paying excessive fees with limited  emphasis on acquisition outcomes. However, according to NASA officials,  the agency has not completed any assessments of the effectiveness of  award fees since the award-fee policy was restructured in the 1990s, nor  has it developed metrics or performance measures to conduct such  evaluations. Further, NASA lacks an agencywide system with the  capability of compiling and aggregating award-fee information and for  identifying trends and outcomes. According to NASA officials, even  NASA’s modern Integrated Enterprise Management Program (IEMP) will  not provide this capability. Thus, NASA cannot meaningfully judge how  well award fees are improving or can improve contractor performance and  program outcomes.
section_title: Conclusions
paragraphs: NASA could better link its award fees to desired results by making greater  use of outcome factors, its preferred criteria for evaluating award fee  contracts. While NASA has established policies and guidance that provide  an appropriate framework for their use, the agency has not always used  award fees as preferred by its guidance. To the extent that NASA uses  input evaluation factors and numerous subfactors for evaluating  performance, NASA may be diluting the leverage of award fees in  achieving desired results. Our review raises questions as to the extent  NASA is achieving the effectiveness it sought through the establishment of  guidance on the use of award fees. However, NASA has not evaluated the  overall effectiveness of its implementation of award fees.
section_title: Recommendations for Executive Action
paragraphs: We are making the following three recommendations to increase the  likelihood that the award fees NASA pays incentivize high performance  from its suppliers.
We recommend that the NASA Administrator reemphasize to the NASA  centers the importance of tying award-fee criteria to desired outcomes  and limiting the number of subfactors used in evaluations.
To ensure that cost-plus-award-fee contracts are used only when their  benefits outweigh the costs, we recommend that the NASA  Administrator direct the centers to consider costs and benefits in  choosing this contract type by requiring documentation explaining how  the perceived benefits will offset the additional cost associated with its  administration as required by the FAR.
Finally, we recommend that the NASA Administrator require the  development of metrics for measuring the effectiveness of award fees,  establish a system for collecting data on the use of award-fee contracts,  and regularly examine the effectiveness of award fees in achieving  desired acquisition outcomes.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, NASA concurred with our  recommendations and indicated that it would reemphasize its current  guidance as recommended, address the issues raised by the report in  training, and cover those issues in its internal reviews of procurement  operations at the individual Space Centers. In terms of our  recommendation to develop metrics for measuring the effectiveness of  award fees and establish a system for collecting data on the use of award-  fee contracts, NASA concurred and indicated it would explore the best  way to develop and use metrics for evaluating the effectiveness of award  fees and set up a system for collecting data on award-fee contracts. NASA  said it planned to contact the Department of Defense to obtain information  on its process, since DOD is also developing such a data collection system  and metrics for measuring the effectiveness of award fees. NASA also  provided technical comments on the draft, which have been incorporated  as appropriate.
As agreed with your office, unless you announce its contents earlier, we  will not distribute this report further until 30 days from its date. At that  time, we will send copies to interested congressional committees and the  NASA Administrator. We will also make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or calvaresibarra@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Key contributors to this report are  acknowledged in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: Our objectives were to determine (1) the extent the National Aeronautics  and Space Administration’s (NASA) guidance addresses the problems  previously identified with the use of award-fee contracts and (2) whether  NASA follows its guidance in using award fees to achieve desired  outcomes.
We selected 10 NASA cost-plus-award-fee (CPAF) contracts to review. Our  selection was based on contract data from the Federal Procurement Data  System. We extracted information on all NASA contracts active between  fiscal years 2002 and 2004 that were coded as CPAF. To ensure the validity  of the database from which we drew our contracts, we confirmed the  contract type of each of the 10 contracts we selected through NASA  contracting officers and contract documentation. The contracts we  selected were the top 10 dollar value contracts active from fiscal years  2002 through 2004. These contracts account for about $7.6 billion, or 44  percent, of obligated cost-plus-award-fee-dollars for the 3-year period.
To determine the extent NASA’s guidance addresses the problems  previously identified with the use of award-fee contracts and whether  NASA follows its guidance in using award fees to achieve desired  outcomes, we interviewed responsible program and procurement officials  at NASA headquarters and six NASA centers. We also reviewed the  Federal Acquisition Regulation (FAR), the NASA FAR Supplement, and  NASA’s Award Fee Contracting Guide. We conducted a literature review  and examined previous reports, studies, and analyses done by GAO,  NASA, the NASA Inspector General, or others that included information  related to NASA’s use of award fees and other relevant issues.
Additionally, we reviewed contract files, obtained information from  program and contracting officials through the use of a structured  questionnaire, and discussed the application of award-fee criteria with  NASA officials involved in the award-fee process. The contract documents  we reviewed contained information related to the development and  implementation of the award fee. This information included the basic  contract and statement of work, acquisition planning documents, award- fee modifications, performance evaluation plan documentation describing  fee criteria for specific evaluation periods, contractor self-assessments,  performance evaluation board reports, and fee determination documents.  We used this information to corroborate and supplement the information  provided by NASA officials in response to structured questionnaires we  prepared and interviews we conducted. We e-mailed the questionnaires  and received written responses for all 10 of the contracts. We conducted  structured interviews with contracting and program officials concerning  the development, implementation, and effectiveness of the award-fee  structure for some of the contracts.
To accomplish our work, we visited NASA headquarters in Washington,  D.C. We also visited and held teleconferences with Goddard Space Flight  Center in Greenbelt, Maryland, responsible for managing 3 of the contracts  we reviewed; Johnson Space Center in Houston, Texas, responsible for  managing 3 of the contracts; and Marshall Space Flight Center in  Huntsville, Alabama, responsible for managing 1 of the contracts. We held  teleconferences with officials at the Jet Propulsion Laboratory in  Pasadena, California; Kennedy Space Center in Cape Canaveral, Florida;  and Ames Research Center in Moffett Field, California, responsible for  managing 1 contract each under our review.
Our work was conducted from August 2005 through October 2006 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Summary Description of the 10 NASA Contracts GAO Reviewed
subsections: section_title: NAS5-60000–Earth Observing System Data and Information System Core System-Goddard Space Flight Center
paragraphs: NAS5-60000 was an end item hardware cost-plus-award-fee contract  between NASA and Hughes Applied Information Systems Incorporation.  Raytheon Information Systems Company acquired Hughes in December  1999 and became the prime contractor. The contract, currently closed, was  managed by Goddard Space Flight Center. The 10-year research and  development contract was awarded in March 1993 for the development  and operation of the Earth Observing System Data and Information System  Core System. The period of performance on the contract actually ended in  April 2005, and the contract has since been closed. According to Goddard  Space Flight Center procurement officials, the desired program outcome  or objective of the contract was to develop a technically capable system to  process data from NASA’s satellites at a reasonable cost. Procurement  officials stated that the Earth Observing System Data and Information  System Core System, a state-of-the-art data-processing system, is currently  dedicated to the processing and dissemination of NASA Earth Science  satellite data.
section_title: NAS15-10000-International Space Station-Johnson Space Center
paragraphs: NAS15-10000 is an end item hardware cost-plus-award-fee contract  between NASA and the Boeing Company. The contract, currently active, is  managed by the Johnson Space Center. A letter contract was awarded in  November 1993 and was definitized in January 1995 as a cost-plus- incentive-fee award-fee contract. In October 1999, during a restructuring  of the contract, the cost-plus-incentive-fee award-fee contract was  converted to a cost-plus-award-fee contract. The contract was extended in  December 2003, partially because of the Columbia accident. This planned  10-year contract is for the design, development, manufacture, and on-orbit  assembly of the U.S. on-orbit segment of the International Space Station.  The contract also included provisions for a level of effort that included (1)  sustaining engineering, (2) multi-element integrated testing, (3) logistics  and maintenance–post production support, (4) technical definition of  contract changes, and (5) other engineering support. According to  Johnson Space Center procurement officials, the desired program  outcomes or objectives of this contract are (1) completion of the U. S. on- orbit segment, delivery, and on-orbit acceptance of the space station; (2)  sustaining engineering of the U.S. on-orbit segment hardware and software  and common hardware and software provided to international  partners/participants and payloads; (3) post-production support of the U.S.  on-orbit segment hardware and common hardware provided to the  international partners/participants; and (4) space station end-to-end  subsystems management for the majority of the subsystems and specialty  engineering disciplines.
section_title: NAS5-32633-Landsat-7 Spacecraft-Goddard Space Flight Center
paragraphs: NAS5-32633 was an end item hardware cost-plus-award-fee contract  between NASA and Lockheed Martin Missiles and Space. The contract,  currently closed, was managed by Goddard Space Flight Center. The  research and development contract was initially awarded by the Air Force  in October 1992 and transferred to NASA in May 1994. The contract was  for the design, development, fabrication, integration, test, and pre- and  post-launch support of the Landsat-7 spacecraft. Landsat-7 was launched  in April 1999; the contract was completed in 2005. The purpose of the  Landsat-7 satellite is to obtain continuous remotely sensed, high-resolution  imagery of the earth’s surface for environmental monitoring, disaster  assessment, land use and regional planning, cartography, range  management, and oil and mineral exploration. According to Goddard  Space Flight Center procurement officials, the desired program outcome  or objective of the contract was to develop an operational satellite that  met the science requirements of users and the laws requiring the data be  obtained at a reasonable cost.
section_title: NAS8-60000-Program Information Systems Mission Services-Marshall Space Flight Center
paragraphs: NAS8-60000 was a cost-plus-award-fee service contract between NASA and  Computer Sciences Corporation. The contract, managed by the Marshall  Space Flight Center, was in the process of being closed as of June 2006. It  was awarded in May 1994, and covered a 2-year period of performance,  but included options to extend the period of performance for an additional  6 years—through April 30, 2002. The contract was extended three times,  with the period of performance ending on March 30, 2004. The primary  purpose of the contract was to provide services in the area of program  information system mission services. The contractor’s responsibilities  were to manage, be responsible for, and provide information services to  meet requirements of the Information Systems Services Office and its  customers. According to Marshall Space Flight Center procurement  officials, the desired program outcome or objective of the contract was to  provide services including operating and maintaining existing equipment  and software; gathering, analyzing, defining, and documenting systems  requirements; and planning, designing, developing, acquiring, integrating,  testing, and implementing new systems or enhancements to existing  systems.
section_title: NAS2-14263-Engineering and Technical Support for Life Sciences-Ames Research Center
paragraphs: NAS2-14263 was a cost-plus-award-fee service contract between NASA and  Lockheed Martin Engineering and Science Company, defined under task  orders. The contract, managed by Ames Research Center, was in the  process of being closed as of June 2006. Its period of performance ended  in September 2003. The 5-year research and development contract was  awarded in May 1995 for the provision of engineering and technical  support services for Ames Research Center life sciences. The work to be  performed included engineering and technical support for life sciences  projects, including space shuttle life sciences payloads, other life science  payloads, the Space Station Biological Research Project, ground-based life  sciences research, and advanced life support technology development.  According to Ames Research Center procurement officials, the desired  program outcome or objective of the contract was to achieve support for  space life science projects, life sciences research, and related technology.
section_title: NAS9-19100-Science, Engineering, Analysis, and Test-Johnson Space Center
paragraphs: NAS9-19100 was a cost-plus-award-fee service contract between NASA and  Lockheed Martin with indefinite delivery, indefinite quantity task orders;  performance-based; and level-of-effort provisions. Following the merger of  Lockheed and Martin in 1995, NASA consolidated two existing contracts to  form NAS9-19100 with an effective date of October 1, 1996. The contract,  managed by Johnson Space Center, was in the process of being closed as  of June 2006. The period of performance ended in January 2005. The  contract included requirements related to hardware, government- furnished crew equipment, facilities, laboratory maintenance, life sciences,  flight hardware, and support for the science and engineering requirements  of the Space Shuttle Program and the International Space Station Program.  According to Johnson Space Center procurement officials, the desired  program outcomes or objectives of the contract were to provide  engineering and science support to all engineering directorates at Johnson  Space Center as well as support both the science and engineering  requirements of the shuttle and space station programs.
section_title: NAS9-98100-Consolidated Space Operations Contract- Johnson Space Center
paragraphs: NAS9-98100 was a cost-plus-award-fee service contract between NASA and  Lockheed Martin Space Operations Company, with task orders and level- of-effort provisions. The contract, which was in the process of being  closed as of June 2006, was managed by the Johnson Space Center. It was  awarded on September 25, 1998, with a basic 5-year period of performance  and an option for an additional 5-year period. NASA chose not to exercise  the option for the second 5-year period of performance. The contract  required (1) developing an integrated operations approach to spacecraft  design, operations, and data processing that minimized cost and the  support infrastructure required to conduct space operations; (2) obtaining  a highly capable and accountable contractor that would be responsible for  providing space operations mission and data services; and (3) providing a  contract and management structure that would enable outsourcing,  commercialization, or privatization of some or all service under the  contract. According to Johnson Space Center procurement officials, the  desired program outcomes or objectives of the contract were to (1)  provide excellent quality and reliable mission and data services at a  significantly reduced cost; (2) move end-to-end mission and service  responsibility and accountability to industry; (3) implement an integrated  architecture that reduces overlap, eliminates unnecessary duplication, and  reduces life cycle costs; (4) define streamlined processes that minimize  intermediaries required to define requirements and deliver services; and  (5) adopt private sector commercial practices and services.
section_title: NAS10-99001-Joint Base Operation and Support- Kennedy Space Center
paragraphs: NAS10-99001 is a cost-plus-award-fee service contract between NASA and  Space Gateway Support. The contract, currently active, is managed by  Kennedy Space Center. The contract was awarded on October 1, 1998, for  a basic 5-year period of performance and included an option for an  additional 5 years. NASA exercised that option on October 1, 2003. The  purpose was to provide for base operations support at NASA’s Kennedy  Space Center and the Air Force’s Cape Canaveral Air Force Station, as  well as specific requirements at Patrick Air Force Base and Florida  Annexes into one consolidated contract. In addition to NASA and the Air  Force, other primary customers include the Navy, Department of Interior,  Spaceport Florida, and commercial customers such as Boeing, Lockheed  Martin, Orbital Science, and Astrotech. According to Kennedy Space  Center procurement officials, the desired program outcomes or objectives  of the contract are to (1) enhance safety for the public and on-site  workforce; (2) provide protection of human, national, and environmental  resources; (3) provide high-quality and responsive service to customers;  (4) reduce the cost of doing business for NASA and the Air Force; (5)  provide flexibility to respond to new requirements and unplanned events;  (6) improve supportability and reliability through innovative  methodologies and concepts; (7) provide common support practices and  systems; and (8) increase small business subcontracting goals.
section_title: NAS5-01090-Mechanical System Engineering Services-Goddard Space Flight Center
paragraphs: NAS5-01090 is a cost-plus-award-fee service contract between NASA and  Swales and Associates, with a line item for indefinite delivery, indefinite  quantity task orders. The contract, currently active, is managed by  Goddard Space Flight Center. NAS5-01090 was awarded in January 2001  with a period of performance of 5 years and 30 days. According to  Goddard Space Flight Center procurement officials, the period of  performance was extended and was currently scheduled to end on August  15, 2006. The purpose of the contract is to provide engineering services for  the study, design, development, fabrication, integration, testing,  verification, and operations of space flight and ground system hardware  and software, including development and validation of new technologies  to enable future science missions. According to Goddard Space Flight  Center procurement officials, the desired program outcomes or objectives  of the contract were to obtain high-quality performance, desired results,  and output.
section_title: NAS7-03001-Jet Propulsion Laboratory
paragraphs: NAS7-03001 is a cost-plus-award-fee contract between NASA and the  California Institute of Technology, a private nonprofit educational  institution, which establishes the relationship for the operation of the Jet  Propulsion Laboratory (JPL) federally funded research and development  center. The contract, currently active, is a 5-year research and  development contract awarded in November 2002 for the operation and  management of JPL. The contract allows for extension or decrease to the  initial period of performance in 3- or 9-month increments. JPL is a NASA- owned facility as well as an operating division of Caltech. Caltech has  operated JPL as a federally funded research and development center since  1959 to meet certain government research and development needs, which,  according to the contract, could not be met as effectively by existing  government resources or normal contractor relationships. The contract  includes both service and product deliverables, which are defined in task  orders issued under the contract. The contract encompasses a large  number of discrete programs and projects—approximately 500 active task  orders. According to NASA procurement officials, the desired program  outcomes or objectives of the contract are specific performance  requirements defined in task orders issued under the contract. The  contract encompasses support of exploration of the solar system,  including earth-based investigations, investigations and studies to support  NASA missions in the fields of earth science and astrophysics and  astrobiology, as well as development of supporting fundamental  technologies.
section_title: Appendix III: Comments from the National Aeronautics and Space Administration
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Thomas Denomme, Assistant  Director; James Beard; Shirley Johnson; Julia Kennon; Heather Barker  Miller; Kenneth Patton; Sylvia Schatz; and Robert Swierczek made key  contributions to this report.