Defense Contracting: Enhanced Training Could Strengthen DOD's Best Value Tradeoff Decisions
Published: 2010-10-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) obligated about $380 billion in fiscal year 2009 to acquire products and services. One approach DOD can take to evaluate offerors' proposals is the best value tradeoff process in which the relative importance of price varies compared to non-cost factors. The National Defense Authorization Act for Fiscal Year 2010 required GAO to review DOD's use of the best value tradeoff process, specifically when non-cost factors were more important than price. In response, GAO determined (1) how often and for what types of contracts DOD used the best value tradeoff process; (2) why and how DOD used such an approach; and (3) challenges, if any, DOD faces in using the best value tradeoff process. GAO identified a probability sample of new, competitively awarded fiscal year 2009 contracts in which DOD obligated $25 million or more. GAO reviewed guidance, solicitations, source selection decisions, and other documents for 129 contracts and interviewed DOD contracting and program staff about the use of the best value tradeoff process.
section_title: What GAO Found
paragraphs: In fiscal year 2009, DOD used best value processes for approximately 95 percent of its new, competitively awarded contracts in which $25 million or more was obligated. Almost half of DOD's contracts--47 percent--were awarded using a tradeoff process in which non-cost evaluation factors, when combined, were more important than price. DOD used best value tradeoffs principally to acquire services, such as construction of troop housing, as well as for professional management services. DOD used the best value tradeoff process in 88 of the 129 contracts GAO reviewed. For 60 of the 88 contracts, DOD weighted non-cost factors as more important than price. In these cases, DOD was willing to pay more for a contractor that demonstrated it understood complex technical issues more thoroughly, could provide a needed good or service to meet deadlines, or had a proven track record in successfully delivering products or services of a similar nature. In making tradeoff decisions, GAO found that DOD selected a lower priced proposal nearly as often as it selected a higher technically rated, but more costly proposal. Overall, GAO found that DOD paid a combined total of more than $230 million in price differentials--the difference in price between the awardee and the offeror next in line for award--on 21 contracts, but chose not to pay more than $800 million in proposed costs by selecting a lower priced offer over a higher technically rated offer in 18 contracts. DOD does not track whether the use of best value tradeoff processes correlates with the contractor successfully meeting the terms of the contract and noted that many factors ultimately contribute to an acquisition's success or failure. DOD officials identified several challenges in using the best value tradeoff process, including the difficulty in determining meaningful evaluation factors and the business judgment of acquisition staff required. DOD officials also noted that the complexity of the tradeoff process increases the risk of bid protests. For example, GAO found that 15 of the 88 contracts awarded using a best value tradeoff process reviewed were protested to GAO, resulting in 4 cases in which DOD terminated the contract or made a new source selection decision when DOD determined that it failed to adhere to the solicitations' requirements. Such concerns are heightened given the expected influx of more than 6,400 new contracting personnel over the next few years. According to DOD officials, making sound tradeoff decisions, and in particular, deciding whether or not a price differential is warranted, is one of the most difficult aspects of using a best value tradeoff process. DOD is developing a new departmentwide source selection guide and intends to subsequently revise its training curriculum, but neither the guide nor DOD's current training curriculum provides agency personnel with information on assessing price differentials when performing tradeoff analyses.
section_title: What GAO Recommends
paragraphs: GAO recommends that to help DOD effectively employ best value tradeoff processes, DOD develop training elements, such as case studies, that focus on reaching tradeoff decisions, as it updates its training curriculum. DOD concurred with this recommendation.

section_title: Letter
paragraphs: The Department of Defense (DOD) obligated about $380 billion in fiscal  year 2009 to acquire products and services needed to support its  missions. DOD has several approaches it can use to evaluate offerors’  proposals. For example, DOD can use a best value tradeoff process, in  which it can vary the relative importance of cost or price to other factors,  such as technical approach or past performance in its solicitations to  offerors. In these cases, DOD may award a contract to other than the  lowest-priced offeror if DOD determines that a higher-priced proposal  provides a greater benefit to DOD, and this greater benefit is worth pay an additional cost, or price differential. Proper execution of the tradeoff  process is essential, however, to ensure that DOD lays the foundation for  successful acquisition outcomes. We have identified DOD contract  management as a long-standing high-risk area, due in part to DOD’s use of  ill-suited business arrangements that have contributed to unmet  expectations and placed the department at risk of potentially paying more  than necessary.  ing  Section 845 of the National Defense Authorization Act for Fiscal Year 2010  directed GAO to report on DOD’s use of the best value tradeoff process,  and specifically for cases in which DOD determined that it would evaluate  contractors’ proposals on factors other than cost or price, if these non-cost  factors, when combined, were to be considered more important than cost  or price. To respond to the mandate, we determined (1) how often and for  what types of contracts DOD used the best value tradeoff process; (2) why  and how DOD used the best value tradeoff process; and (3) what  challenges, if any, DOD faces in using the best value tradeoff process.
To conduct our work we used the Federal Procurement Data System-Next  Generation (FPDS-NG) to identify new, competitively awarded contracts  in which DOD obligated $25 million or more in fiscal year 2009. We  selected the $25 million threshold based on a Defense Federal Acquisition  Regulation Supplement (DFARS) requirement that contracts for  production or services with $25 million or more in estimated total costs for  any fiscal year have written acquisition plans, which contain information  on the anticipated source selection approach. This analysis identified 363  contracts on which DOD had obligated a total of $39.2 billion, or about 10  percent of the total amount DOD obligated on contracts in fiscal year 2009.  From this population, we selected a random sample of 160 contracts,  including 60 indefinite delivery contracts. We identified errors in the  information provided by FPDS-NG on 31 contracts, including contracts  that were incorrectly coded as competitively awarded or had incorrect  award amounts. We excluded these contracts from our sample and  determined that FPDS-NG was sufficiently reliable for the purposes of our  review after adjusting for these errors. Based on our analysis of the  remaining 129 sample contracts, we produced estimates of source  selection approaches used, contract type, and whether a product or  service was acquired. To determine the type of and rationale for the  source selection process used for the contracts, we reviewed the  associated acquisition plans, solicitations, source selection decision  memorandums, and other relevant documents for each of the contracts in  our sample. In particular, we reviewed source selection decision  documents for contracts that used a tradeoff process to determine  whether DOD paid a price differential. For the purposes of this report, we  defined a price differential as the difference in price between the  awardee’s price and the price of the offeror next in line for award.
From our sample, we also judgmentally selected buying activities from  each military department and one defense agency based on such factors as  the number of best value contracts, the type of contract, and the type of  product or service acquired. Results based on these selected buying  activities are not generalizable to a larger population. At each activity, we  reviewed contract files and interviewed program and contracting officials  to discuss their rationale for selecting a tradeoff process including the  selection of non-cost evaluation factors for 34 contracts and 23 task orders  as illustrative case studies. We also reviewed contract documents, DOD  and military department source selection guidance, and interviewed  program and contracting officials from DOD, the military departments,  and one defense agency to identify what challenges, if any, DOD faces in  using the best value tradeoff process. A more detailed description of our  scope and methodology is included in appendix I.
We conducted this performance audit from March 2010 through October  2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Federal Acquisition Regulation (FAR) Part 15 allows the use of several  best value competitive source selection techniques to meet agency needs.  Within the best value continuum, DOD may choose an approach that it  considers the most advantageous to the government, including the lowest  price technically acceptable (LPTA) process and the tradeoff process.
DOD may elect to use the LPTA process in acquisitions where the  requirement is clearly definable and the risk of unsuccessful contract  performance is minimal. In such cases, DOD may determine that cost or  price should play a dominant role in source selection. When using the  LPTA process, DOD specifies its minimal technical requirements in the  solicitation. Once DOD determines that the contractors meet or exceed  the technical requirements, no tradeoffs between cost or price and non- cost factors are permitted and the award is made based on the lowest  price offered to the government.
By contrast, DOD may elect to use a tradeoff process in acquisitions where  the requirement is less definitive, more development work is required, or  the acquisition has greater performance risk. In these instances, non-cost  evaluation factors, such as technical capabilities or past performance, may  play a dominant role in the source selection and tradeoffs among price and  non-cost factors allow DOD to accept other than the lowest priced  proposal. This report focuses on DOD’s use of the tradeoff process, and  specifically, in which non-cost factors, when combined, were considered  more important than cost or price.
When using a tradeoff process, the FAR requires that evaluation factors  and significant subfactors that affect contract award and their relative  importance be clearly stated in the solicitation; and the solicitation must  provide whether all evaluation factors other than cost or price, when  combined, are significantly more important than, approximately equal to,  or significantly less important than cost or price. Additionally, the FAR  requires that each factor represent key areas of importance and emphasis  to be considered in the source selection decision and that support  meaningful comparison and discrimination between and among competing  proposals. The FAR also requires the source selection authority  document the perceived benefits of the higher priced proposal and the  rationale for tradeoffs in the contract file. The resulting source select decision should be based on a comparative assessment of proposals  against all source selection criteria in the solicitation. The decision must  also include the rationale for any business judgments and tradeoffs made  or relied on by the source selection official, including benefits asso ciated  with additional costs. Although the rationale for the source selection  decision must be documented, the documentation need not quantify the  tradeoffs that led to the d ecision.
In fiscal year 2009, DOD obligated about $380 billion on contracts for  goods and services. Our analysis of data reported by DOD to FPDS-NG  indicates that $69.9 billion or 18 percent of DOD’s obligations were made  on new contracts competitively awarded in fiscal year 2009 (see figure 1).  By contrast, about $176 billion were modifications to or orders issued  under contracts that were awarded prior to fiscal year 2009 and $133  billion were awarded non-competitively, which in combination totaled to  nearly 82 percent of DOD’s reported contract obligations in fiscal year  2009.
Properly managing the acquisition of goods and services requires an  acquisition workforce with the right skills and capabilities. In March 2009,  however, we reported that DOD lacked complete information on the skill  sets of the current acquisition workforce and whether these skill sets were  sufficient to accomplish its missions. In April 2009, the Secretary of  Defense announced his intent to grow the acquisition workforce by 15  percent by fiscal year 2015. As part of this strategy, DOD indicated that it  intends to grow its contracting career field by more than 6,400 personnel,  an increase of more than 28 percent from fiscal year 2008 staffing levels.
section_title: DOD Relies Heavily on Best Value Processes to Evaluate Contractor Offers
paragraphs: DOD relies heavily on the use of the best value process to evaluate offers  from potential contractors. DOD chose a best value process for  approximately 95 percent of its new, competitively awarded contracts on  which it had obligated $25 million or more in fiscal year 2009. Almost half  of DOD’s contracts—47 percent—were awarded using a tradeoff process  in which non-cost evaluation factors, when combined, were more  more  important than price. Figure 2 shows how often DOD used the different  important than price. Figure 2 shows how often DOD used the different  best value processes and other source selection approaches.  best value processes and other source selection approaches.
Bet ve process (95%)
Bet ve trdeoff process (69%)
In 69 percent of the contract awards, DOD used the best value tradeoff  process. When doing so, it acquired services approximately four times as  often as it acquired products. Over half of these procurements were for  building or civil engineering construction services, including projects for  troop housing, administrative facilities, and hurricane protection systems.  Other services procured using the tradeoff process were equipment  maintenance and professional management services. For example, in fiscal  year 2009, the Army Corps of Engineers awarded a contract worth more  than $963 million to construct one of the largest pumping stations in the  world, along with floodgates and floodwalls for hurricane protection.  Similarly, the Air Force awarded the Contract Field Team program  multiple-award contract, with an estimated base value of $2.6 billion for  modification, maintenance and repair of systems including aircraft and  missile defense for the departments of the Army, Navy, Air Force and  several federal agencies. Small arms and electronic countermeasure  equipment were among the products most frequently procured using a  tradeoff process, including the contracts for the Squad Automatic  Weapons—lightweight, automatic rifles issued to each Army and Marine  rifle squad—and an Army contract to procure devices that counteract  radio-controlled improvised explosives. Our analysis of selected  characteristics of contracts awarded using a best value tradeoff process in  fiscal year 2009 is shown in figure 3.
Percentages may not sum to 100 percent due to rounding.
Combination refers to contracts that allow for orders to be placed using more than one pricing  arrangement.
As part of our work, we reviewed 10 IDIQ contracts that had been  awarded using a best value tradeoff process and 23 task and delivery  orders under these contracts which had obligations ranging from             $11 million to over $319 million. In most cases, DOD did not issue the task  or delivery orders we reviewed using a tradeoff process. For example, an  Air Force official explained that the initial task orders for the Contract  Field Team program, including 13 orders in our sample, were issued under  an LPTA process because data needed to assess contractor performance  and timeliness were not yet available to use given the short time between  award of the base contract and issuance of the first orders. An Air Force  official indicated that once they had obtained sufficient performance data,  they intended to issue task orders using a tradeoff process when possible.  DOD officials issued six other task orders on the basis of negotiating with  a contractor who had been awarded a single award IDIQ contract. The  four remaining orders were awarded using a tradeoff process. For  example, the Army wanted infrared vision enhancement equipment for  nighttime and battlefield use in Iraq and Afghanistan to be delivered as  quickly as possible. Consequently, the Army used a contractor’s ability to  meet delivery requirements as the principal evaluation factor in selecting  the contractor for delivery order award.
Some DOD officials noted that the use of various source selection  evaluation methods can change over time. For example:    Defense Logistics Agency (DLA) officials noted that they have recently  transitioned from principally using the tradeoff process to using the LPTA  process for most fuel purchases because the majority of their  procurements were for a commercial product in relatively stable domestic  and international markets. They noted, however, that they still use the  tradeoff process in less stable areas, such as Iraq and Afghanistan, where  they require more information about vendors’ past performance and  technical capability when operating in war zones.
Conversely, Army Corps officials in New Orleans reported that they have  been using the tradeoff process more frequently since the increase in civil  works construction projects following Hurricane Katrina. While they  typically used sealed bids in the past, they told us that use of the tradeoff  process enabled them to better assess contractors’ ability to meet safety  and schedule requirements.
section_title: DOD Used a Best Value Tradeoff Process to Address Complex or Time Sensitive Needs, but Paid Relatively Few Price Differentials
paragraphs: DOD officials tended to use a best value tradeoff process with non-cost  factors weighted more important than price when they were willing to  accept a higher price if a contractor could demonstrate certain  advantages, such as meet a deadline, demonstrate that it understood  complex technical issues, or propose an innovative approach. DOD often  indicated in tradeoff solicitations that non-cost factors would be  significantly more important than price in making award decisions, but our  analysis indicated that DOD selected a lower priced proposal among those  offerors remaining in the final competition almost as often as it selected a  higher technically rated, but more costly, proposal. Overall, DOD paid a  price differential—the difference in the price of the offeror awarded the  contract and the price of the offeror next in line for award—in 21 of the 68  contracts in which a price differential was considered. Most differentials  were less than 5 percent. While DOD officials told us that the tradeoff  process provides an essential tool to obtain desired capabilities, they rely  on the case-by-case judgment of contracting and program officials to  determine the best acquisition approach suited to program requirements  and do not specifically track whether use of the tradeoff process is in  DOD’s interest.
subsections: section_title: Contract Requirements Drove DOD’s Selection of Technical and Past Performance Evaluation Factors
paragraphs: The FAR and DOD guidance generally provide acquisition staff flexibility  to develop evaluation factors that meet their procurement needs and does  not indicate which evaluation factors should be most important. The FAR  requires that DOD officials consider, among other things, past  performance on all negotiated competitive acquisitions exceeding  $100,000, but DOD officials have broad discretion in selecting other non- cost factors and their relative importance. The factors are intended to  provide meaningful discriminators to evaluate proposals. Army, Navy, and  Air Force officials told us that they formed interdisciplinary teams that  developed evaluation factors and the factors’ relative importance by  consensus.
We found that 88 of the 129 contracts we reviewed used a best value  tradeoff process. Our analysis shows that DOD considered past  performance and technical evaluation factors as the most important  among the non-cost factors. Figure 4 shows the five most frequently used  non-cost evaluation factors for the 88 contracts in our review in which a  tradeoff was conducted and how often the technical and past performance  factors were most important among the non-cost factors.
section_title: Air Force
paragraphs: DOD officials told us that the selection of these evaluation factors and  their relative importance was based on specific acquisition requirements,  such as the ability to meet production deadlines, ensure compatibility with  existing ship and aircraft systems, or provide needed security for delivery  of goods in war zones. Our analysis found that DOD considered non-cost  factors more important than price in 60 of the 88 contracts awarded using  a tradeoff process. The following illustrate instances where DOD’s  acquisition needs led them to make non-cost factors the principle criteria  for source selection.
Army officials had to quickly meet surge requirements based on a Joint  Urgent Operational Needs Statement for roadside bomb detectors as well  as services to provide training and support the system once fielded, and  accordingly, made technical capability the most important evaluation  factor. The acquisition plan specified that deliveries of the critical  technology and support services needed to be made within 6 months of  contract award.
Army officials sought contractors with innovative approaches and a  superior understanding on how to counter the threat of roadside bombs in  awarding a professional services contract for a range of training programs  to be used within the Military Service Combat Training Centers. According  to the Army, the selected contractor provided a proposal that was superior  in nearly all the technical categories sought by the Army.
The Navy considered contractors’ proposed technical approach the most  important evaluation factor for a helicopter upgrade kit procurement  because the design had to be compatible with existing helicopters. In this  case, the timing of fleet deployment was also critical and the Navy sought  a contractor that could meet their schedule.
DLA used a tradeoff process primarily for commercial fuel contracts in  dangerous areas, such as Iraq and Afghanistan, due to the heightened need  for contractor reliability in these war zones. In these situations, DLA  officials explained the tradeoff process allowed them to emphasize  security and past performance in their evaluations to mitigate acquisition  risks, especially since they do not know the vendors well.
Army Corps of Engineers officials that needed to procure construction  services for barracks for wounded soldiers made the technical and  performance capability factors most important because they needed to be  responsive to new schedule and price targets. These officials used the  tradeoff process to incentivize timeliness and price reductions, and they  were also able to obtain better features, such as more durable materials.
DOD officials also told us they used these non-cost factors to encourage  contractors to provide innovative solutions to meet DOD’s needs. For  example, Army officials expressed a need for technological innovation in a  solicitation for equipment, field support services, and associated  maintenance needed to intercept enemy communications. The Army  encouraged the contractors to develop a system that would enable them to  upgrade the equipment frequently over the life of the contract. The  statement of work clarified that these upgrades would be essential to  maintain relevancy in the battlefield and keep pace with technology  advancements. Similarly, Marine Corps officials we spoke to about an  urban warfare training system told us that they used the tradeoff process  to seek innovative designs when awarding a 5-year, $1 billion dollar  contract. Marine Corps officials indicated that they had a system that  worked, but wanted to push industry to come up with a solution that  allowed the Marines to reconfigure building structures more quickly and  to provide more realistic and current combat scenarios prior to  deployment. In the winning design, the offeror proposed using modular  building sets that Marines could assemble more quickly to maximize the  training opportunities available in the field.
In contrast, our analysis found that the 28 cases in which DOD officials  considered non-cost factors as equal to or less important than price were  nearly all related to construction projects. For example, in 15 cases we  reviewed, the Army Corps of Engineers considered non-cost factors such  as management and technical, experience, and past performance as equal  to price to address less complex project requirements such as building a  new runway for aircraft and constructing a maintenance facility. In these  instances, the contracting and program officials were able to request and  review information from potential contractors and conduct a tradeoff  process that would not be available through an LPTA approach, but still  considered price of equal importance to non-cost factors in the award  decision.
section_title: DOD Paid Relatively Few Price Differentials Despite Best Value Tradeoff Solicitations That Emphasized Non-Cost Factors
paragraphs: For the 88 contracts awarded using a best value tradeoff process, DOD  considered whether to pay a price differential in 68 contracts. Our  analysis indicated that DOD selected the lower priced option nearly as  often as it selected the highest rated, but more costly, proposal. In the 18  cases in which DOD officials decided not to pay a price differential, they  determined that the lower price outweighed the advantages of the offeror  with the higher technical rating. In doing so, DOD officials decided not to  pay over $800 million in price differentials. In 29 other cases, DOD  awarded contracts to the offerors that had both the lowest price and the  highest non-cost factor rating.
DOD accepted a higher price in 21 of the 68 contracts in which a price  differential was considered, for a combined difference of more than      $230 million. Most differentials paid were less than 5 percent above the  price submitted by the offeror next in line for award. The largest price  differential from the contracts in our sample was 48 percent higher, or  roughly $13.6 million more, than the next in line offeror’s price. In this  case, Marine Corps officials determined that the product—burn resistant  clothing for use by soldiers in Iraq—was worth the price difference  because it provided substantially greater 2nd and 3rd degree burn  protection than the product proposed by the other offeror. Figure 5 shows  the frequency with which DOD elected to pay or not pay a price  differential for the 68 contracts in which a price differential was  considered, as well as the value of the price differentials either paid or not  paid.
section_title: DOD Relies on the Judgment of Contracting and Program Officials to Determine Whether Use of Best Value Tradeoffs Meet DOD’s Needs
paragraphs: DOD contracting and program officials believed that the use of best value  tradeoffs provide DOD an essential tool, which allows them to obtain  better insights into the contractors’ capabilities and their understanding of  the government’s needs, and the reasonableness of the contractor’s  approach. DOD and military department officials stated that they do not  specifically track use of the tradeoff process to determine if DOD’s  interests are met. Instead, they rely on the judgment of contracting and  program officials to select the best acquisition approach suited to program  requirements on a case by case basis. Further, DOD officials stated that  they do not track whether the solicitation approach used correlated with  whether the contractor successfully met the terms of the contract and  noted that many factors ultimately contribute to the success or failure of  an individual acquisition that may not have been foreseeable when  awarding the contract. For example, DOD officials noted that DOD would  often use a best value tradeoff process to award a contract to develop a  major weapons system. As our work has found, DOD often encounters  cost increases, schedule delays, and performance shortfalls on its major  systems.
section_title: DOD Faces Several Challenges in Using the Best Value Tradeoff Process
paragraphs: DOD officials acknowledged several challenges in using the best value  tradeoff process such as the difficulties in developing meaningful  evaluation factors, the additional time investment needed to conduct best  value procurements, and the business judgment required of acquisition  staff when compared to other acquisition approaches. DOD officials also  noted that the complexity of the tradeoff process increases the risk of bid  protests. To help address source selection challenges, DOD is drafting a  source selection guide to improve consistency and standardize source  selection procedures for competitively awarded negotiated procurements.
DOD officials told us that developing non-cost factors that meaningfully  discriminate between offers is a challenging part of the tradeoff process.  They noted that as the complexity of the acquisition increases, so does the  need for individuals with the expertise to help develop the evaluation  factors. For example, Army Corps of Engineers officials told us that the  contract for one of the world’s largest flood pump stations required  experts with experience in issues ranging from water flow management to  real estate to develop evaluation factors. Further, Navy officials explained  that while they often use past performance as a non-cost discriminator, it  can be difficult to identify differences between contractor proposals  because contractors often provide their best performance examples and  the government often lacks data to evaluate additional contractor projects.  Our past work has also identified governmentwide challenges in obtaining  needed past performance information to support contract award  decisions. Further, the absence of meaningful non-cost discriminators  can result in offerors receiving equal scores on the factors that were  identified as being significantly more important than price. As such, the  decision may default simply to a consideration of price alone. For  example, Air Force officials noted that they are considering updating  factors used to award task orders under the Contract Field Team contract  because contractors tend to receive the highest ratings for each non-cost  factor reviewed, so price is typically the only discriminator.
DOD officials also noted that using the best value tradeoff process is often  far more time-consuming than other approaches. Navy officials told us  that the tradeoff process is administratively burdensome and requires a  large time investment from program staff, which can make it challenging  to keep the same acquisition team together for an entire procurement.  During our site visits, many contract and program staff told us that the  tradeoff process often takes between 18 and 24 months. In addition, in  Afghanistan and Iraq, the challenges of conducting a tradeoff process have  contributed to decisions by the CENTCOM Joint Theater Support  Contracting Command and Army Corps of Engineers to discourage its use.  For example, recent Army Corps of Engineer projects in Afghanistan have  emphasized using simpler, less complex designs or requirements that are  more suitable for the use of a lowest price technically acceptable  approach.
The complex nature of the best value tradeoff process, including decisions  on whether to pay a price differential, requires much greater business  judgment when compared to other acquisition approaches. DOD officials  stated that making tradeoff decisions, particularly when to pay a price  differential, is among the most difficult aspects of the tradeoff process,  which will become more challenging with less experienced staff coming  into the acquisition workforce. DOD officials indicated that DOD intends  to increase the size of its contracting career field by more than 6,400  personnel through fiscal year 2015. With the influx of new staff, many of  the contracting officers we met with noted challenges in preparing staff to  conduct the tradeoff process. For example, a Navy contracting officer told  us that guidance and training only go so far to prepare acquisition staff to  conduct best value tradeoff procurements. Instead, acquisition staff need  to be involved in a number of best value tradeoff procurements to develop  the business judgment necessary to conduct a successful acquisition.
DOD officials stated that the complexity of the tradeoff process also  increases the risk of bid protests. Of the 88 contracts we reviewed that  used a tradeoff process, 15 were the subject of a bid protest to GAO. While  most of the protests were denied, DOD took corrective actions in 5 cases,  including 4 cases in which DOD terminated the contract or made a new  source selection decision when it determined that it failed to adhere to the  solicitations’ requirements.
Some of the services have developed initiatives to address these  challenges. For example, the Air Force set up an Acquisition Center of  Excellence (ACE) at Tinker Air Force Base, which provides pre-award  source selection assistance to contract and program staff. Air Force  officials stated that ACE reviews the evaluation factors within individual  source selection plans, serves in an advisory capacity on source selection  teams and holds workshops for contracting officers. Similarly, Army  officials at Ft. Monmouth’s Communications—Electronics Command have  developed an online business tool—the ASSIST tool—that shepherds  contracting officers through the solicitation process. For example, the tool  provided a list of steps that must be completed for best value tradeoff  procurements and automatically routes documents through source  selection evaluation boards and other participating officials for review, as  required.
DOD is also drafting a departmentwide source selection guide to improve  consistency and standardize source selection procedures for competitively  awarded negotiated procurements. Given the influx of new acquisition  staff, DOD officials stated they wanted to develop a more prescriptive  guide for best value procurements. While the DOD draft source selection  guidance contains information on various aspects of the best value  process, such as the source selection decision document, DOD officials  told us it does not address price differentials. Numerous DOD officials  underscored the importance of training in the use of the best value  process, particularly training that addresses the tradeoff decision that  acquisition staff must make. For example, one Army Corps of Engineers  official told us that source selection officials would benefit by training that  contains real life lessons on how other officials have made price  differential decisions during the tradeoff process. Similarly, Marine Corps  and Army officials told us that while decisions are made on a case-by-case  basis, informal rules of thumb regarding price differentials can come into  play and indicated that additional guidance or training, especially case  studies or scenarios, would be helpful.
The Defense Acquisition University (DAU) is responsible for providing  training to the DOD acquisition workforce. According to DAU officials,  they offer more than 10 courses that contain elements of the best value  tradeoff process, but none of the current courses provide case studies or  scenarios that focus on reaching price differential decisions during source  selection. They noted that once the new source selection guidance is  implemented, which is anticipated for January 2011, they plan to augment  existing contracting courses to reflect the new guidance.
section_title: Conclusions
paragraphs: The best value tradeoff process underlies the vast majority of DOD  competitively awarded contracts, and effective use of this process hinges  on making sound tradeoffs between price and non-cost factors. By  focusing on non-cost factors, DOD anticipates that it will obtain technical  solutions that are innovative and address complex and time-sensitive  program requirements. Applying a tradeoff process, however, does not  guarantee successful acquisitions, nor is it without other challenges. In  particular, using a tradeoff process can be more complex and take more  time than other source selection methods, and requires that acquisition  staff have proper guidance, needed skills, and sound business judgment.  With the anticipated influx of more than 6,400 DOD contracting personnel  over the next few years, providing a firm foundation for use of the tradeoff  process is essential. While DOD and the military departments have taken  steps to improve source selection procedures, acquisition personnel noted  a lack of training to assist them in deciding whether or not a price  differential is warranted when making tradeoff decisions. For example,  while DOD’s new source selection guide provides insights on the source  selection process, it is silent on how to reach decisions on when to pay a  price differential, as is DOD’s current training curriculum. DOD has an  opportunity as it updates its training curriculum to provide acquisition  staff with better insights using real life examples on reaching tradeoff  decisions. Taking this step can help DOD minimize the risk of paying a  price differential when not warranted or losing the benefit of a technically  superior solution.
section_title: Recommendation for Executive Action
paragraphs: To help DOD effectively employ the best value tradeoff process, we  recommend that the Secretary of Defense direct the Director of Defense  Procurement and Acquisition Policy to work with the Defense Acquisition  University to develop training elements, such as case studies or scenarios  that focus on reaching tradeoff decisions, including consideration of price  differentials, as it updates the source selection curriculum.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided written comments on a draft of this report. DOD concurred  with our recommendation and intends to request the Panel on Contracting  Integrity—comprised of senior DOD leaders tasked, in part, to help  improve DOD's performance—to assist the Defense Acquisition University  in developing training case studies and scenarios that focus on reaching  tradeoff decisions. DOD’s letter is reprinted in appendix II.
We are sending copies of this report to interested congressional  committees and the Secretary of Defense. In addition, this report will be  available at no charge on GAO’s Web site at http://www.gao.gov.
Should you or your staff have any questions on the matters covered in this  report, please contact me at (202) 512-4841 or huttonj@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: Section 845 of the National Defense Authorization Act for Fiscal Year 2010  directed GAO to report on the Department of Defense’s (DOD) use of the  best value tradeoff process, and specifically for cases in which DOD  evaluated contractors’ proposals on factors other than cost or price, if  these non-cost factors, when combined, were considered more important  than cost or price. To respond to the mandate, we determined (1) how  often and for what types of contracts DOD used the best value tradeoff  process; (2) why and how DOD used the best value tradeoff process; and  (3) what challenges, if any, DOD faces in using the best value tradeoff  process.
To determine how often and for what types of contracts DOD used the  best value tradeoff process, we used data from the Federal Procurement  Data System-Next Generation (FPDS-NG) as of January 2010 to identify a  population of contracts based on the following criteria: (1) newly awarded  by DOD in fiscal year 2009; (2) competitively awarded, and (3) had  obligations of $25 million or more in fiscal year 2009.
We established the $25 million threshold because the Defense Federal  Acquisition Regulation Supplement (DFARS) requires contracts with total  estimated costs of $25 million or more in any fiscal year to prepare written  acquisition plans, which contain information about the source selection  approach. This analysis identified 363 contracts. From this population, we  selected a probability sample of 160 contracts, including 60 indefinite  delivery contracts, and reviewed associated solicitations, source selection  decision documents, and other contract documents to determine the  solicitation approach DOD used.
We verified the obligations and contract award fields in FPDS-NG with  contract data to ensure that the contracts within our sample were within  scope. Thirty-one contracts from our initial sample of 160 contracts were  outside the scope of our review because they were incorrectly coded in  key parameters, such as being coded as competitively awarded when they  were not or had misreported the amount of obligations made on the  contract or task order. We excluded these contracts from our sample and  determined that FPDS-NG was sufficiently reliable for the purposes of our  review after adjusting for these errors.
Because we followed a probability procedure based on random selections,  our sample is only one of a large number of samples that we might have  drawn. Since each sample could have provided different estimates, we  express our confidence in the precision of our particular sample’s results  as a 95 percent confidence interval (e.g., plus or minus 8 percentage  points). This is the interval that would contain the actual population value  for 95 percent of the samples we could have drawn. Unless otherwise  noted, percentage estimates based on our sample have 95 percent  confidence intervals that are within plus or minus 8 percentage points of  the estimate itself. Confidence intervals for other numeric estimates are  reported along with the estimate itself. Table 1 summarizes the estimated  percentage of contracts of various source selection approaches reviewed.
Based on our analysis of the remaining 129 sample contracts, we estimate  that the total number of best value tradeoff, lowest price technically  acceptable, or sealed bid award decisions (in-scope contracts) in the full  population of interest was about 293. For contracts that utilized a best  value tradeoff process, we categorized them based on the relative  importance placed on price. In addition, we determined contract type, the  type of procurement (product versus services), and the type of product or  service for our sample contracts using FPDS-NG data and verified this  information with the contract documents.
To determine why and how DOD used the best value tradeoff process and,  in particular, when non-cost factors were considered more important than  price, we obtained and reviewed DOD and service level acquisition  guidance related to source selection policies and procedures that describe  how and when the tradeoff process may be used, including those used for  issuing task orders. In addition, for each of the contracts within our  sample, we obtained contract documentation including the acquisition  plan, solicitation, and source selection decision memorandum and  reviewed them in preparation for interviews with DOD officials. In several  cases, the solicitation was unclear as to which type of tradeoff process  was used. In these cases, we relied on the source selection decision  document to categorize the tradeoff process used.
We judgmentally selected buying activities to visit based on factors  including the number of contracts awarded on a best value tradeoff basis,  contract type, and goods or services procured. Buying activities included  at least one command from each military department as well as a defense  agency. We reviewed 27 contracts and 23 task orders through our site  visits. Specifically, we judgmentally selected 23 of 48 task orders for  review by compiling all task orders issued on indefinite delivery/indefinite  quantity contracts obligating over $10 million that were administered by  the officials at the sites we visited. We chose this dollar threshold to  exceed the FAR requirement to provide fair opportunity notices for task  orders valued at $5 million or more. Results from these selected contracts  or task orders cannot be generalized beyond the specific contract  contracts or task orders selected.
During the course of our review, we interviewed officials from the  following commands:    Department of the Army, U.S. Army Corps of Engineers, New Orleans  District Office, Louisiana, and Afghanistan Engineering District, Kabul and  Kandahar, Afghanistan;    Department of the Army, Armament Research, Development and  Engineering Center, Picatinny Arsenal, New Jersey;    Department of the Army, Communications–Electronics Command, Fort  Monmouth, New Jersey;    Department of the Navy, Marine Corps Combat Development Command,    Department of the Navy, Naval Air Systems Command, Patuxent River,  Maryland and Lakehurst, New Jersey;    Department of the Air Force, Air Force Materiel Command, Tinker Air  Force Base, Oklahoma;    Defense Logistics Agency Energy, Ft. Belvoir, Virginia; and    Joint Theater Support Contracting Command, U.S. Central Command,  Kabul and Kandahar, Afghanistan, and Baghdad, Iraq.
We interviewed DOD acquisition and contracting officials to identify their  rationale for the selected source selection approach (e.g., the thought  process behind why a best value approach was chosen over other  approaches). For award decisions that used a best value tradeoff process,  we discussed why the evaluation factors were chosen and how their  relative weights were assigned. We also interviewed officials about the  process used and the underlying rationale when issuing selected task  orders. We also interviewed officials to determine what the expected  outcomes were from using the best value tradeoff process.
We reviewed applicable DOD source selection decision documents and  related memoranda to determine how often DOD paid a price differential,  the amount of the price differential, and the reasons that were given  underlying the decision to pay a higher price. We defined a price  differential as a positive difference in price between the offeror who  received the award and the offeror next in line for award.
To determine what challenges if any, DOD faces in using the best value  tradeoff process, we reviewed DOD guidance and interviewed officials  from Defense Procurement and Acquisition Policy, the military  departments and defense agencies.
We conducted this performance audit from March 2010 through October  2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Timothy DiNapoli, Assistant  Director; William Russell, Katheryn Hubbell, Paige Muegenburg, Jodi  Munson, Anna Russell, Sylvia Schatz, Roxanna Sun, and Bob Swierczek                    made key contributions to this report.