Defense Contracting: Factors DOD Considers When Choosing Best Value Processes Are Consistent with Guidance for Selected Acquisitions
Released: 2014-07-30T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: DOD obligated about $310 billion in fiscal year 2013 for products and services needed to support its mission. To competitively acquire what is needed, DOD may use best value processes—including tradeoff and LPTA—to evaluate vendors' proposals. When using the tradeoff process, DOD weighs the relative importance of price against non-cost factors. By contrast, DOD may use the LPTA process and award the contract based on lowest price once technical requirements are met.

Congress mandated GAO to review DOD's use of best value processes. GAO identified, among other things, (1) the extent to which DOD used best value processes in fiscal year 2013, (2) the factors DOD considers when choosing a source selection process, and (3) training DOD provides to its acquisition personnel on source selection processes.

GAO identified and reviewed solicitations for a projectable sample of 183 contracts out of 2,851 new, competitively awarded contracts that DOD awarded in fiscal year 2013 with obligations over $1 million. GAO also reviewed DOD and military departments' guidance regarding their use of the best value process. GAO selected 16 contracts for case studies based on military department, best value process used, and other factors. GAO reviewed contract documents and interviewed program and contracting officials for these case studies. GAO also reviewed DAU and military departments' training on source selection procedures.

DOD provided technical comments that GAO incorporated as appropriate.

section_title: What GAO Found
paragraphs: The Department of Defense (DOD) used two best value processes—tradeoff and lowest price technically acceptable (LPTA)—for approximately 93 percent of the 2,851 new, competitively awarded contracts awarded in fiscal year 2013 with obligations greater than $1 million. DOD used the tradeoff process most often in GAO's sample of contracts to acquire services, regardless of obligation value. For contracts with higher obligations, DOD used the LPTA process primarily to acquire commercial products, such as fuel. In contrast, for contracts in GAO's sample with lower obligations, DOD used the LPTA process to acquire both products and services. Several contracting and program officials said that their commands gave more attention to whether LPTA is an alternative option in light of declining budgets and efficiency initiatives. For contracts with obligations of $25 million or more, GAO found that DOD increased its use of LPTA since GAO last reported on this issue in October 2010 using fiscal year 2009 data. GAO's prior report did not include contracts with lower obligations.

Source: GAO analysis of DOD contract and solicitation documents. | GAO-14-584

a The 95 percent confidence intervals for estimates in this table are within +/- 8 percentage points of the estimates themselves.

DOD's ability to clearly define its requirements and its knowledge of potential vendors were key factors that underpinned decisions about whether to use tradeoff or LPTA in GAO's 16 case studies. In the eight case studies in which DOD used LPTA, contracting and program officials generally stated that they had sufficient knowledge of the requirements or vendors to feel confident that the lowest priced vendor, meeting DOD's technical requirements, could deliver the product or service. In contrast, in the eight tradeoff case studies, contracting and program officials were less certain about requirements, were looking for innovative solutions, or wanted to use non-cost factors to differentiate vendors. For example, the United States Army Corps of Engineers used technical non-cost factors to evaluate vendors' abilities to use robotics for explosives disposal. These factors are generally consistent with guidance in the Federal Acquisition Regulation and DOD's March 2011 source selection procedures.

DOD, through courses offered by the Defense Acquisition University (DAU) and the military departments, provides both classroom and online training related to source selection processes to its acquisition personnel. Both DAU and military department officials stressed, however, the importance of on-the-job training in preparing personnel to make informed source selection determinations. For example, Naval Facilities Engineering Command officials told GAO that determining when requirements are better suited for tradeoff or LPTA is learned through gaining experience from on-the-job training.

section_title: Letter
paragraphs: The Department of Defense (DOD) obligated about $310 billion in fiscal  year 2013 to acquire products and services needed to support its  missions. DOD has several source selection processes it can use to  evaluate vendors’ proposals. For example, DOD can use a best value  tradeoff process, in which it can vary the relative importance of cost or  price to other factors such as technical capability or past performance in  its solicitations to vendors. In these cases, DOD may award a contract   to other than the lowest-priced vendor if DOD determines that a           higher-priced vendor provides a greater benefit to DOD, and this greater  benefit is worth paying an additional cost, or price differential.  Alternatively, DOD may also use a best value, lowest price technically  acceptable (LPTA) process. In the LPTA process, DOD awards to the  lowest priced vendor that is technically acceptable, and no tradeoffs are  permitted. Proper implementation of these processes is essential to  ensure that DOD lays the foundation for successful acquisition outcomes,  especially at a time of declining spending.

Committee reports from the Senate and House Armed Services  committees and the Joint Explanatory Statement accompanying the  National Defense Authorization Act for Fiscal Year 2014 mandated GAO  to report on DOD’s use of best value processes. GAO determined (1) the  extent to which DOD used best value processes in fiscal year 2013,       (2) the factors DOD considers when choosing a source selection process,  and (3) training DOD provides to its acquisition personnel on source  selection processes. In addition, in response to a matter identified in a  2013 report from the House Armed Services Committee, appendix II  includes information on the military departments’ acquisitions of body  armor vests in fiscal year 2013.

To determine the extent DOD used best value processes in fiscal year  2013, we used the Federal Procurement Data System-Next Generation  (FPDS-NG) to identify new, competitively awarded DOD contracts with  obligations of over $1 million. From a population of 2,851 contracts with  over $1 million in obligations, we selected a random projectable sample of  227 contracts. We then used a $25 million threshold to divide our data set  based on a Defense Federal Acquisition Regulation Supplement  (DFARS) requirement that contracts for products or services with          $25 million or more in estimated total costs for any fiscal year have  written acquisition plans, which contain information on the anticipated  source selection process. For contracts with obligations of $25 million or  more, we compared the percentage of contracts solicited using the best  value processes to fiscal year 2009 data that we reported in October  2010. Our prior report did not include contracts with lower obligations.  Ninety-three contracts from this sample had higher obligations of          $25 million or more and 134 contracts had lower obligations over            $1 million and less than $25 million. We determined that 44 contracts  were either incorrectly coded, such as non-competed contracts that were  identified as competed or contracts that used source selection processes  other than the best value processes. We excluded these contracts, which  resulted in a total of 183 contracts in our sample that we reviewed,  including 86 contracts with obligations of $25 million or more and 97  contracts with obligations over $1 million and less than $25 million. After  accounting for these errors and assessing the reliability of FPDS-NG data  by electronically testing the data to identify problems with consistency,  completeness, or accuracy and reviewing relevant documentation, we  determined that FPDS-NG data were sufficiently reliable for the purposes  of our review. We obtained and analyzed solicitation documents for all  contracts in our sample to identify what source selection process DOD  used. Further, we reviewed our sample of contracts that DOD awarded  using tradeoff to identify how often DOD considered non-cost factors  more important than price, and identified the most frequently used       non-cost factors. We also used FPDS-NG and contract and solicitation  documentation to identify characteristics of the contracts in our sample.

To identify what factors DOD considers when choosing a source selection  process, we analyzed the Federal Acquisition Regulation (FAR), DFARS,  and DOD and military departments’ policy and guidance on source  selection. We supplemented the review of these documents with  interviews of senior DOD policy officials at Defense Procurement and  Acquisition Policy, and at the Army, Navy, and Air Force headquarters.  We also interviewed officials from at least two commands—based upon  such factors as the number of contract actions and obligation amounts— at each military department (Army, Navy, and Air Force), as well as the  Defense Logistics Agency (DLA) to discuss factors affecting their decision  on which source selection process to use. In addition, we selected 16  new, competitively awarded contracts from our sample with obligations  ranging from $1.1 million to $150.7 million in fiscal year 2013 to further  our understanding of why acquisition—including contracting and  program—officials chose a particular source selection process. Our 16  case studies included a selection from each military department and DLA,  different product and service types, and a mix of tradeoff and LPTA  processes. We also interviewed industry associations to identify their  perspectives about DOD’s source selection processes.

To determine what training DOD provides to its acquisition personnel on  source selection processes, we met with Defense Acquisition University  (DAU) officials and instructors and reviewed training materials. We also  obtained workforce data from the DOD Office of the Under Secretary of  Defense (Acquisition, Technology and Logistics), Human Capital  Initiatives. Further, we collected and reviewed DOD and military  departments’ training documents to identify additional source selection  training that is given by them. A more detailed description of our scope  and methodology is included in appendix I.

We conducted this performance audit from September 2013 through July  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: FAR Part 15 allows the use of several competitive source selection  processes to meet agency needs. Within the best value continuum, DOD  may choose a process that it considers the most advantageous to the  government, either the LPTA or the tradeoff process (see figure 1).

DOD may elect to use the LPTA process where the requirement is clearly  defined and the risk of unsuccessful contract performance is minimal. In  such cases, DOD may determine that cost or price should play a  dominant role in the source selection. When using the LPTA process,  DOD specifies its requirements in the solicitation. Contractors submit their  proposals and DOD determines which of the contractors meet or exceed  those requirements, no tradeoffs between cost or price and non-cost  factors are permitted, and the award is made based on the lowest price  technically acceptable proposal submitted to the government.

By contrast, DOD may elect to use a tradeoff process in acquisitions  where the requirement is less definitive, more development work is  required, or the acquisition has a greater performance risk. In these  instances, non-cost evaluation factors, such as technical capabilities or  past performance, may play a dominant role in the source selection  process. Tradeoffs among price and non-cost factors allow DOD to  accept other than the lowest priced proposal. The FAR requires DOD to  state in the solicitation whether all evaluation factors other than cost or  price, when combined, are significantly more important than,  approximately equal to, or significantly less important than cost or price.

In October 2010, we reported that DOD used best value processes for  approximately 95 percent of its new, competitively awarded contracts in  which $25 million or more was obligated in fiscal year 2009. DOD  awarded approximately 26 percent using the LPTA process and 69  percent using the tradeoff process. DOD awarded the remaining 5  percent using sealed bidding, which is a competitive process where  award is made to the responsible bidder whose bid conforms to the  invitations for bid and is most advantageous for the government  considering only price and price-related factors included in the  solicitation. At that time, we found that the majority of the contracts were  awarded using a tradeoff process in which all evaluation factors other  than cost or price, when combined, were significantly more important than  cost or price. Our analysis showed that DOD considered past  performance and technical capability evaluation factors as the most  important among the non-cost factors. Further, we found using a tradeoff  process can be more complex and take more time than other source  selection methods, and requires that acquisition staff have proper  guidance, needed skills, and sound business judgment. While DOD and  the military departments had taken steps to improve source selection  procedures, acquisition personnel noted a lack of training to assist them  in deciding whether or not a price differential is warranted when making  tradeoff decisions. We recommended that to help DOD effectively employ  best value tradeoff processes, DOD develop training elements, such as  case studies, that focus on reaching tradeoff decisions, as it updates its  training curriculum. DOD concurred and implemented the  recommendation in August 2012.

DOD issued new guidance that emphasizes affordability and  standardization of best value processes since our analysis of fiscal year  2009 contracts. In September 2010, the Under Secretary of Defense for  Acquisition, Technology, and Logistics (USD(AT&L)) issued a  memorandum that established its Better Buying Power Initiative to obtain  greater efficiency and productivity in defense spending. In its  memorandum, USD(AT&L) emphasized that DOD must prepare to  continue supporting the warfighter through the acquisition of products and  services in potentially fiscally constrained times. USD(AT&L) noted that  DOD must “do more without more.” In April 2013, USD(AT&L) issued  another memorandum to update the Better Buying Power Initiative. This  memorandum identifies seven areas USD(AT&L) is pursuing to increase  efficiency and productivity in defense spending. One area is incentivizing  productivity and innovation in industry and government. As part of this  guidance, USD(AT&L) states that “best value” in a competitive source  selection should generally indicate that the government is open to paying  more (up to some amount) than the minimum price bid in return for a  product that provides more than the minimum needed performance. In  addition, USD(AT&L) states that LPTA should be used in situations where  DOD would not realize any value from a proposal exceeding its minimum  technical or performance requirements and that another process should  be used when standards of performance and quality are subjective. A  second area of this guidance includes improving the professionalism of  the total acquisition workforce. DOD has previously reported that training  is a critical element of improving and sustaining a high quality workforce  with the right skills and capabilities.

USD(AT&L) also issued source selection procedures in March 2011 to  standardize the methodology and process that DOD uses to conduct  competitively negotiated source selections. For example, USD(AT&L)  outlined a common set of principles and procedures for conducting  acquisitions using the best value processes including the use of  standardized rating criteria and descriptions for technical capability and  past performance factors. Further, similar to information presented in the  Better Buying Power Initiative, USD(AT&L) stated in the procedures that  the LPTA process may be used in situations where the government would  not realize any value from a proposal exceeding minimum technical or  performance requirements, often for acquisitions of commercial or       non-complex services or supplies which are clearly defined and expected  to be low risk. In its April 2013 memorandum updating the Better Buying  Power Initiative, USD(AT&L) directed the director of Defense  Procurement and Acquisition Policy to update the guidance to describe  the characteristics of a technically acceptable solution by July 1, 2013. As  of July 2014, DOD officials are coordinating comments on a draft revision  of the guidance. The Defense Procurement and Acquisition Policy official  in charge of the revision told us the original due date of July 1, 2013 was  established before they decided to do a more comprehensive update of  the guidance, which has contributed to the date slipping for its  completion.

During the time that USD(AT&L) issued these initiatives and guidance— specifically, between fiscal years 2009 and 2013—DOD experienced a  number of changes in its contracting activity, including:

Total obligations for products and services decreased from            $380 billion in fiscal year 2009 to $310 billion in fiscal year 2013,

Obligations on new, competed contracts decreased from $70 billion in  fiscal year 2009 to $43 billion in fiscal year 2013, and

Obligations on new, competed contracts of $25 million or more  decreased from $39 billion in fiscal year 2009 to $24 billion in fiscal  year 2013.

See figure 2 for our analysis of DOD’s contract obligations from  FPDS-NG for fiscal year 2013.

Even though DOD’s obligations decreased between fiscal year 2009 and  2013, it did acquire a similar mix of products and services in both years.  In addition, the percentage of commercial items purchased in those 2  fiscal years was approximately the same.

section_title: DOD Predominately Used Best Value Processes in Fiscal Year 2013, but Increased Its Use of LPTA for Higher Dollar Contracts since Fiscal Year 2009
paragraphs: DOD predominately used best value processes—tradeoff and LPTA—to  evaluate offers from potential vendors in fiscal year 2013. DOD used best  value processes for approximately 93 percent of the 2,851 new,  competed contracts for which it had obligated over $1 million in fiscal year  2013 and used sealed bid for approximately 7 percent. For contracts with  obligations of $25 million or more, DOD used the tradeoff process for  approximately 58 percent of the contracts and the LPTA process for  approximately 36 percent of the contracts. For contracts with obligations  over $1 million and less than $25 million, DOD used tradeoff and LPTA at  about the same overall rate—47 percent and 45 percent, respectively.

In our sample of 171 contracts that used best value processes, DOD  used tradeoff for 96 contracts and LPTA for 75 contracts. We found some  variation in terms of what process was used to acquire products and  services at the different thresholds we reviewed (see figure 3).

As seen in the above figure, DOD used the tradeoff process most often in  our sample to acquire services, including those related to construction  projects, aircraft maintenance, and other support services, regardless of  obligation amount. For contracts with obligations of $25 million or more,  DOD used the LPTA process primarily to acquire commercial products  such as fuel. In contrast, for contracts with obligations over $1 million and  less than $25 million, DOD used the LPTA process to acquire a mix of  products and services, including fuel, aircraft parts, computer equipment,  construction-related services, engineering support services, and ship  maintenance and repairs.

The desire to weigh non-cost factors such as technical approach and past  performance was a key factor cited in the majority of the solicitations  issued for the 96 contracts in our sample that DOD awarded using the  tradeoff process, regardless of obligation value (see table 1).

For the 76 contracts for which non-cost factors were more important than  price, DOD acquired both products and services, such as computer  equipment, aircraft maintenance services, and communication network  support services.and past performance were the factors most often identified as more  important than price among the non-cost factors. For example, 48 out of  the 76 contracts in our sample identified technical approach as the most  important factor. Additionally, 23 out of the 76 contracts in our sample  identified past performance as the most important factor. Other non-cost  factors considered in some of the solicitations with much less frequency  than technical approach and past performance include small business  participation and delivery schedule.

In addition, our analysis found that technical approach  While data on DOD’s use of source selection processes were not readily  available, our analysis found that DOD increased its use of LPTA from  fiscal year 2009 to fiscal year 2013 for contracts with obligations of       $25 million or more (see table 2). We cannot make a comparison  between fiscal year 2009 and fiscal year 2013 for the lower dollar range,  because our prior report only focused on contracts with obligations of   $25 million or more in fiscal year 2009.

Several contracting and program officials said that their commands gave  more attention to whether LPTA is an alternative option in light of  declining budgets and Better Buying Power Initiatives. Further, declining  budgets encouraged contracting and program officials to streamline  requirements. For example:

The Executive Director of Army Contracting Command—Aberdeen  Proving Ground, one of five Army Contracting Command centers— said that overall there is an increased cost consciousness regarding  acquisitions, resulting from the Better Buying Initiatives and declining  budgets. As a part of that increased cost consciousness, there is an  increased willingness and necessity to re-examine tools that could  present better prices. For example, the Executive Director referred to  LPTA as “a tool that has been at the bottom of the source selection  tool box collecting dust for some time.” As it became necessary to  take a look at what is really needed, they have “dusted off” the LPTA  tool and had more discussions about how to set the technical  acceptability at an appropriate level where there is no additional  benefit from paying for more than that level.

Contracting officials from Naval Facilities and Engineering Command  stated that in the current fiscal environment of “doing more with less,”  they are educating their contracting personnel to use LPTA when  appropriate. For example, on March 28, 2013, the Command sent an  email communication to its contracting staff that provided guidance on  the use of LPTA for task orders on multiple award contracts that are  less than $10 million. The guidance stated that the contracting officer  may choose to consider only price or cost for award purposes when  the requirement is valued at less than $10 million, considered to be  non-complex, and where non-cost factors are deemed unnecessary.  These officials stated LPTA is less complex and less time consuming  than tradeoff, and as a result, they can save personnel resources. In  addition to internal guidance, Navy officials told us that the Better  Buying Power Initiative also directs acquisition personnel to look for  efficiencies and streamlining in acquisitions.

Contracting officials from Naval Supply Systems Command stated  they increased their scrutiny on tradeoff acquisitions, which has  contributed to a cultural shift to increase the consideration of LPTA as  an alternative source selection process. The command issued an  October 9, 2012 memorandum to contracting activities that states if  non-cost factors are more important than price, the acquisition must  be reviewed by a senior level acquisition executive.

Similarly, Air Force Materiel Command contracting and program  officials stated that given the budget environment, it is increasingly  difficult to justify higher dollar solutions from a technical standpoint  when solutions may exist that meet the minimum requirement.

DLA contracting officials stated that in light of resource constraints, it  is increasingly common to purchase products that meet the program’s  needs without overstating the requirement. These officials told us  LPTA is a good choice for mature, commercial requirements where  there is no added value in conducting a tradeoff given the need to  stretch budgets.

section_title: Knowledge of Requirements and Potential Vendors Underpin Decisions about Source Selection Process
paragraphs: Our review of contract documents and interviews with program and  contracting officials from our 16 case studies found that for these specific  acquisitions, DOD’s ability to clearly define its requirements and its  knowledge of potential vendors were the key factors that underpinned the  decisions about whether to use tradeoff or LPTA. For example, in the  eight case studies in which DOD used LPTA, DOD contracting and  program officials generally stated they had sufficient knowledge of the  requirements or vendors to feel confident that the lowest priced vendor,  after meeting technical acceptability requirements, could deliver the  product or service. In contrast, in our eight tradeoff case studies,  contracting and program officials were less certain about requirements,  were looking for innovative solutions, or wanted to use non-cost factors,  such as past performance, as a differentiator when selecting the vendor.  We found that for these 16 case studies DOD’s reasons for choosing  LPTA or tradeoff were generally consistent with guidance in the FAR and  DOD’s source selection procedures. Table 3 provides several highlights  from the case studies that illustrate where DOD’s ability to clearly define  its requirements and its knowledge of potential vendors affected the  source selection decision making process.

Policy officials from some military departments noted that setting  technical acceptability levels is important for contracts awarded through  LPTA to be successful. Defense Procurement and Acquisition Policy  officials told us the ongoing efforts to revise DOD’s 2011 source selection  procedures is intended, in part, to further define how to conduct best  value processes. According to these officials, the revised guidance will  emphasize that for LPTA, the solicitation must clearly describe the  minimum evaluation standards. In addition, they expect the guide will  provide additional information on how to determine when to pay a price  premium.

section_title: DOD Provides Online and Classroom Training on Source Selection Processes, but On-the-Job Training Considered Essential for Making Sound Source Selection Decisions
paragraphs: DOD, through courses offered by DAU and the military departments,  provides both classroom and online training related to source selection  processes to its acquisition personnel. Both DAU and military department  officials stressed, however, the importance of on-the-job training in  preparing personnel to make informed source selection determinations.

Congress passed the Defense Acquisition Workforce Improvement Act  (DAWIA) in 1990 to both ensure effective and uniform education, training,  and career development of members of the acquisition workforce,  including contracting and other career fields, and established DAU to  provide training. The act also required DOD to establish career paths,  referred to by DOD as certification requirements, for the acquisition  workforce. DOD military departments must track acquisition workforce  personnel to ensure that they meet mandatory standards established for  level I (basic or entry), level II (intermediate or journeyman), or level III  (advanced or senior) in a career field, such as contracting, life cycle  logistics, and program management. Similar requirements and levels are  established for each of the acquisition career fields identified by DOD.

DOD identified a need to increase the capacity and size of the acquisition  workforce over the past several years. For example, in a DOD  assessment of the contracting workforce completed in September 2008,  senior DOD contracting leaders identified the importance of not only  mastering the “what,” but in using critical thinking and sound judgment to  apply the knowledge—thus mastering the “how” of contracting among its  entry-level and mid-career personnel. To help address concerns that  DOD had become too reliant on contractors to support core functions and  to rebuild the capacity and skill sets that eroded in the years that followed  the downsizing of the workforce in the 1990s, DOD increased its number  of acquisition workforce positions from 133,103 in fiscal year 2009 to  151,355 in fiscal year 2013—including a 9.5 percent increase or an  additional 2,616 positions—in the contracting career field.

DAU officials identified five training courses that are taken either online or  in the classroom to provide acquisition personnel, including contracting  and program officials, the knowledge and skills necessary to make source  selection decisions. Contracting personnel are required or recommended  to complete all five of the identified training courses at some point in their  career to obtain specific DAWIA certifications. Additionally, DAU makes  these classes available to personnel outside the DAWIA acquisition  workforce. Based on our analysis of student self-reported exit data in  fiscal year 2013 and our discussion with DAU officials, we found that  many graduates for these courses did not indicate their career field when  completing the course registration or exit survey, particularly for online  courses, which makes it difficult to know how many personnel outside of  the DAWIA workforce with acquisition-related responsibilities took these  courses. In September 2011, we reported on personnel working on  service acquisitions who are outside the DAWIA acquisition workforce  with acquisition-related responsibilities and found the number of these  individuals to be substantial. As such, we recommended that the  Secretary of Defense establish criteria and a time frame for identifying  personnel outside the DAWIA acquisition workforce with            acquisition-related responsibilities. DOD concurred with the  recommendation and, as of June 2014, is developing a way to identify all  of the non-DAWIA personnel with acquisition-related responsibilities and  the appropriate training curriculum they should receive. Table 4 outlines  each of these five courses.

We also found that military departments provided source selection  training—offering both overview and refresher courses—to contracting  staff and others involved in the source selection process. Table 5  identifies examples of the training courses offered by various military  departments.

DAU and military department officials we spoke with pointed to their  training as providing educational resources from which the acquisition  workforce can understand the basics of appropriate source selection  processes. These officials also stressed the role on-the-job training plays  when making such determinations. For example, policy officials within the  office of the Assistant Secretary of the Army for Acquisition, Technology,  and Logistics told us that on-the-job training provides important exposure  for less experienced acquisition staff to the source selection decision  making processes. As a result, contracting officials have a better  understanding of situations where a particular source selection process  may be more appropriate than others. Many officials told us that  contracting officials can best understand the acquisition process and  apply their in-classroom training through making real world source  selection decisions. As such, several military department officials,  including contracting officials from our case studies, provided examples of  why they consider on-the-job training to be important, including the  following:

Air Force Installation Contracting Agency contracting officials from  one of our case studies and a command official told us that on-the-job  training and experience are important factors that affect the source  selection process determination. They stated that on-the-job training  provides experience and opportunities for contracting officers to make  critical decisions that can only occur in a source selection  environment. To that end, these officials told us that informal  mentoring relationships are established wherein newer, less  experienced staff is assigned to work with more senior staff.

Naval Facilities Engineering Command officials and contracting  officials from one of our case studies stated that the task of identifying  when requirements would better suit a particular source selection  process is learned through gaining experience from on-the-job  training.

Naval Sea Systems Command officials from one of our case studies  stated that the best training they received is on-the-job training. These  officials explained that more senior contracting officers help newer  contracting staff with their acquisitions. They consider mentor type  training invaluable in learning how to conduct an acquisition.

section_title: Concluding Observations
paragraphs: Best value processes continued to underlie the vast majority of DOD’s  new, competitively awarded contracts. DOD has increased its use of the  LPTA process in recent years for higher value contracts, and its decision  making regarding which source selection process to use did not appear to  be ill-advised. Its decision making was generally rooted in knowledge  about the requirements and vendors. In our sample of 16 cases, we  identified instances in which DOD used LPTA for what appeared to be  complex acquisitions, such as the system to mimic an anti-aircraft missile,  but the acquisition team had considerable knowledge about the  requirements or vendors. In other cases, DOD used the tradeoff process  for what appeared to be relatively simple acquisitions, such as fabric  dyeing, yet the acquisition team identified complexities about the  proposed acquisition. Amid the climate of rapidly building fiscal pressures  and cost consciousness, selecting the right source selection approach  continues to be essential to ensure the department acquires what it needs  without paying more than necessary.

section_title: Agency Comments
paragraphs: We are not making recommendations in this report. We provided a draft  of this report to DOD for comment. DOD did not provide written  comments on this report but did provide technical comments, which we  incorporated as appropriate.

We are sending copies of this report to appropriate congressional  committees and the Secretary of Defense. The report will be available at  no charge on GAO’s website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at  (202) 512-4841 or dinapolit@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Committee reports from the Senate and House Armed Services  committees and the Joint Explanatory Statement accompanying the  National Defense Authorization Act for Fiscal Year 2014 mandated GAO  to report on the Department of Defense’s (DOD) use of best value  processes. We determined 1) the extent to which DOD used best value  processes in fiscal year 2013; (2) the factors DOD considers when  choosing a source selection process; and (3) training DOD provides to its  acquisition personnel on source selection processes. In addition, in  response to a matter identified in a 2013 report from the House Armed  Services Committee, appendix II includes information on the military  departments’ acquisitions of body armor vests in fiscal year 2013.

To determine the extent DOD used the best value processes in fiscal year  2013, we used data from the Federal Procurement Data System-Next  Generation (FPDS-NG) as of October 2013 to identify a population of  contracts based on the following criteria: (1) newly awarded by DOD in  fiscal year 2013, (2) competitively awarded, and (3) had obligations of  over $1 million in fiscal year 2013. This analysis identified a population of  2,851 contracts, and from this population we selected a stratified random  sample of 227 contracts, with the strata defined by whether the contract  had obligations of $25 million or more, or whether its obligations totaled  over $1 million and less than $25 million. We divided the data into two  groups including contracts with higher obligations of $25 million or more  and contracts with lower obligations over $1 million and less than          $25 million. We used the $25 million threshold to divide our data set  based on a Defense Federal Acquisition Regulation Supplement  (DFARS) requirement that contracts for products or services with          $25 million or more in estimated total costs for any fiscal year have  written acquisition plans, which contain information on the anticipated  source selection process. more, we compared the percentage of contracts solicited using best value  processes to fiscal year 2009 data we reported in October 2010.prior report did not include contracts with lower obligations of less than  $25 million.

DFARS § 207.103(d)(i)(B).

We obtained and analyzed the solicitation documents for all of the  contracts in our sample to identify the source selection process DOD  used. We verified the contract award fields in FPDS-NG with contract and  solicitation data to ensure that the contracts within our sample were        in-scope. Based on that analysis, we determined that a total of 44  contracts were out of scope for our review. These 44 contracts were  excluded from our analysis, because they were either incorrectly coded in  our key parameters, or were awarded using processes outside of the  Federal Acquisition Regulation (FAR) Part 14 on sealed bidding or Part  15 on contracting by negotiation (which includes best value processes)  and consequently should not have been in our sample, resulting in a total  of 183 contracts in our review (see table 6). After accounting for these  errors, we assessed the reliability of FPDS-NG data by electronically  testing the data to identify problems with consistency, completeness, or  accuracy and reviewed relevant documentation. We determined that the  FPDS-NG data were sufficiently reliable for the purposes of our review.

Because we followed a probability procedure based on random selection,  our sample is only one of a large number of samples that we might have  drawn. Since each sample could have provided different estimates, we  express our confidence in the precision of our particular sample’s results  as a 95 percent confidence interval (e.g., plus or minus 8 percentage  points). This is the interval that would contain the actual population value  for 95 percent of the samples we could have drawn. Unless otherwise  noted, percentage estimates of contracts with obligations of $25 million or  more have 95 percent confidence intervals within +/- 8 percentage points  of the estimate itself. Similarly, for contracts with obligations over            $1 million and less than $25 million, percentage estimates have  confidence intervals within +/- 10 percentage points of the estimate itself.

In addition, to compare characteristics of contracts in our sample that  used best value processes for both strata, we determined contract type,  the type of procurement (product versus service), and if commercial item  procedures were used for our sample using FPDS-NG data and  conducted data reliability analysis on these fields, by verifying this  information with the contract and solicitation documents. For the contracts  identified as tradeoff, we analyzed the contract and solicitation  documentation to identify the most frequently used non-cost evaluation  factors and their relative importance to price.

To identify what factors DOD considers when choosing a source selection  process, we analyzed the FAR, DFARS, and DOD and military  departments’ regulation, policy, and guidance on source selection. We  interviewed senior DOD policy officials at Defense Procurement and  Acquisition Policy, and at the Army, Navy, and Air Force headquarters.  We also interviewed officials from at least two buying commands—based  upon such factors as the number of contract actions and obligation  amounts—at each military department (Army, Navy, and Air Force), as  well as the Defense Logistics Agency (DLA) to discuss factors affecting  their decision process on which source selection process to use. In  addition, we analyzed our sample of 183 contracts and selected 16 new,  competitively awarded contracts with obligations ranging from $1.1 million  to $150.7 million to further our understanding of why acquisition officials  chose the source selection process. Our 16 case studies—8 tradeoff and  8 LPTA—included at least 1 from each military department and DLA,  different product and service types, and amount of dollars obligated in  fiscal year 2013. For the case studies, we interviewed DOD contracting  and program officials and reviewed contract documentation, including the  acquisition plan, solicitation, and source selection decision memorandum  to further understand the source selection decision making process. The  results from our review of these selected contracts cannot be generalized  beyond the specific contracts selected.

During the course of our review, we also interviewed officials from the  following commands:

Department of the Army, Army Contracting Command, Aberdeen  Proving Ground, Maryland; Medical Command, Fort Detrick,  Maryland; and Intelligence and Security Command, Fort Belvoir,  Virginia

Department of the Army, United States Army Corps of Engineers,  Washington, D.C., and Huntsville Center, Alabama

Department of the Navy, Naval Air Systems Command, Patuxent  River, Maryland; Naval Facilities Command, Navy Yard, Washington,  D.C.; and Naval Supply Systems Command, Mechanicsburg,  Pennsylvania

Department of the Navy, United States Marine Corps Installations and  Logistics Command, Navy Annex, Virginia; and Marine Corps  Systems Command, Quantico, Virginia

Department of the Air Force, Installation Contracting Agency and Air  Force Materiel Command, both located at Wright-Patterson Air Force  Base, Ohio

Defense Logistics Agency-Energy, Ft. Belvoir, Virginia; and Defense  Logistics Agency-Troop Support, Philadelphia, Pennsylvania Joint Theater Support Contracting Command, Kabul, Afghanistan.

To determine what training DOD provides to its acquisition personnel on  source selection processes, we met with Defense Acquisition University  (DAU) officials and instructors and reviewed training materials. We also  obtained attendance and workforce data from the DOD Office of the  Under Secretary of Defense (Acquisition, Technology, and Logistics),  Human Capital Initiatives. Further, we collected and reviewed military  department and command specific training documents to identify if  additional source selection training is given in addition to DAU provided  training. We also interviewed DOD policy officials at Defense  Procurement and Acquisition Policy, several commands at the military  departments, as well as contracting and program personnel at the  contracting offices of the selected military departments from the 16 case  studies on training provided related to source selection processes.

We supplemented these case studies with interviews with industry  associations to identify their perspectives about DOD’s source selection  processes.

We conducted this performance audit from September 2013 through July  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Body Armor Vest Acquisitions in Fiscal Year 2013
paragraphs: The Marine Corps, Defense Logistics Agency (DLA), and Army bought  similar soft body armor vests made of ballistic material in fiscal year 2013  using different source selection processes. Knowledge of requirements or  vendors were key considerations in each acquisition, but distinct needs  led to different decisions about which source selection process to use  even when acquiring the similar product.

The Marine Corps issued one delivery order to purchase soft body  armor vests for $2.3 million in fiscal year 2013 using the lowest price  technically acceptable (LPTA) process. It issued this order from a  multiple award, indefinite delivery indefinite quantity (IDIQ) contract  awarded to two vendors in fiscal year 2009 using the LPTA process.  The contracting officer told us they chose to use LPTA for the base  contract because they consider soft body armor vests to be a  commodity product with clearly defined technical performance  specifications. Further, the contracting officer, in consultation with the  program office, saw no opportunity for tradeoff above industry  standard because the industry standard met their current needs.  Ongoing research and development showed that any tradeoff for  enhanced performance would lead to the armor being heavier, an  unacceptable outcome. For the base contracts, the Marine Corps  awarded to the second and third lowest priced vendors, because the  lowest priced vendor was deemed non-responsible.

DLA issued 23 delivery orders to purchase soft body armor vests for  $288.1 million in fiscal year 2013. It issued these orders from three  separate IDIQ contracts awarded to three vendors in fiscal years 2011  and 2012 using the tradeoff process. DLA contracting officials told us  that they chose to use the tradeoff process for these contracts  because they wanted to use past performance as a key discriminator,  which is generally not allowed using the LPTA process. Further,  because DLA buys for sustainment purposes and its quantity needs  fluctuate, officials told us that past performance was a critical  determination factor requiring the use of the tradeoff process, in  addition to the vendor’s historic production capacity, delivery  schedule, and other performance capabilities.

The Army issued one delivery order to purchase soft body armor  vests for $10,201 in fiscal year 2013 using the LPTA process. It  issued this order from one of the multiple award, IDIQ contracts  awarded to eight vendors in fiscal years 2009 and 2010 using the  tradeoff process. Army contracting officials told us they chose to use  the tradeoff process for the base contract, because it provided the  Army more discretion in evaluating past performance as well as  leaving open the possibility that industry vendors might offer a more  innovative solution. Once the Army had a group of qualified vendors  on contract, they could then use the LPTA process for subsequent  buys.

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, the following staff members made  key contributions to this report: Molly Traci, Assistant Director; James  Kim; Anh Nguyen; Erin Stockdale; Jina Yu; Claire Li; Jessica Drucker;  Danielle Greene; Roxanna Sun; John Krump; Mark Ramage; Julia  Kennon; Virginia Chanley; and Carol Petersen.


Question-summary annotations: questions: question: What were tradeoff and LPTA processes used for?
answer: The Department of Defense (DOD) used two best value processes—tradeoff and lowest price technically acceptable (LPTA)—for approximately 93 percent of the 2,851 new, competitively awarded contracts awarded in fiscal year 2013 with obligations greater than $1 million.
child_questions: question: What did GAO find about DOD's use of tradeoff processes?
answer: DOD used the tradeoff process most often in GAO's sample of contracts to acquire services, regardless of obligation value.

question: When was the LPTA process used?
answer: For contracts with higher obligations, DOD used the LPTA process primarily to acquire commercial products, such as fuel. In contrast, for contracts in GAO's sample with lower obligations, DOD used the LPTA process to acquire both products and services.
child_questions: question: What did interviewees report about the LPTA process?
answer: Several contracting and program officials said that their commands gave more attention to whether LPTA is an alternative option in light of declining budgets and efficiency initiatives.

question: What did GAO find about higher-budget contracts?
answer: For contracts with obligations of $25 million or more, GAO found that DOD increased its use of LPTA since GAO last reported on this issue in October 2010 using fiscal year 2009 data. GAO's prior report did not include contracts with lower obligations.

questions: question: What influenced the choice between tradeoff or LPTA?
answer: DOD's ability to clearly define its requirements and its knowledge of potential vendors were key factors that underpinned decisions about whether to use tradeoff or LPTA in GAO's 16 case studies.
child_questions: question: What influenced the choice to use LPTA?
answer: In the eight case studies in which DOD used LPTA, contracting and program officials generally stated that they had sufficient knowledge of the requirements or vendors to feel confident that the lowest priced vendor, meeting DOD's technical requirements, could deliver the product or service.

question: What characterized the use of tradeoff?
answer: In contrast, in the eight tradeoff case studies, contracting and program officials were less certain about requirements, were looking for innovative solutions, or wanted to use non-cost factors to differentiate vendors.
child_questions: question: What is an example of such a situation?
answer: For example, the United States Army Corps of Engineers used technical non-cost factors to evaluate vendors' abilities to use robotics for explosives disposal.

question: To what extent are the factors of decisions in compliance with DOD procedures?
answer: These factors are generally consistent with guidance in the Federal Acquisition Regulation and DOD's March 2011 source selection procedures.

questions: question: What training does the DOD provide to acquisition personnel?
answer: DOD, through courses offered by the Defense Acquisition University (DAU) and the military departments, provides both classroom and online training related to source selection processes to its acquisition personnel.

question: What other training for acquisition personnel is important?
answer: Both DAU and military department officials stressed, however, the importance of on-the-job training in preparing personnel to make informed source selection determinations.
child_questions: question: What is one situation for which this type of training is better suited?
answer: For example, Naval Facilities Engineering Command officials told GAO that determining when requirements are better suited for tradeoff or LPTA is learned through gaining experience from on-the-job training.

questions: question: What fiscal support did DOD give its central products and services?
answer: DOD obligated about $310 billion in fiscal year 2013 for products and services needed to support its mission.

question: How does DOD need to manage their spending?
answer: To competitively acquire what is needed, DOD may use best value processes—including tradeoff and LPTA—to evaluate vendors' proposals.
child_questions: question: What is the tradeoff process?
answer: When using the tradeoff process, DOD weighs the relative importance of price against non-cost factors.

question: What is the LPTA process?
answer: By contrast, DOD may use the LPTA process and award the contract based on lowest price once technical requirements are met.

questions: question: What solicitations did GAO review?
answer: GAO identified and reviewed solicitations for a projectable sample of 183 contracts out of 2,851 new, competitively awarded contracts that DOD awarded in fiscal year 2013 with obligations over $1 million.

question: What DOD process did the GAO review?
answer: GAO also reviewed DOD and military departments' guidance regarding their use of the best value process.
child_questions: question: What material did the GAO use to do so?
answer: GAO selected 16 contracts for case studies based on military department, best value process used, and other factors. GAO reviewed contract documents and interviewed program and contracting officials for these case studies.

question: What training did GAO review?
answer: GAO also reviewed DAU and military departments' training on source selection procedures.