Defense Acquisition Workforce: DOD's Training Program Demonstrates Many Attributes of Effectiveness, but Improvement Is Needed
Published: 2010-10-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The President has announced his intention to improve the acquisition process, particularly given the half a trillion dollars the federal government spent in fiscal year 2009 on acquiring goods and services. The Department of Defense (DOD) spent $384 billion in fiscal year 2009 on goods and services--double what it spent in 2001. A high-quality workforce with the right competencies and skill sets will be critical to improving DOD acquisitions. GAO was mandated to determine the efficacy of DOD's certification training for its acquisition workforce. GAO assessed (1) DOD's capability to provide certification training, (2) the extent that such training reaches members of the workforce, and (3) the extent that previous training recommendations have been implemented. To conduct this work, GAO compared DOD's certification training to GAO guidance for effective training programs and analyzed policies, data, and previous reports on acquisition training.
section_title: What GAO Found
paragraphs: DOD's certification training program--provided by the Defense Acquisition University (DAU)--generally demonstrates the capability to provide effective training, though some attributes of an effective training program are lacking. DAU ensures that strategic and tactical changes are promptly incorporated into training; uses centralized and decentralized training approaches in design and development; collects data during implementation to ensure feedback on its training programs; and analyzes its training during evaluation. However, DOD lacks complete information on the skill sets of the current acquisition workforce and does not have outcome-based metrics to assess results achieved in enhancing workforce proficiency and capability through training efforts. In 2009, GAO recommended that the Secretary of Defense identify and update on an ongoing basis the number and skill sets of the total acquisition workforce--including civilian, military, and contractor personnel--that the department needs to fulfill its mission. DOD agreed and to date has completed about one-fifth of its workforce competency assessments. At the end of fiscal year 2009, 90 percent of DOD's acquisition workforce personnel had completed required certification training or were within required time frames to do so, according to DAU data. However, DAU reports that it cannot provide for all training requested for the entire acquisition workforce. DAU has offered more courses in recent years, and high-priority personnel--those needing to complete classes for certification in their current position--constitute the majority in DAU classes. DAU plans the number and location of its classes based on data that DOD officials noted are generally incomplete when submitted, and DAU must adapt during the year to support new requirements as they are identified. DAU has identified the need for a new, integrated student information system that will provide better insight into the workforce it supports and is in the early stages of its procurement. DOD reports that most of the training-related recommendations from previous reviews--the Gansler Commission, the Panel on Contracting Integrity, and a prior GAO report--have been fully implemented and some actions are still under way. DOD has either fully or partially implemented 15 of the 19 recommendations GAO reviewed. Both the Army and the Office of the Secretary of Defense have taken steps to respond to the Gansler Commission recommendations. Most of the recommendations made by the Panel on Contracting Integrity have been implemented, with the exception of two recommendations related to assessing guidance and reviewing a specific training topic. GAO made four recommendations pertaining to the Defense Contract Audit Agency's government auditing standards training and expertise, of which one has been partially implemented and three have not been implemented, but some actions have been taken.
section_title: What GAO Recommends
paragraphs: GAO recommends DOD establish milestones for developing metrics to measure how certification training improves acquisition workforce capability and a time frame for acquiring and implementing an integrated information system. DOD concurred with the second but not the first recommendation. GAO continues to believe DOD needs to develop additional metrics.

section_title: Letter
paragraphs: The President has announced the administration’s intention to achieve  improvements in the acquisition system. Current fiscal challenges  highlight the need to scrutinize the processes used by the federal  government to spend half a trillion dollars in fiscal year 2009 acquiring  goods and services. The Department of Defense’s (DOD) spending on  goods and services has more than doubled since 2001 to nearly               $384 billion; however, the number of acquisition personnel has remained  relatively stable. According to the Office of the Under Secretary of Defense  for Acquisition, Technology and Logistics (USD(AT&L)), to successfully  accomplish the acquisition mission and ensure the best value for the  expenditure of public resources, DOD will place greater emphasis on  having a high-quality workforce with the right competencies and skill sets.  In addition to building the capability of the existing workforce, the  Secretary of Defense plans to augment the capacity of the defense  acquisition workforce by increasing its numbers by 20,000 employees over  the next 5 years.
In 1990 the Defense Acquisition Workforce Improvement Act (DAWIA)  established career paths for the improvement of training for the  acquisition workforce. Given the enduring critical role of the acquisition  workforce in achieving the administration’s goal of improving federal  acquisition, it is important they have the skills and support needed to do  their jobs. Accordingly, in the National Defense Authorization Act for  Fiscal Year 2010, Congress mandated that GAO study the efficacy of  DOD’s training for its acquisition and audit workforce. As agreed with  your respective offices, we assessed (1) DOD’s capability to provide  defense acquisition workforce certification training, (2) the extent that  such training reaches members of DOD’s acquisition workforce, and        (3) the extent that training recommendations from previous reviews,  including the Gansler Commission, have been implemented.
To determine DOD’s capability to provide acquisition certification training,  we focused on training for DOD personnel covered under DAWIA. We  compared DOD’s certification training programs and processes with the  attributes of effective training and development programs identified in our  previous work on strategic training and development efforts in the federal  government. We interviewed officials at the Defense Acquisition  University (DAU) and Defense Contract Audit Institute (DCAI) to obtain  an understanding of their training programs and processes, and we  obtained documents—such as briefings, guidance, strategic plans, and  course catalogs—describing the training programs and processes. In  addition, we selected a nongeneralizable sample of one command from  each military service near a DAU regional office—based on level of  procurement dollars spent and number of DAU courses completed in  fiscal years 2008 through 2009—and program offices within those  commands that would likely have a large cross-section of acquisition  workforce personnel to obtain customer perspectives on the effectiveness  and usefulness of the DAU training and to determine the use of  supplementary training. We also interviewed Defense Contract  Management Agency officials to obtain their views on DAU training.  Finally, we visited a nongeneralizable sample of two Defense Contract  Audit Agency (DCAA) locations to obtain customer perspectives on DCAI  training. We did not examine the appropriateness of the certification  training itself nor the content of courses required for certification.
To assess the extent to which acquisition training reaches acquisition  personnel, we reviewed DAU and DCAI policies on determining training  requirements, allocating resources, and scheduling classes. We collected  and analyzed defense acquisition workforce and training data maintained  in the AT&L Data Mart system used by DAU to determine course demand  and certification status of acquisition workforce members. This provided  an understanding of the number of class requests received, class seats  scheduled, and students who registered and completed these courses in  past fiscal years. We assessed the reliability of these data by reviewing  data query information and interviewing knowledgeable officials who  collect and use these data. We determined that the data were sufficiently  reliable for the purposes of this report. In addition, we interviewed DAU  officials and obtained budget documents to determine DOD’s use of the  Defense Acquisition Workforce Development Fund for training.
To determine the extent to which training recommendations from  previous reviews have been implemented, we identified previous reviews  and interviewed and obtained documentation from agency officials on the  status of DOD’s implementation of the training recommendations.
We conducted this performance audit from December 2009 to September  2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. See appendix I for  additional details on our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Congress passed DAWIA in 1990 to ensure effective and uniform  education, training, and career development of members of the acquisition  workforce. Accordingly, the act established DAU to provide training for  the DOD acquisition workforce and charged DOD officials with  designating acquisition positions, setting qualification requirements, and  establishing policies and procedures for training the acquisition  workforce. DOD, as part of implementing DAWIA, established career  fields, such as program management (See table 1).
The act also required DOD to establish career paths, referred to by DOD as  certification requirements, for the acquisition workforce. DOD military  services and defense agencies must track that acquisition workforce  members meet mandatory standards established for level I (basic or  entry), level II (intermediate or journeyman), or level III (advanced or  senior) in a career field, such as contracting, life-cycle logistics, and  program management. DAU is responsible for certification training and for  designing, maintaining, and overseeing the delivery of certification training  courses at each level, among other things. For each career field and level,  there are requirements in three areas—education, experience, and  training. Certification requirements are the same for civilian and military  acquisition workforce members. Table 2 shows the nature of certification  training for one of the DAWIA career fields—system planning, research,  development, and engineering (SPRDE)—systems engineering, as well as  shows the education and experience requirements for each level in the  career field.
Besides the certification training it offers, DAU approves alternative  certification training providers based on a review by an independent  organization—the American Council on Education—of the capability of a  potential provider to offer acquisition training and whether the provider’s  course content addresses the DAU course’s learning outcomes. An  equivalent course provider must certify annually that its course is current  with the DAU plan of instruction for the course. Similarly, DCAI provides  both required certification training and supplemental training for the  auditor career field.
In addition to certification training, DAU offers supplemental training for  each career field and for particular types of assignments. For example, for  level II contracting in contingency or combat operations, DAU provides  courses such as a contingency contracting simulation, a contingency  contracting officer refresher, and a joint contingency contracting course.  DAU also provides continuous learning modules online to provide  acquisition workforce members with a quick reference for material  already introduced and courses to help them maintain currency in their  career field by achieving the required 80 continuous learning points  biennially. Additionally, DAU provides consulting support to program  offices, rapid-deployment training on new initiatives, and training targeted  to the needs of acquisition field organizations. DAU also engages in  knowledge-sharing initiatives, including hosting a number of acquisition  communities of practice and providing Web-based acquisition policy and  reference materials.
In March 2004, we issued a guide for assessing federal training programs  that breaks the training and development process into four broad,  interrelated components—(1) planning and front-end analysis, (2) design  and development, (3) implementation, and (4) evaluation. The guide  discusses attributes of effective training and development programs that  should be present in each of the components and identifies practices that  indicate the presence of the attribute. For example, under the design and  development component, to determine whether an organization possesses  the attribute of incorporating measures of effectiveness into courses it  designs, the guide suggests looking for practices, such as (1) clear linkages  between specific learning objectives and organizational results and           (2) well-written learning objectives that are unambiguous, achievable, and  measurable. For a complete list of the attributes of effective training and  development programs, see appendix II.
Figure 1 depicts the training and development process along with the  general relationships between the four components that help to produce a  strategic approach to federal agencies’ training and development  programs. These components are not mutually exclusive and encompass  subcomponents that may blend with one another. Evaluation, for example,  should occur throughout the process.
section_title: DOD’s Certification Training Program Has the Capability to Provide Training, Although It Lacks Some Attributes of Effective Training
paragraphs: DOD’s acquisition workforce certification training—-centrally  administered by DAU—has many attributes of effective training programs  that demonstrate the capability to deliver training. DAU’s certification  training program has a formal process in planning and front-end analysis  to ensure that strategic and tactical changes are promptly incorporated  into training; use of centralized and decentralized training approaches in  design and development; data collection during implementation to ensure  feedback on its training programs; and appropriate analytical approaches  to assess its training during evaluation. However, DOD lacks complete  information on the skill sets of the current acquisition workforce for  planning and front-end analysis and does not have metrics to assess  results achieved in enhancing workforce proficiency and capability  through training efforts during evaluation. Complete data on acquisition  skill sets are needed to accurately identify workforce gaps, and  appropriate metrics are necessary to increase the likelihood that desired  changes will occur in the acquisition workforce’s skills, knowledge,  abilities, attitudes, or behaviors.
subsections: section_title: DOD’s Certification Training Program Has Many Attributes of Effective Training Programs
paragraphs: DOD’s certification training program possesses attributes of effective  training programs in each of the four components of the training and  development process. Following are examples of the attributes we  observed in DOD training categorized by the components of effective  training programs.
Planning and front-end analysis: Planning and front-end analysis can help  ensure that training efforts are not initiated in an ad hoc, uncoordinated  manner, but rather are strategically focused on improving performance  toward the agency’s goals. DAU had processes to ensure that training  efforts were coordinated and focused on improving agency goals. Through  a formal process that ensures that strategic and tactical changes are  promptly incorporated into training, DAU and other DOD stakeholders  plan for and evaluate the effectiveness of DAU’s training efforts. Each  career field has a functional leader, a senior subject-matter expert in the  career field who is responsible for annually certifying that course content  for certification is current, technically accurate, and consistent with DOD  acquisition policies. Functional leaders are supported by a functional  integrated process team for each career field, which consists of subject- matter experts, acquisition career management representatives from the  military services and other DOD agencies, and DAU representatives. The  functional integrated process team analyzes and reviews data, including  end-of-course evaluations, number of students completing a class, wait  lists, and certification rates, as well as DOD policy changes and  recommendations from reviews, such as the Gansler Commission to  support functional leaders. DAU designs courses in accordance with the  functional leader and functional integrated process team decisions. Using  this process, strategic and tactical changes were promptly incorporated  into training. For example, DAU developed and fielded a new contracting  course on federal acquisition regulation fundamentals within a year of  direction by the functional leader’s organization to create it.
Design and development: In design and development, it is important that  agencies consciously consider the advantages and disadvantages of using  centralized and decentralized approaches. Centralizing design can  enhance consistency of training content and offer potential cost savings.  DAU evaluates and uses centralized and decentralized approaches for  training after considering the advantages and disadvantages. DAU’s  curriculum development and technologies organizations located at Fort  Belvoir, Virginia, provide centralized, integrated design and development  of certification courses. These courses are then delivered to the  acquisition workforce by five regionally-oriented campuses and the  Defense Systems Management College School of Program Managers. DAU  also compares training delivery mechanisms to determine the appropriate  use of different delivery mechanisms (such as classroom or computer- based training) and to ensure efficient and cost-effective delivery. In  addition, supplementary training is offered at the Army, Navy, and Air  Force commands and program offices we visited, as well as at the Defense  Contract Management Agency. While DAU provides a foundation for  acquisition and career field knowledge in its certification training, various  decentralized sources provide supplementary training more targeted to  specific jobs, such as training on service-specific processes or databases  and technical topics. Acquisition workforce members at the commands we  visited provided the following examples of supplementary training.
The contracting offices at the Army Aviation and Missile Command  (AMCOM), Alabama, and the Air Force Aeronautical Systems Center  (ASC), Ohio, provided unique training in the contracting area. AMCOM’s  Contracting Center University teaches employees how to do day-to-day  tasks associated with their job, such as price analysis, price negotiation,  and how to use the Army Materiel Command-unique system for preparing  contract documents. ASC’s “jump start” program teaches, reinforces, and  supplements DAU certification training in the contracting career field with  illustrative examples not provided in the computer-based contracting  courses as well as offers an opportunity to interact with instructors and  other students.
The Naval Air Systems Command (NAVAIR), Maryland, provides  supplementary training for DAWIA career fields. For example, in the  program management career field, NAVAIR offers courses in configuration  management and on NAVAIR’s technical directives system.
Other acquisition workforce members provided examples of training from  other federal agencies or commercial vendors, such as financial training  from the Graduate School, United States Department of Agriculture, and  Management Concepts, while others said they had brown bag lunches on  various topics.
Figure 2 below identifies DOD’s multifaceted training approach, both  centralized and decentralized. The objective of the multifaceted training,  in conjunction with the other two certification components—education  and experience— is acquisition personnel with the training, education, and  experience to perform the acquisition job.
Implementation/Evaluation: As with other programs and services that  agencies deliver, it is important that agencies collect program performance  data during implementation and select an analytical approach that best  measures the program’s effect to evaluate their training and development  efforts. DAU collects customer feedback data during implementation and,  during evaluation, uses the four-level Kirkpatrick model as an analytical  approach for measuring training effectiveness. As a part of evaluating  training, DAU conducts student end-of-course surveys (Level 1-Reaction)  and, to a lesser degree, follow-up surveys of students and their managers  60 and 120 days, respectively, after course completion (Level 3-Behavior).  DAU tracks the scores from the various surveys by survey section, such as  job impact, and uses red-yellow-green stoplight indicators to identify areas  of concern overall and by specific courses. DAU also administers pre- and  post-training tests to measure learning (Level 2-Learning). To measure  organizational impact (Level 4-Business Results), DAU employs measures  of efficiency in evaluating and analyzing multiyear data, such as number of  students completing courses, cost efficiency, and customer satisfaction  trends. Level 4 assessments are resource intensive and have not been  extensively used by DAU.
section_title: Some Attributes of DOD’s Certification Training Programs are Inadequate
paragraphs: DOD is deficient in two attributes of an effective training program— determining the skills and competencies of its workforce for planning and  front-end analysis and using performance data to assess the results  achieved through training efforts during evaluation. In March 2009, we  reported that USD(AT&L) lacks complete information on the skill sets of  the current acquisition workforce and whether these skill sets are  sufficient to accomplish DOD’s missions. We recommended and DOD  agreed to identify and update on an ongoing basis the number and skill  sets of the total acquisition workforce—including civilian, military, and  contractor personnel—that the department needs to fulfill its mission.  Complete data on skill sets are needed to accurately identify its workforce  gaps. Not having these data limits DOD’s ability to make informed  workforce allocation decisions. We reported that USD(AT&L) was  conducting a competency assessment to identify the skill sets of its  current acquisition workforce, but also found that the lack of key data on  the in-house acquisition workforce identified in the prior report still exists,  though progress has been made. Since we released that report, DOD  issued its Strategic Human Capital Plan Update in April 2010. According  to DOD, progress was made in completing over 22,000 assessments  involving 3 of the 15 career fields—program management, life-cycle  logistics, and contracting career fields. The assessments completed to date  represent approximately one-fifth of the personnel and career fields.
Although DAU uses performance data—including customer feedback,  number of students completing classes, and cost—to assess the results  achieved through training efforts during evaluation, USD(AT&L) has only  partially established metrics required in 2005 by its own guidance to  provide senior leaders with appropriate oversight and accountability for  management and career development of the acquisition workforce. The  purpose of these metrics is to help DOD ensure a sufficient pool of highly  qualified individuals for acquisition positions and, therefore, relates to the  knowledge, skills, abilities, and size of the acquisition workforce, while the  DAU performance data measure the performance of DAU against its goals.  By incorporating these metrics into the training and development  programs they offer, DOD can better ensure that they adequately address  training objectives and thereby increase the likelihood that desired  changes will occur in the acquisition workforce’s skills, knowledge,  abilities, attitudes, or behaviors. AT&L programs lacking appropriate  outcome metrics will be unable to demonstrate how the certification  training contributes to organizational performance results. According to  USD(AT&L)’s Deputy Director for Human Capital Initiatives, DOD has  established some metrics to measure the size of the acquisition workforce  that partially satisfy the requirements identified in DOD Instruction  5000.66. For example, DOD measures the cumulative number of civilian  and military acquisition positions added as a result of in-sourcing  acquisition functions performed by contractors. However, for metrics  related to acquisition workforce proficiency and capability, there are no  discernable targets, except improvement over the previous year. In  addition, DOD’s April 2010 Strategic Human Capital Plan Update  identified an initiative to establish certification goals as a management tool  for improving workforce quality by June 10, 2010. According to the Deputy  Director, certification goals are being discussed but they had not been  established at the time of this report.
section_title: Despite Incomplete Data and High Demand, Most Workforce Members Receive Training Required for Certification
paragraphs: Although DAU is unable to provide all training requested for acquisition  workforce personnel and receives incomplete data for planning its training  schedule, most personnel who need required DAWIA certification training  receive it within required time frames. DAU plans the number and location  of its classes based on data submitted by the Directors of Acquisition  Career Management (DACM). However, DOD acquisition and training  officials noted that data are generally incomplete when submitted and  additional steps must be taken during the year to meet new requirements  as they are identified. DAU has identified the need for an integrated  student information system to improve the quality of the data and to  provide greater insight into the workforce it supports. Additionally, though  the number of DAU course graduates has grown over the past 5 years,  DAU has not been able to provide enough class seats to meet the training  requirements reported by military departments and defense components.
subsections: section_title: DAU Receives Incomplete Information to Develop an Annual Teaching Schedule
paragraphs: DAU receives annual DACM data submissions for the course scheduling  process, but the submissions do not provide the exact information needed  to determine training demand for the acquisition workforce. DAU receives  class requirements data annually from the DACM offices that it uses when  developing course schedules to identify the number and location of DAU  courses. DACM offices compile this information for all offices to establish  the overall demand for each military department and the defense agencies  for each DAU course. DAU and DACM offices work together throughout  the process to improve the accuracy of this information when possible.  According to DAU and DACM officials, however, data that are transmitted  for schedule development do not fully reflect all demand for the upcoming  year as new requirements arise once the schedule is developed. As a result  additional planning and coordination between DAU and DACM offices is  necessary to meet the training requirements of the acquisition workforce.  For example, in fiscal year 2009, DAU received requests for 142 additional  classes outside of the normal scheduling process. DAU was able to  support 45 of these requested classes in such areas as program  management, contracting, business management, and logistics. According  to DAU officials, resources for additional classes are made available when  other classes are cancelled. Also, DAU may reallocate allotted classroom  seats among departments and agencies to fill additional training needs.  DAU officials stated that data on selected acquisition support services that  are currently performed by contractors who may transition to in-house  DOD personnel are not adequate for planning specific training  requirements. Though DOD has established goals for the number of  contracted personnel to be converted, DAU officials noted that the exact  time or training backgrounds of the personnel are not known in advance.
DAU also uses acquisition workforce data provided quarterly by the  DACM offices that include information such as the number of personnel in  each acquisition career field as well as the career level, job titles, and  status of progress against certification requirements of each workforce  member to inform course demand management. According to DAU  officials, these data provide a snapshot of the acquisition workforce and  certification status, and they use this information to estimate the number,  location, and type of classes needed by the acquisition workforce for  certification. The data are compiled to create a demand management tool  that provides DAU with an imprecise estimate of course requirements and  are used to supplement and inform the estimates developed during the  scheduling process. However, this demand management tool alone cannot  be used by DAU to determine the exact number of classroom seats  required each fiscal year. According to DAU officials, the workforce data  collected may overstate training requirements because it does not account  for training that has already been completed when individuals held a  previous acquisition position, nor does it discern between multiple classes  that may fulfill the same training requirement.
Citing incomplete data for scheduling, as well as other deficiencies, DAU  has taken steps to procure a student information system that will improve  insight into and enhanced management of the defense acquisition  workforce’s training needs. DAU began its market research for an  integrated student information system in December 2007, viewed vendor  presentations and demonstrations throughout 2008, and issued a request  for proposal in August 2010. In the request for proposal, DAU identified  the need for an integrated system for registration, student services, career  management, schedule management, catalog requirements, recording  transcripts, and reporting intended to improve its management of training  needs and schedules. Without an integrated system, DAU states that it will  remain reliant on a web of decentralized information that makes reporting  and trend analysis difficult and time-consuming. A primary goal of the new  system is to provide a comprehensive approach that improves, among  other things, tracking of certification status and ensures training reaches  the right people at the right time. DAU plans call for the contractor to  complete implementation of this new student information system              24 months after the date of contract award, which had not been made as of  September 2010.
DCAI develops its training schedule based on the requirements expressed  in the individual development plans and availability of DCAI resources.  Registration for DCAI courses is prescribed and based largely on the  individual developmental plans submitted by DCAA’s approximately 3,700- member auditing workforce in fiscal year 2009. Each year DCAA  employees develop an individual development plan that lists DCAI courses  as well as outside training deemed necessary with the input and approval  of their supervisors. This information is input into a system that tracks  course requirements and individuals’ status against training requirements.  Individuals are automatically enrolled into the scheduled DCAI courses.
section_title: Personnel Receive Required Training within Required Time frames According to DAU Data, However, DAU Cannot Support All Training Requirements for the Defense Acquisition Workforce
paragraphs: Most of the acquisition workforce receives training within required  certification time frames. At the end of fiscal year 2009, approximately      90 percent of the 133,103 members of the defense acquisition workforce  had met certification requirements associated with their position or were  within allowed time frames to do so. Acquisition workforce members we  met with from all three military departments and the Defense Contract  Management Agency (DCMA) noted challenges for receiving training at  the time and location they desired, noting that local DAU locations would  fill up quickly and that they would often have to register for courses  multiple times prior to enrollment. However, acquisition staff and  supervisors told us that this had little effect on being certified within the  required time frames for their current positions. Nearly all of the  remaining uncertified personnel required training to become certified.  While additional training was needed, these individuals may also have  been deficient in meeting education or experience requirements also  needed for DAWIA certification. Furthermore, DACM officials noted that  there could be a number of reasons these individuals had not received  required training and stated that while some individuals may not have  adequately planned for their training needs, other factors, such as  deployment of military personnel abroad, may have limited their access to  training.
DCAA auditors do not face the same issues with DAWIA certification as  the rest of the acquisition workforce. According to DCAI officials, this is  largely because they do not have to coordinate demand for courses across  several different agencies. All new hires are automatically enrolled in the  courses required for level I and level II DAWIA certification. Additionally,  DOD reported that approximately 99 percent of the auditing workforce  had met certification requirements or were within allowed time frames to  do so. By completing the mandatory learning track taught through DCAI  classes, DCAA auditors complete certification training within required  time frames.
Even though 90 percent of the acquisition personnel who required  certification training for their current position received training on time or  were within allowed time frames to do so, DAU acknowledges that  requests for acquisition workforce training as a whole submitted by the  DOD components and military departments exceed what DAU can  provide. DAU has incorporated expansion of training into its strategic  plans. In its Strategic Plan for 2010-2015, DAU notes that it will play a key  role in the USD(AT&L) acquisition workforce growth strategy. For  example, USD(AT&L) efforts to grow, train, and develop the defense  acquisition workforce will affect DAU’s strategic planning over the next  several years. DAU notes that workforce growth goals put forth by the  Secretary of Defense in April 2009 will increase the demand for DAU  training and therefore affect how DAU plans for development of  acquisition personnel, requiring careful consideration of resource  allocation. The strategic plan also points out a number of other factors  that will drive the demand for acquisition workforce training in the coming  years, including annual workforce turnover, turnover related to Base  Realignment and Closure, and support for new acquisition development  needs. As part of its strategy, DAU has also established short-term goals to  expand training capacity in its fiscal year 2010 Organizational Performance  Plan, including expanding classroom training by 10,000 seats over fiscal  year 2009 levels. DAU officials stated that they plan to increase capacity  further to provide 54,000 classroom seats in fiscal year 2011. In addition,  DAU established and has fulfilled a strategic goal of graduating 150,000  students from its Web-based courses annually.
DAU has increased the total number of course graduates and classes in  recent years to address demand for acquisition training. DAU has  supported more classes than in the past, seeing an increase from 1,279  classroom courses in fiscal year 2005 to 1,505 in fiscal year 2009. In  addition, from fiscal year 2005 through 2009, the number of individual  graduates from DAU classroom and Web-based courses rose by  approximately 77 percent (see fig. 3).
To support increases in certification training demand due to workforce  growth through new hiring and in-sourcing, DAU uses funding from the  Defense Acquisition Workforce Development Fund to provide additional  facilities and courses. Though the majority of funding is intended to  support the hiring of new staff, DAU, military departments, and defense  agencies received more than $225 million dollars to support new training  and additional seats in fiscal years 2008 and 2009. Funds have been used  by the military departments to support Army and Navy acquisition boot  camps, the Air Force’s mission-ready contracting course, and other  acquisition training developed by specific military commands. For  example, funding was used to develop and implement the “jump start”  program at the Air Force’s Aeronautical Systems Center that combines  material taught through DAU’s Level I contracting courses with Air Force- specific information. The Defense Acquisition Workforce Development  Fund has also been used by DAU to expand its teaching facilities, hire  additional instructors, and schedule additional classes needed for DAWIA  certification. DAU received nearly $165 million in fiscal years 2008 and  2009 to expand training. In fiscal year 2009, this funding permitted DAU to  offer nearly 7,000 additional classroom seats in 31 different courses. DAU  has also used these funds to develop new training—such as a 4-week  course focusing on the Federal Acquisition Regulation that senior DOD  contracting officials said was needed to provide a foundation for  acquisition fundamentals—and to support acquisition professionals in the  field through Service Acquisition Workshops and expanded contingency  acquisition training.
Despite these increased class offerings that have accommodated more  graduates, DAU has not been able to provide the total number of  classroom seats that are requested by the defense acquisition workforce  through the DACMs. Classroom seats requested and class seats scheduled  both increased from fiscal year 2007 through 2009. For example, in fiscal  year 2009, DOD components requested 52,998 seats for the acquisition  workforce across 66 different DAU classroom courses; DAU was able to  allocate resources to meet 71 percent of this demand based on its annual  budget. However, DAU made use of the Defense Acquisition Workforce  Development Fund to provide additional classroom seats to meet the  demand for training, allowing them to meet 87 percent of the workforce’s  requirement in fiscal year 2009. Further, DAU data demonstrate that  workforce personnel who require certification training for their current or  future position within their career field constitute a large majority of  classroom students graduating from DAU courses.
section_title: DOD Has Addressed Most Previous Training Recommendations, However, Some Remain to Be Implemented
paragraphs: DOD reports that most of the training-related recommendations from  previous reviews—the Gansler Commission, the Panel on Contracting  Integrity, and our prior report—have been fully implemented. We reviewed  19 recommendations addressing some aspect of acquisition training and  found that 11 have been fully implemented, 4 have been partially  implemented, and 4 have not been implemented but action has been taken.
Two of the four Gansler Commission Report recommendations have been  implemented; however, the Army and Office of the Secretary of Defense  (OSD) need to take additional steps to ensure the Army “trains as it fights”  and that DAU has the resources it needs to train the acquisition workforce.
Nine of the 11 Panel on Contracting Integrity recommendations have been  fully implemented. DOD has taken actions to address performance-based  acquisitions training; however, DOD has not conducted a formal  assessment of its guidance or the training. Also, on the basis of  information from DOD, we could not determine whether it conducted a  review of its Fraud Indicator Training and the Continuity  Book/Contracting Office Transition Plan.
One of the training-related recommendations we made to DCAA has been  partially implemented, and three have not been implemented but action  was taken. DCAA needs to take further steps to develop appropriate  training for its auditors and it should seek outside expertise in doing so.
subsections: section_title: Army and OSD Have Taken Steps to Respond to Gansler Commission Training Recommended Changes, but Additional Steps Need to Be Taken
paragraphs: In response to the Gansler Commission report, the Army and OSD have  taken steps to improve training and implement the report’s  recommendations. In 2007 the Gansler Commission made 4 overarching  recommendations and, within those 4, the Commission described 35 more  in-depth recommendations on Army acquisition and program management  in expeditionary operations. Four of those in-depth recommendations  pertain to training the DAWIA workforce. As shown in table 3, 2 of the  commission’s training recommendations have been fully implemented,  while the remaining 2 training recommendations require additional action.
While DOD has taken action, additional steps are needed to fully  implement the Gansler Commission training recommendations. The  following is our rationale for ongoing efforts needed to continue for the  Army and OSD to fully address the Gansler Commission training  recommendations.  “Train as we fight:” DOD officials stated that training exercises include  contracting and logistics, incorporate lessons learned, and may include  training for commanders, but we could not determine the extent to which  they are included due to lack of documentation. The Army has  mechanisms to capture lessons learned, but it is unclear how they are  incorporated into training exercises. For example, the Expeditionary  Contracting Command informally receives lessons learned from other  Army commands and brigades, but we could not determine whether and  how they are incorporated into training exercises because they are not  tracked or formally documented.
Provide DAU with needed resources to certify Army individuals requiring  level I certification: DAU and the Army do not have the needed resources  to emphasize level I DAWIA certification, according to DOD officials. DAU  is not adequately funded to meet the acquisition training demand DOD- wide. For example, according to OSD officials, DAU is not fully funded to  meet the fiscal year 2011 services and defense-wide agency demand for  contracting level I courses. DAU projects meeting 60 percent of the fiscal  year 2011 requested seats for these level I courses. The Army depends not  only on DAU, but also on the Army Logistics University and the Air Force  Mission Ready Airman Course to provide the contracting training needed  to its active component personnel.
section_title: DOD Has Implemented Most Training Recommendations from the Panel on Contracting Integrity
paragraphs: DOD has not implemented all recommended actions related to defense  acquisition workforce training included in the Panel on Contracting  Integrity’s 2008 and 2009 reports to Congress. The Panel recommended a  total of 49 actions to improve acquisition outcomes. Of these  recommended actions, 11 specifically addressed acquisition training. See  table 4 for a complete list of the recommended actions related to training  included in the Panel’s reports to Congress in 2008 and 2009.
While the Panel reported that all of the recommended actions had been  completed, we determined that two of the recommended actions  pertaining to training had not been fully implemented; we determined that  one was not implemented, but action was taken, and one has been  partially implemented.
Assess effectiveness of DOD guidance and training for executing  performance-based acquisition and perform gap analysis in conjunction  with DAU: The report did not indicate if DOD conducted a formal  assessment of departmental guidance or a gap analysis of training. The  Panel’s Appropriate Contracting Approaches & Techniques Subcommittee  worked with DAU to determine if training needed to be updated and  collected examples of complex and high-dollar acquisitions and posted  them to an Acquisition Community Connection Web site. The report also  noted that DAU would select the best examples from this group for  inclusion in its web-based integrated training tool.
Review Fraud Indicator Training and Continuity Book/Contracting Office  Transition Plan: The Panel report did not specifically address whether a  formal review determined specific gaps in training, as recommended. In  2008, the Panel’s Contracting Integrity in a Combat/Contingent  Environment Subcommittee reported that DOD incorporated transition  planning and fraud indicator training into the Joint Contingency  Contracting Handbook and updated DAU’s Joint Contingency Contracting  Course.
In addition to the recommendations above that are specific to training, the  Panel recommended other actions that also affected training, one of which  was not fully implemented. The Contractor Employee Conflicts of Interest  Subcommittee reviewed and recommended that the Secretary of Defense  issue guidance to clarify the circumstances in which contracts risk  becoming improper personal services contracts. DOD formed an ad hoc  team to respond to the recommendation which focused on establishing a  Defense Federal Acquisition Regulation Supplemental case, DAU course  updates, and a DOD instruction update. While the DOD instruction was  published, the Panel’s report did not mention the status of the Defense  Federal Acquisition Regulation Supplemental case or the DAU course  updates.
section_title: DCAA Has Not Fully Implemented GAO’s Recommendations on Auditor Training
paragraphs: In 2009, we made four recommendations regarding DCAA auditor training,  which have not been fully implemented (see table 5 for our full  recommendations). Three of the recommendations have not been  implemented but action was taken, and one has been partially  implemented. As stated in our September 2009 report, DCAA faces many  challenges and fundamental structural and cultural changes related to  developing a strong management environment and human capital strategic  plan. First, we recommended that once DCAA establishes a risk-based  audit approach it will need to develop a staffing plan that identifies auditor  resource requirements including training needs. Second, we recommended  that DCAA establish a position for an expert or consult with an outside  expert on auditing standards to shape audit policy, provide guidance, and  develop training. While DCAA has taken steps to improve its audit training,  such as implementing an initiative to identify the knowledge, skills, and  competencies required for DCAA auditors and develop training, according  to a DCAA official, it has not yet hired or consulted with an outside expert  on auditing to shape its policies and provide guidance. Third, we  recommended that DCAA develop agencywide training on government  audit standards. Agency officials stated that as of July 2010, DCAA had  developed a new online, introductory course on Generally Accepted  Government Auditing Standards (GAGAS) all DCAA auditors are supposed  to complete by September 30, 2010. We are reviewing the new course  content and continue to work with DCAA on planned improvements to  address the fundamental structural and cultural changes previously  identified. Fourth, as DCAA’s audit quality assurance program identifies  actions needed to address serious deficiencies and GAGAS  noncompliance, we recommended that DCAA provide training and follow- up to ensure that appropriate corrective actions have been taken. DCAA  has issued audit alerts and provided some guidance through periodic  regional office and field office conferences, but has not yet incorporated  this guidance into the body of its DCAI audit courses.
section_title: Conclusions
paragraphs: DOD’s acquisition workforce training program demonstrates many  attributes of effective training and development programs; however, there  is room for further improvement. DOD recognizes the need to continue its  efforts to assess competencies for its acquisition workforce. Importantly,  if this effort is not completed, DOD will be limited in its ability to identify  gaps in the skill sets of acquisition personnel, ultimately hampering its  ability to effectively acquire the goods and services it needs to accomplish  its mission. Notably, opportunities exist to improve the measurement of  training’s impact on overall organizational performance. If DOD is to fully  assess performance improvements, it needs to go beyond measuring the  size of the workforce. To provide appropriate oversight of the proficiency  and capability of its acquisition workforce, DOD will need metrics to  measure skills, knowledge, and abilities, and how certification training  contributes to organizational performance results. Furthermore, DAU  faces challenges with the management and forecasting of training demand  data for specific training courses, which hinders its ability to accurately  plan the course schedule for the upcoming year in a manner that will  facilitate getting the required training to acquisition workforce members in  an efficient and cost-effective manner. Accurate and timely forecasting of  acquisition workforce training requirements and the development of  metrics for the proficiency of the workforce are imperative to support  DOD’s initiatives to improve and grow the acquisition workforce.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct USD(AT&L) to take  the following two actions to improve the development, implementation,  and evaluation of acquisition workforce training.
In order to demonstrate and track how training efforts contribute to  improved acquisition workforce performance, establish milestones for the  development of metrics to measure how acquisition certification training  improves the proficiency and capability of the acquisition workforce.  In order to improve DOD’s ability to identify specific acquisition training  needs for planning and front-end analysis, establish a time frame for  completion and ensure resources are available for implementing an  enterprisewide, integrated student information system.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for comment. In written  comments, DOD did not agree with our first recommendation and did  agree with our second recommendation. DOD’s comments are discussed  below and are reprinted in appendix III.
DOD did not concur with our recommendation that it should develop  milestones for the development of metrics to demonstrate and track how  acquisition certification training improves the acquisition workforce  performance. While DOD agreed that metrics should be used to measure  the capability of the acquisition workforce, it believes developing  milestones for such metrics is unnecessary because existing metrics can  be used to this end. DOD states that workforce capability is a function of  having the correct number of people working in the right areas with the  proper level of education, training, and experience. Specifically, DOD  notes five metrics used to measure size and composition of the workforce  as well as the education, training, and experience levels of the individuals  that comprise it.
We recognize that metrics for measuring these elements are valuable for  gaining insight into the degree to which required workforce personnel are  being certified and filling needed positions. However, as we note in this  report and in GAO’s guidance for assessing strategic training and  development programs, training effectiveness must be measured against  organizational performance. DOD’s existing metrics measure the outputs  for certification training, not the outcome in terms of proficiency or  capability of the acquisition workforce. Without outcome metrics, DOD  cannot demonstrate how certification training contributes to improving  organizational performance results. Given the scale and value of DOD  acquisitions, we maintain that metrics that link training to acquisition  performance outcomes should be developed by the department.
We are sending copies of this report to the Secretary of Defense, the DOD  Inspector General, and other interested parties. The report is also available  at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-4841 or needhamjk1@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: Congress included a provision in the National Defense Authorization Act  for Fiscal Year 2010 requiring us to report on the efficacy of the  Department of Defense’s (DOD) acquisition and audit workforce training.  To determine the efficacy of DOD’s acquisition and audit workforce  training, we assessed (1) DOD’s capability to provide defense acquisition  workforce certification training, (2) the extent that such training reaches  members of DOD’s acquisition workforce, and (3) the extent that training  recommendations from previous reviews, including the Gansler  Commission, have been implemented. We were not able to report on the  efficacy of training for the Defense Contract Audit Agency’s (DCAA)  auditing career field because DCAA lacks a strategic plan. A strategic plan  is a key document for assessing training programs using the strategic  training efforts attributes.
For this engagement, we focused on training for DOD personnel covered  under the Defense Acquisition Workforce Improvement Act (DAWIA). To  assess DOD’s capability to provide defense acquisition workforce  certification training, we compared DOD’s certification training programs  and processes with the attributes of effective training and development  programs identified in GAO’s 2004 guide for assessing strategic training  and development efforts in the federal government, which we identified as  the most comprehensive source for attributes of effective training  programs for our purpose. We interviewed officials at the Defense  Acquisition University (DAU) and Defense Contract Audit Institute (DCAI)  to obtain an understanding of their training programs and processes, and  we obtained documents—such as briefings, guidance, strategic plans, and  course catalogs—describing the training programs and processes. We  interviewed the Directors of Acquisition Career Management (DACM) for  the military services and defense agencies to obtain an understanding of  their role in DOD training, to obtain their views on the effectiveness and  usefulness of DAU training, and to find out whether supplementary  training is provided by the military services. We interviewed the leaders of  the functional integrated process teams that support the functional leaders  of the 15 DAWIA career fields to obtain an understanding of their role in  Acquisition, Technology, and Logistics’ (AT&L) process and criteria for  reviewing and approving acquisition workforce training. In addition, we  visited selected military commands and program offices within those  commands to obtain customer perspectives on the effectiveness and  usefulness of DAU training and to determine the use of supplementary  training. For this purpose, we selected a nongeneralizable sample of one  command from each military service based on the following criteria: (1)  high level of procurement dollars spent in fiscal years 2008 and 2009  relative to other commands in their military service, based on data from  the Federal Procurement Data System-Next Generation; (2) large number  of DAU courses completed in fiscal years 2008-2009; and (3) proximity to a  DAU regional office with an on-site dean. The commands we visited were  the Army Aviation and Missile Command (AMCOM) in Huntsville,  Alabama; the Air Force Materiel Command (AFMC) in Dayton, Ohio; and  the Naval Air Systems Command (NAVAIR) at Patuxent River, Maryland.  In selecting program offices to visit, we reviewed our assessment of  selected weapon programs and consulted with the GAO team responsible  for our assessment to determine which program offices would likely have  a large cross-section of acquisition workforce personnel with whom to  discuss training. We visited the following program offices: Joint Attack  Munition Systems and Apache at AMCOM; Broad Area Maritime  Surveillance Unmanned Aircraft System and E-2D Advance Hawkeye at  NAVAIR; and Global Hawk Unmanned Aircraft System at AFMC. At AFMC,  we also visited the Aeronautical Systems Center’s Contracting Directorate,  and, at AMCOM, we visited the Contracting Center. We also visited  Defense Contract Management Agency personnel to obtain their  perspectives on DAU training and to find out about their use of  supplementary training. Finally, we visited a nongeneralizable sample of  two DCAA locations—the Alabama Branch Office in Huntsville, Alabama,  and the Boston Branch Office in Boston, Massachusetts—to obtain the  customers’ perspectives on DCAI training and determine the use of  supplementary training. We did not examine the appropriateness of the  certification training itself nor the content of courses required for  certification. We did not assess the efficacy of training provided by  supplementary training sources.
To assess the extent to which acquisition training reaches appropriate  acquisition personnel, we reviewed DAU and DCAI policies, and we  received briefings from DAU and DCAI personnel concerning the  determination of training requirements, resource allocation, and  scheduling of classes. We reviewed and analyzed the training requirements  for all defense acquisition career fields. We collected and analyzed defense  acquisition workforce and training data maintained in the AT&L Data Mart  system used by DAU for determining course demand and certification  status of acquisition workforce members. This provided an understanding  of the number of class requests received, class seats scheduled, and  students who registered and completed these courses in past fiscal years.  We also used these data to analyze the number and reasons for uncertified  acquisition workforce personnel. We assessed the reliability of these data  by reviewing data query information for specific data requests and  interviewed knowledgeable officials who collect and use these data. We  intended to focus all analysis of data for fiscal years 2005 through 2009;  however, due to data reliability concerns, we limited portions of our  analysis to data available for fiscal years 2007 through 2009. We  determined that data were sufficiently reliable for the purposes of this  report. We conducted interviews with DAU, DCAI, military department,  and defense agency representatives who have a role in communicating or  analyzing training requirements demand and training resource allocation  to gain a fuller understanding of the processes and challenges faced when  providing training for the defense acquisition workforce. In addition, we  conducted interviews with acquisition workforce members and  supervisors to understand the degree to which they are able to enroll in  needed acquisition training and challenges they may face in completing  this training. We interviewed DAU officials and obtained budget  documents to determine DOD’s use of the Defense Acquisition Workforce  Development Fund (Section 852 of the National Defense Authorization Act  for Fiscal Year 2008) for training and for helping to meet training demand.
To determine the extent to which training recommendations from  previous reviews, including the Gansler Commission, have been  implemented, we identified previous reviews with training  recommendations, and we interviewed and obtained documentation from  agency officials on the status of DOD’s implementation of the  recommendations. Specifically, for Gansler Commission  recommendations, we interviewed Defense Procurement and Acquisition  Policy (DPAP) officials to determine the applicability of the training  recommendations to the acquisition workforce, and we obtained the  Office of the Secretary of Defense’s (OSD) and the Army’s status in  implementing the recommendations and supporting documents, including  reports detailing the recommendations and action items. We analyzed the  supporting documents to assess the status, and, based on our review, we  assigned one of the following six status assessments to each of the  recommendations.  (1) Fully Implemented. The entire wording of the action item has been  fulfilled.  (2) Partially Implemented. Only a portion of the action has been  implemented. When the wording of the action item had multiple parts, if  one part or a portion of a part had been implemented (but not all parts),  we categorized the action item as “partially implemented.”  (3) Not Implemented-Action Taken. No part of the action item has been  implemented, but steps have been taken toward the completion of the  action item. For example, if legislation had been introduced to address the  action but had not been enacted into law, we categorized the action item  as “not implemented-action taken.”  (4) Not Implemented-No Action. No part of the action item has been  completed, and no action has been taken to address the action item. For  example, if the action item called for changes in legislation but no  legislation has even been proposed, we categorized the action item as “not  implemented-no action.”  (5) Insufficient Information. Insufficient or conflicting information  prevented us from determining the status of the action item.  (6) Other. Implementation has occurred or action has been taken that,  while not responsive to the letter of the action item, generally was  consistent with its purpose. For example, if the action item states that a  particular position should be created to coordinate an effort but the  coordination is achieved without the creation of the position, we  categorized the action item as “other.”
We compared our assessment with OSD’s and the Army’s assessment, and,  in making our final determination on implementation status, we provided  OSD and Army officials the results of our initial determinations. The  officials reviewed these results and provided us with additional, clarifying  information that we considered and, when we believed appropriate, used  in making our final determination. For the Panel on Contracting Integrity  reports, we examined whether DOD had implemented the Panel’s  recommendations in 2007 and 2008 by reviewing the 2007, 2008, and 2009  reports. Specifically, we compared the recommended actions from the  2007 report with the reported action in the 2008 report. The same  comparative analysis was conducted using the recommended actions from  2008 and the 2009 report. We differentiated between recommendations  that specifically mention training from those that did not, as well as  recommendations in which training was involved in the implementation of  the recommendation. We compared our assessment with the Panel’s  assessment. We provided our analysis to DPAP officials to review and  provide additional information that we considered in making our final  determination. To determine whether DCAA has implemented GAO’s  recommendations from a prior report, we interviewed officials at DCAA to  understand what actions had been initiated in response to our  recommendations.
We conducted this performance audit from December 2009 to September  2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Attributes of Effective Training and Development Programs
subsections: section_title: (a) Training goals are consistent with its overall mission, goals, and culture
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the individual named above, key contributors to the report  were Penny Berrier Augustine, Assistant Director; Johana Ayers;  Alezandra Brady; Helena Brink; John Krump; Morgan Delaney Ramaker;  Erin Schoening; Angela Thomas; Desiree Thorp; and Tom Twambly.