Defense Acquisition Workforce: DOD's Training Program Demonstrates Many Attributes of Effectiveness, but Improvement Is Needed
Released: 2010-10-28T00:00:00

Letter

Background

Congress passed DAWIA in 1990 to ensure effective and uniform education, training, and career development of members of the acquisition workforce. Accordingly, the act established DAU to provide training for the DOD acquisition workforce and charged DOD officials with designating acquisition positions, setting qualification requirements, and establishing policies and procedures for training the acquisition workforce. DOD, as part of implementing DAWIA, established career fields, such as program management (See table 1).

The act also required DOD to establish career paths, referred to by DOD as certification requirements, for the acquisition workforce. DOD military services and defense agencies must track that acquisition workforce members meet mandatory standards established for level I (basic or entry), level II (intermediate or journeyman), or level III (advanced or senior) in a career field, such as contracting, life-cycle logistics, and program management. DAU is responsible for certification training and for designing, maintaining, and overseeing the delivery of certification training courses at each level, among other things. For each career field and level, there are requirements in three areas—education, experience, and training. Certification requirements are the same for civilian and military acquisition workforce members. Table 2 shows the nature of certification training for one of the DAWIA career fields—system planning, research, development, and engineering (SPRDE)—systems engineering, as well as shows the education and experience requirements for each level in the career field.

Besides the certification training it offers, DAU approves alternative certification training providers based on a review by an independent organization—the American Council on Education—of the capability of a potential provider to offer acquisition training and whether the provider’s course content addresses the DAU course’s learning outcomes. An equivalent course provider must certify annually that its course is current with the DAU plan of instruction for the course. Similarly, DCAI provides both required certification training and supplemental training for the auditor career field.

In addition to certification training, DAU offers supplemental training for each career field and for particular types of assignments. For example, for level II contracting in contingency or combat operations, DAU provides courses such as a contingency contracting simulation, a contingency contracting officer refresher, and a joint contingency contracting course. DAU also provides continuous learning modules online to provide acquisition workforce members with a quick reference for material already introduced and courses to help them maintain currency in their career field by achieving the required 80 continuous learning points biennially. Additionally, DAU provides consulting support to program offices, rapid-deployment training on new initiatives, and training targeted to the needs of acquisition field organizations. DAU also engages in knowledge-sharing initiatives, including hosting a number of acquisition communities of practice and providing Web-based acquisition policy and reference materials.

In March 2004, we issued a guide for assessing federal training programs that breaks the training and development process into four broad, interrelated components—(1) planning and front-end analysis, (2) design and development, (3) implementation, and (4) evaluation. The guide discusses attributes of effective training and development programs that should be present in each of the components and identifies practices that indicate the presence of the attribute. For example, under the design and development component, to determine whether an organization possesses the attribute of incorporating measures of effectiveness into courses it designs, the guide suggests looking for practices, such as (1) clear linkages between specific learning objectives and organizational results and (2) well-written learning objectives that are unambiguous, achievable, and measurable. For a complete list of the attributes of effective training and development programs, see appendix II.

Figure 1 depicts the training and development process along with the general relationships between the four components that help to produce a strategic approach to federal agencies’ training and development programs. These components are not mutually exclusive and encompass subcomponents that may blend with one another. Evaluation, for example, should occur throughout the process.

DOD’s Certification Training Program Has the Capability to Provide Training, Although It Lacks Some Attributes of Effective Training

DOD’s acquisition workforce certification training—-centrally administered by DAU—has many attributes of effective training programs that demonstrate the capability to deliver training. DAU’s certification training program has a formal process in planning and front-end analysis to ensure that strategic and tactical changes are promptly incorporated into training; use of centralized and decentralized training approaches in design and development; data collection during implementation to ensure feedback on its training programs; and appropriate analytical approaches to assess its training during evaluation. However, DOD lacks complete information on the skill sets of the current acquisition workforce for planning and front-end analysis and does not have metrics to assess results achieved in enhancing workforce proficiency and capability through training efforts during evaluation. Complete data on acquisition skill sets are needed to accurately identify workforce gaps, and appropriate metrics are necessary to increase the likelihood that desired changes will occur in the acquisition workforce’s skills, knowledge, abilities, attitudes, or behaviors.

DOD’s Certification Training Program Has Many Attributes of Effective Training Programs

DOD’s certification training program possesses attributes of effective training programs in each of the four components of the training and development process. Following are examples of the attributes we observed in DOD training categorized by the components of effective training programs.

Planning and front-end analysis: Planning and front-end analysis can help ensure that training efforts are not initiated in an ad hoc, uncoordinated manner, but rather are strategically focused on improving performance toward the agency’s goals. DAU had processes to ensure that training efforts were coordinated and focused on improving agency goals. Through a formal process that ensures that strategic and tactical changes are promptly incorporated into training, DAU and other DOD stakeholders plan for and evaluate the effectiveness of DAU’s training efforts. Each career field has a functional leader, a senior subject-matter expert in the career field who is responsible for annually certifying that course content for certification is current, technically accurate, and consistent with DOD acquisition policies. Functional leaders are supported by a functional integrated process team for each career field, which consists of subject- matter experts, acquisition career management representatives from the military services and other DOD agencies, and DAU representatives. The functional integrated process team analyzes and reviews data, including end-of-course evaluations, number of students completing a class, wait lists, and certification rates, as well as DOD policy changes and recommendations from reviews, such as the Gansler Commission to support functional leaders. DAU designs courses in accordance with the functional leader and functional integrated process team decisions. Using this process, strategic and tactical changes were promptly incorporated into training. For example, DAU developed and fielded a new contracting course on federal acquisition regulation fundamentals within a year of direction by the functional leader’s organization to create it.

Design and development: In design and development, it is important that agencies consciously consider the advantages and disadvantages of using centralized and decentralized approaches. Centralizing design can enhance consistency of training content and offer potential cost savings. DAU evaluates and uses centralized and decentralized approaches for training after considering the advantages and disadvantages. DAU’s curriculum development and technologies organizations located at Fort Belvoir, Virginia, provide centralized, integrated design and development of certification courses. These courses are then delivered to the acquisition workforce by five regionally-oriented campuses and the Defense Systems Management College School of Program Managers. DAU also compares training delivery mechanisms to determine the appropriate use of different delivery mechanisms (such as classroom or computer- based training) and to ensure efficient and cost-effective delivery. In addition, supplementary training is offered at the Army, Navy, and Air Force commands and program offices we visited, as well as at the Defense Contract Management Agency. While DAU provides a foundation for acquisition and career field knowledge in its certification training, various decentralized sources provide supplementary training more targeted to specific jobs, such as training on service-specific processes or databases and technical topics. Acquisition workforce members at the commands we visited provided the following examples of supplementary training.

The contracting offices at the Army Aviation and Missile Command (AMCOM), Alabama, and the Air Force Aeronautical Systems Center (ASC), Ohio, provided unique training in the contracting area. AMCOM’s Contracting Center University teaches employees how to do day-to-day tasks associated with their job, such as price analysis, price negotiation, and how to use the Army Materiel Command-unique system for preparing contract documents. ASC’s “jump start” program teaches, reinforces, and supplements DAU certification training in the contracting career field with illustrative examples not provided in the computer-based contracting courses as well as offers an opportunity to interact with instructors and other students.

The Naval Air Systems Command (NAVAIR), Maryland, provides supplementary training for DAWIA career fields. For example, in the program management career field, NAVAIR offers courses in configuration management and on NAVAIR’s technical directives system.

Other acquisition workforce members provided examples of training from other federal agencies or commercial vendors, such as financial training from the Graduate School, United States Department of Agriculture, and Management Concepts, while others said they had brown bag lunches on various topics.

Figure 2 below identifies DOD’s multifaceted training approach, both centralized and decentralized. The objective of the multifaceted training, in conjunction with the other two certification components—education and experience— is acquisition personnel with the training, education, and experience to perform the acquisition job.

Implementation/Evaluation: As with other programs and services that agencies deliver, it is important that agencies collect program performance data during implementation and select an analytical approach that best measures the program’s effect to evaluate their training and development efforts. DAU collects customer feedback data during implementation and, during evaluation, uses the four-level Kirkpatrick model as an analytical approach for measuring training effectiveness. As a part of evaluating training, DAU conducts student end-of-course surveys (Level 1-Reaction) and, to a lesser degree, follow-up surveys of students and their managers 60 and 120 days, respectively, after course completion (Level 3-Behavior). DAU tracks the scores from the various surveys by survey section, such as job impact, and uses red-yellow-green stoplight indicators to identify areas of concern overall and by specific courses. DAU also administers pre- and post-training tests to measure learning (Level 2-Learning). To measure organizational impact (Level 4-Business Results), DAU employs measures of efficiency in evaluating and analyzing multiyear data, such as number of students completing courses, cost efficiency, and customer satisfaction trends. Level 4 assessments are resource intensive and have not been extensively used by DAU.

Some Attributes of DOD’s Certification Training Programs are Inadequate

DOD is deficient in two attributes of an effective training program— determining the skills and competencies of its workforce for planning and front-end analysis and using performance data to assess the results achieved through training efforts during evaluation. In March 2009, we reported that USD(AT&L) lacks complete information on the skill sets of the current acquisition workforce and whether these skill sets are sufficient to accomplish DOD’s missions. We recommended and DOD agreed to identify and update on an ongoing basis the number and skill sets of the total acquisition workforce—including civilian, military, and contractor personnel—that the department needs to fulfill its mission. Complete data on skill sets are needed to accurately identify its workforce gaps. Not having these data limits DOD’s ability to make informed workforce allocation decisions. We reported that USD(AT&L) was conducting a competency assessment to identify the skill sets of its current acquisition workforce, but also found that the lack of key data on the in-house acquisition workforce identified in the prior report still exists, though progress has been made. Since we released that report, DOD issued its Strategic Human Capital Plan Update in April 2010. According to DOD, progress was made in completing over 22,000 assessments involving 3 of the 15 career fields—program management, life-cycle logistics, and contracting career fields. The assessments completed to date represent approximately one-fifth of the personnel and career fields.

Although DAU uses performance data—including customer feedback, number of students completing classes, and cost—to assess the results achieved through training efforts during evaluation, USD(AT&L) has only partially established metrics required in 2005 by its own guidance to provide senior leaders with appropriate oversight and accountability for management and career development of the acquisition workforce. The purpose of these metrics is to help DOD ensure a sufficient pool of highly qualified individuals for acquisition positions and, therefore, relates to the knowledge, skills, abilities, and size of the acquisition workforce, while the DAU performance data measure the performance of DAU against its goals. By incorporating these metrics into the training and development programs they offer, DOD can better ensure that they adequately address training objectives and thereby increase the likelihood that desired changes will occur in the acquisition workforce’s skills, knowledge, abilities, attitudes, or behaviors. AT&L programs lacking appropriate outcome metrics will be unable to demonstrate how the certification training contributes to organizational performance results. According to USD(AT&L)’s Deputy Director for Human Capital Initiatives, DOD has established some metrics to measure the size of the acquisition workforce that partially satisfy the requirements identified in DOD Instruction 5000.66. For example, DOD measures the cumulative number of civilian and military acquisition positions added as a result of in-sourcing acquisition functions performed by contractors. However, for metrics related to acquisition workforce proficiency and capability, there are no discernable targets, except improvement over the previous year. In addition, DOD’s April 2010 Strategic Human Capital Plan Update identified an initiative to establish certification goals as a management tool for improving workforce quality by June 10, 2010. According to the Deputy Director, certification goals are being discussed but they had not been established at the time of this report.

Despite Incomplete Data and High Demand, Most Workforce Members Receive Training Required for Certification

Although DAU is unable to provide all training requested for acquisition workforce personnel and receives incomplete data for planning its training schedule, most personnel who need required DAWIA certification training receive it within required time frames. DAU plans the number and location of its classes based on data submitted by the Directors of Acquisition Career Management (DACM). However, DOD acquisition and training officials noted that data are generally incomplete when submitted and additional steps must be taken during the year to meet new requirements as they are identified. DAU has identified the need for an integrated student information system to improve the quality of the data and to provide greater insight into the workforce it supports. Additionally, though the number of DAU course graduates has grown over the past 5 years, DAU has not been able to provide enough class seats to meet the training requirements reported by military departments and defense components.

DAU Receives Incomplete Information to Develop an Annual Teaching Schedule

DAU receives annual DACM data submissions for the course scheduling process, but the submissions do not provide the exact information needed to determine training demand for the acquisition workforce. DAU receives class requirements data annually from the DACM offices that it uses when developing course schedules to identify the number and location of DAU courses. DACM offices compile this information for all offices to establish the overall demand for each military department and the defense agencies for each DAU course. DAU and DACM offices work together throughout the process to improve the accuracy of this information when possible. According to DAU and DACM officials, however, data that are transmitted for schedule development do not fully reflect all demand for the upcoming year as new requirements arise once the schedule is developed. As a result additional planning and coordination between DAU and DACM offices is necessary to meet the training requirements of the acquisition workforce. For example, in fiscal year 2009, DAU received requests for 142 additional classes outside of the normal scheduling process. DAU was able to support 45 of these requested classes in such areas as program management, contracting, business management, and logistics. According to DAU officials, resources for additional classes are made available when other classes are cancelled. Also, DAU may reallocate allotted classroom seats among departments and agencies to fill additional training needs. DAU officials stated that data on selected acquisition support services that are currently performed by contractors who may transition to in-house DOD personnel are not adequate for planning specific training requirements. Though DOD has established goals for the number of contracted personnel to be converted, DAU officials noted that the exact time or training backgrounds of the personnel are not known in advance.

DAU also uses acquisition workforce data provided quarterly by the DACM offices that include information such as the number of personnel in each acquisition career field as well as the career level, job titles, and status of progress against certification requirements of each workforce member to inform course demand management. According to DAU officials, these data provide a snapshot of the acquisition workforce and certification status, and they use this information to estimate the number, location, and type of classes needed by the acquisition workforce for certification. The data are compiled to create a demand management tool that provides DAU with an imprecise estimate of course requirements and are used to supplement and inform the estimates developed during the scheduling process. However, this demand management tool alone cannot be used by DAU to determine the exact number of classroom seats required each fiscal year. According to DAU officials, the workforce data collected may overstate training requirements because it does not account for training that has already been completed when individuals held a previous acquisition position, nor does it discern between multiple classes that may fulfill the same training requirement.

Citing incomplete data for scheduling, as well as other deficiencies, DAU has taken steps to procure a student information system that will improve insight into and enhanced management of the defense acquisition workforce’s training needs. DAU began its market research for an integrated student information system in December 2007, viewed vendor presentations and demonstrations throughout 2008, and issued a request for proposal in August 2010. In the request for proposal, DAU identified the need for an integrated system for registration, student services, career management, schedule management, catalog requirements, recording transcripts, and reporting intended to improve its management of training needs and schedules. Without an integrated system, DAU states that it will remain reliant on a web of decentralized information that makes reporting and trend analysis difficult and time-consuming. A primary goal of the new system is to provide a comprehensive approach that improves, among other things, tracking of certification status and ensures training reaches the right people at the right time. DAU plans call for the contractor to complete implementation of this new student information system 24 months after the date of contract award, which had not been made as of September 2010.

DCAI develops its training schedule based on the requirements expressed in the individual development plans and availability of DCAI resources. Registration for DCAI courses is prescribed and based largely on the individual developmental plans submitted by DCAA’s approximately 3,700- member auditing workforce in fiscal year 2009. Each year DCAA employees develop an individual development plan that lists DCAI courses as well as outside training deemed necessary with the input and approval of their supervisors. This information is input into a system that tracks course requirements and individuals’ status against training requirements. Individuals are automatically enrolled into the scheduled DCAI courses.

Personnel Receive Required Training within Required Time frames According to DAU Data, However, DAU Cannot Support All Training Requirements for the Defense Acquisition Workforce

Most of the acquisition workforce receives training within required certification time frames. At the end of fiscal year 2009, approximately 90 percent of the 133,103 members of the defense acquisition workforce had met certification requirements associated with their position or were within allowed time frames to do so. Acquisition workforce members we met with from all three military departments and the Defense Contract Management Agency (DCMA) noted challenges for receiving training at the time and location they desired, noting that local DAU locations would fill up quickly and that they would often have to register for courses multiple times prior to enrollment. However, acquisition staff and supervisors told us that this had little effect on being certified within the required time frames for their current positions. Nearly all of the remaining uncertified personnel required training to become certified. While additional training was needed, these individuals may also have been deficient in meeting education or experience requirements also needed for DAWIA certification. Furthermore, DACM officials noted that there could be a number of reasons these individuals had not received required training and stated that while some individuals may not have adequately planned for their training needs, other factors, such as deployment of military personnel abroad, may have limited their access to training.

DCAA auditors do not face the same issues with DAWIA certification as the rest of the acquisition workforce. According to DCAI officials, this is largely because they do not have to coordinate demand for courses across several different agencies. All new hires are automatically enrolled in the courses required for level I and level II DAWIA certification. Additionally, DOD reported that approximately 99 percent of the auditing workforce had met certification requirements or were within allowed time frames to do so. By completing the mandatory learning track taught through DCAI classes, DCAA auditors complete certification training within required time frames.

Even though 90 percent of the acquisition personnel who required certification training for their current position received training on time or were within allowed time frames to do so, DAU acknowledges that requests for acquisition workforce training as a whole submitted by the DOD components and military departments exceed what DAU can provide. DAU has incorporated expansion of training into its strategic plans. In its Strategic Plan for 2010-2015, DAU notes that it will play a key role in the USD(AT&L) acquisition workforce growth strategy. For example, USD(AT&L) efforts to grow, train, and develop the defense acquisition workforce will affect DAU’s strategic planning over the next several years. DAU notes that workforce growth goals put forth by the Secretary of Defense in April 2009 will increase the demand for DAU training and therefore affect how DAU plans for development of acquisition personnel, requiring careful consideration of resource allocation. The strategic plan also points out a number of other factors that will drive the demand for acquisition workforce training in the coming years, including annual workforce turnover, turnover related to Base Realignment and Closure, and support for new acquisition development needs. As part of its strategy, DAU has also established short-term goals to expand training capacity in its fiscal year 2010 Organizational Performance Plan, including expanding classroom training by 10,000 seats over fiscal year 2009 levels. DAU officials stated that they plan to increase capacity further to provide 54,000 classroom seats in fiscal year 2011. In addition, DAU established and has fulfilled a strategic goal of graduating 150,000 students from its Web-based courses annually.

DAU has increased the total number of course graduates and classes in recent years to address demand for acquisition training. DAU has supported more classes than in the past, seeing an increase from 1,279 classroom courses in fiscal year 2005 to 1,505 in fiscal year 2009. In addition, from fiscal year 2005 through 2009, the number of individual graduates from DAU classroom and Web-based courses rose by approximately 77 percent (see fig. 3).

To support increases in certification training demand due to workforce growth through new hiring and in-sourcing, DAU uses funding from the Defense Acquisition Workforce Development Fund to provide additional facilities and courses. Though the majority of funding is intended to support the hiring of new staff, DAU, military departments, and defense agencies received more than $225 million dollars to support new training and additional seats in fiscal years 2008 and 2009. Funds have been used by the military departments to support Army and Navy acquisition boot camps, the Air Force’s mission-ready contracting course, and other acquisition training developed by specific military commands. For example, funding was used to develop and implement the “jump start” program at the Air Force’s Aeronautical Systems Center that combines material taught through DAU’s Level I contracting courses with Air Force- specific information. The Defense Acquisition Workforce Development Fund has also been used by DAU to expand its teaching facilities, hire additional instructors, and schedule additional classes needed for DAWIA certification. DAU received nearly $165 million in fiscal years 2008 and 2009 to expand training. In fiscal year 2009, this funding permitted DAU to offer nearly 7,000 additional classroom seats in 31 different courses. DAU has also used these funds to develop new training—such as a 4-week course focusing on the Federal Acquisition Regulation that senior DOD contracting officials said was needed to provide a foundation for acquisition fundamentals—and to support acquisition professionals in the field through Service Acquisition Workshops and expanded contingency acquisition training.

Despite these increased class offerings that have accommodated more graduates, DAU has not been able to provide the total number of classroom seats that are requested by the defense acquisition workforce through the DACMs. Classroom seats requested and class seats scheduled both increased from fiscal year 2007 through 2009. For example, in fiscal year 2009, DOD components requested 52,998 seats for the acquisition workforce across 66 different DAU classroom courses; DAU was able to allocate resources to meet 71 percent of this demand based on its annual budget. However, DAU made use of the Defense Acquisition Workforce Development Fund to provide additional classroom seats to meet the demand for training, allowing them to meet 87 percent of the workforce’s requirement in fiscal year 2009. Further, DAU data demonstrate that workforce personnel who require certification training for their current or future position within their career field constitute a large majority of classroom students graduating from DAU courses.

DOD Has Addressed Most Previous Training Recommendations, However, Some Remain to Be Implemented

DOD reports that most of the training-related recommendations from previous reviews—the Gansler Commission, the Panel on Contracting Integrity, and our prior report—have been fully implemented. We reviewed 19 recommendations addressing some aspect of acquisition training and found that 11 have been fully implemented, 4 have been partially implemented, and 4 have not been implemented but action has been taken.

Two of the four Gansler Commission Report recommendations have been implemented; however, the Army and Office of the Secretary of Defense (OSD) need to take additional steps to ensure the Army “trains as it fights” and that DAU has the resources it needs to train the acquisition workforce.

Nine of the 11 Panel on Contracting Integrity recommendations have been fully implemented. DOD has taken actions to address performance-based acquisitions training; however, DOD has not conducted a formal assessment of its guidance or the training. Also, on the basis of information from DOD, we could not determine whether it conducted a review of its Fraud Indicator Training and the Continuity Book/Contracting Office Transition Plan.

One of the training-related recommendations we made to DCAA has been partially implemented, and three have not been implemented but action was taken. DCAA needs to take further steps to develop appropriate training for its auditors and it should seek outside expertise in doing so.

Army and OSD Have Taken Steps to Respond to Gansler Commission Training Recommended Changes, but Additional Steps Need to Be Taken

In response to the Gansler Commission report, the Army and OSD have taken steps to improve training and implement the report’s recommendations. In 2007 the Gansler Commission made 4 overarching recommendations and, within those 4, the Commission described 35 more in-depth recommendations on Army acquisition and program management in expeditionary operations. Four of those in-depth recommendations pertain to training the DAWIA workforce. As shown in table 3, 2 of the commission’s training recommendations have been fully implemented, while the remaining 2 training recommendations require additional action.

While DOD has taken action, additional steps are needed to fully implement the Gansler Commission training recommendations. The following is our rationale for ongoing efforts needed to continue for the Army and OSD to fully address the Gansler Commission training recommendations. “Train as we fight:” DOD officials stated that training exercises include contracting and logistics, incorporate lessons learned, and may include training for commanders, but we could not determine the extent to which they are included due to lack of documentation. The Army has mechanisms to capture lessons learned, but it is unclear how they are incorporated into training exercises. For example, the Expeditionary Contracting Command informally receives lessons learned from other Army commands and brigades, but we could not determine whether and how they are incorporated into training exercises because they are not tracked or formally documented.

Provide DAU with needed resources to certify Army individuals requiring level I certification: DAU and the Army do not have the needed resources to emphasize level I DAWIA certification, according to DOD officials. DAU is not adequately funded to meet the acquisition training demand DOD- wide. For example, according to OSD officials, DAU is not fully funded to meet the fiscal year 2011 services and defense-wide agency demand for contracting level I courses. DAU projects meeting 60 percent of the fiscal year 2011 requested seats for these level I courses. The Army depends not only on DAU, but also on the Army Logistics University and the Air Force Mission Ready Airman Course to provide the contracting training needed to its active component personnel.

DOD Has Implemented Most Training Recommendations from the Panel on Contracting Integrity

DOD has not implemented all recommended actions related to defense acquisition workforce training included in the Panel on Contracting Integrity’s 2008 and 2009 reports to Congress. The Panel recommended a total of 49 actions to improve acquisition outcomes. Of these recommended actions, 11 specifically addressed acquisition training. See table 4 for a complete list of the recommended actions related to training included in the Panel’s reports to Congress in 2008 and 2009.

While the Panel reported that all of the recommended actions had been completed, we determined that two of the recommended actions pertaining to training had not been fully implemented; we determined that one was not implemented, but action was taken, and one has been partially implemented.

Assess effectiveness of DOD guidance and training for executing performance-based acquisition and perform gap analysis in conjunction with DAU: The report did not indicate if DOD conducted a formal assessment of departmental guidance or a gap analysis of training. The Panel’s Appropriate Contracting Approaches & Techniques Subcommittee worked with DAU to determine if training needed to be updated and collected examples of complex and high-dollar acquisitions and posted them to an Acquisition Community Connection Web site. The report also noted that DAU would select the best examples from this group for inclusion in its web-based integrated training tool.

Review Fraud Indicator Training and Continuity Book/Contracting Office Transition Plan: The Panel report did not specifically address whether a formal review determined specific gaps in training, as recommended. In 2008, the Panel’s Contracting Integrity in a Combat/Contingent Environment Subcommittee reported that DOD incorporated transition planning and fraud indicator training into the Joint Contingency Contracting Handbook and updated DAU’s Joint Contingency Contracting Course.

In addition to the recommendations above that are specific to training, the Panel recommended other actions that also affected training, one of which was not fully implemented. The Contractor Employee Conflicts of Interest Subcommittee reviewed and recommended that the Secretary of Defense issue guidance to clarify the circumstances in which contracts risk becoming improper personal services contracts. DOD formed an ad hoc team to respond to the recommendation which focused on establishing a Defense Federal Acquisition Regulation Supplemental case, DAU course updates, and a DOD instruction update. While the DOD instruction was published, the Panel’s report did not mention the status of the Defense Federal Acquisition Regulation Supplemental case or the DAU course updates.

DCAA Has Not Fully Implemented GAO’s Recommendations on Auditor Training

In 2009, we made four recommendations regarding DCAA auditor training, which have not been fully implemented (see table 5 for our full recommendations). Three of the recommendations have not been implemented but action was taken, and one has been partially implemented. As stated in our September 2009 report, DCAA faces many challenges and fundamental structural and cultural changes related to developing a strong management environment and human capital strategic plan. First, we recommended that once DCAA establishes a risk-based audit approach it will need to develop a staffing plan that identifies auditor resource requirements including training needs. Second, we recommended that DCAA establish a position for an expert or consult with an outside expert on auditing standards to shape audit policy, provide guidance, and develop training. While DCAA has taken steps to improve its audit training, such as implementing an initiative to identify the knowledge, skills, and competencies required for DCAA auditors and develop training, according to a DCAA official, it has not yet hired or consulted with an outside expert on auditing to shape its policies and provide guidance. Third, we recommended that DCAA develop agencywide training on government audit standards. Agency officials stated that as of July 2010, DCAA had developed a new online, introductory course on Generally Accepted Government Auditing Standards (GAGAS) all DCAA auditors are supposed to complete by September 30, 2010. We are reviewing the new course content and continue to work with DCAA on planned improvements to address the fundamental structural and cultural changes previously identified. Fourth, as DCAA’s audit quality assurance program identifies actions needed to address serious deficiencies and GAGAS noncompliance, we recommended that DCAA provide training and follow- up to ensure that appropriate corrective actions have been taken. DCAA has issued audit alerts and provided some guidance through periodic regional office and field office conferences, but has not yet incorporated this guidance into the body of its DCAI audit courses.

Conclusions

DOD’s acquisition workforce training program demonstrates many attributes of effective training and development programs; however, there is room for further improvement. DOD recognizes the need to continue its efforts to assess competencies for its acquisition workforce. Importantly, if this effort is not completed, DOD will be limited in its ability to identify gaps in the skill sets of acquisition personnel, ultimately hampering its ability to effectively acquire the goods and services it needs to accomplish its mission. Notably, opportunities exist to improve the measurement of training’s impact on overall organizational performance. If DOD is to fully assess performance improvements, it needs to go beyond measuring the size of the workforce. To provide appropriate oversight of the proficiency and capability of its acquisition workforce, DOD will need metrics to measure skills, knowledge, and abilities, and how certification training contributes to organizational performance results. Furthermore, DAU faces challenges with the management and forecasting of training demand data for specific training courses, which hinders its ability to accurately plan the course schedule for the upcoming year in a manner that will facilitate getting the required training to acquisition workforce members in an efficient and cost-effective manner. Accurate and timely forecasting of acquisition workforce training requirements and the development of metrics for the proficiency of the workforce are imperative to support DOD’s initiatives to improve and grow the acquisition workforce.

Recommendations for Executive Action

We recommend that the Secretary of Defense direct USD(AT&L) to take the following two actions to improve the development, implementation, and evaluation of acquisition workforce training.

In order to demonstrate and track how training efforts contribute to improved acquisition workforce performance, establish milestones for the development of metrics to measure how acquisition certification training improves the proficiency and capability of the acquisition workforce. In order to improve DOD’s ability to identify specific acquisition training needs for planning and front-end analysis, establish a time frame for completion and ensure resources are available for implementing an enterprisewide, integrated student information system.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for comment. In written comments, DOD did not agree with our first recommendation and did agree with our second recommendation. DOD’s comments are discussed below and are reprinted in appendix III.

DOD did not concur with our recommendation that it should develop milestones for the development of metrics to demonstrate and track how acquisition certification training improves the acquisition workforce performance. While DOD agreed that metrics should be used to measure the capability of the acquisition workforce, it believes developing milestones for such metrics is unnecessary because existing metrics can be used to this end. DOD states that workforce capability is a function of having the correct number of people working in the right areas with the proper level of education, training, and experience. Specifically, DOD notes five metrics used to measure size and composition of the workforce as well as the education, training, and experience levels of the individuals that comprise it.

We recognize that metrics for measuring these elements are valuable for gaining insight into the degree to which required workforce personnel are being certified and filling needed positions. However, as we note in this report and in GAO’s guidance for assessing strategic training and development programs, training effectiveness must be measured against organizational performance. DOD’s existing metrics measure the outputs for certification training, not the outcome in terms of proficiency or capability of the acquisition workforce. Without outcome metrics, DOD cannot demonstrate how certification training contributes to improving organizational performance results. Given the scale and value of DOD acquisitions, we maintain that metrics that link training to acquisition performance outcomes should be developed by the department.

We are sending copies of this report to the Secretary of Defense, the DOD Inspector General, and other interested parties. The report is also available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or needhamjk1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Congress included a provision in the National Defense Authorization Act for Fiscal Year 2010 requiring us to report on the efficacy of the Department of Defense’s (DOD) acquisition and audit workforce training. To determine the efficacy of DOD’s acquisition and audit workforce training, we assessed (1) DOD’s capability to provide defense acquisition workforce certification training, (2) the extent that such training reaches members of DOD’s acquisition workforce, and (3) the extent that training recommendations from previous reviews, including the Gansler Commission, have been implemented. We were not able to report on the efficacy of training for the Defense Contract Audit Agency’s (DCAA) auditing career field because DCAA lacks a strategic plan. A strategic plan is a key document for assessing training programs using the strategic training efforts attributes.

For this engagement, we focused on training for DOD personnel covered under the Defense Acquisition Workforce Improvement Act (DAWIA). To assess DOD’s capability to provide defense acquisition workforce certification training, we compared DOD’s certification training programs and processes with the attributes of effective training and development programs identified in GAO’s 2004 guide for assessing strategic training and development efforts in the federal government, which we identified as the most comprehensive source for attributes of effective training programs for our purpose. We interviewed officials at the Defense Acquisition University (DAU) and Defense Contract Audit Institute (DCAI) to obtain an understanding of their training programs and processes, and we obtained documents—such as briefings, guidance, strategic plans, and course catalogs—describing the training programs and processes. We interviewed the Directors of Acquisition Career Management (DACM) for the military services and defense agencies to obtain an understanding of their role in DOD training, to obtain their views on the effectiveness and usefulness of DAU training, and to find out whether supplementary training is provided by the military services. We interviewed the leaders of the functional integrated process teams that support the functional leaders of the 15 DAWIA career fields to obtain an understanding of their role in Acquisition, Technology, and Logistics’ (AT&L) process and criteria for reviewing and approving acquisition workforce training. In addition, we visited selected military commands and program offices within those commands to obtain customer perspectives on the effectiveness and usefulness of DAU training and to determine the use of supplementary training. For this purpose, we selected a nongeneralizable sample of one command from each military service based on the following criteria: (1) high level of procurement dollars spent in fiscal years 2008 and 2009 relative to other commands in their military service, based on data from the Federal Procurement Data System-Next Generation; (2) large number of DAU courses completed in fiscal years 2008-2009; and (3) proximity to a DAU regional office with an on-site dean. The commands we visited were the Army Aviation and Missile Command (AMCOM) in Huntsville, Alabama; the Air Force Materiel Command (AFMC) in Dayton, Ohio; and the Naval Air Systems Command (NAVAIR) at Patuxent River, Maryland. In selecting program offices to visit, we reviewed our assessment of selected weapon programs and consulted with the GAO team responsible for our assessment to determine which program offices would likely have a large cross-section of acquisition workforce personnel with whom to discuss training. We visited the following program offices: Joint Attack Munition Systems and Apache at AMCOM; Broad Area Maritime Surveillance Unmanned Aircraft System and E-2D Advance Hawkeye at NAVAIR; and Global Hawk Unmanned Aircraft System at AFMC. At AFMC, we also visited the Aeronautical Systems Center’s Contracting Directorate, and, at AMCOM, we visited the Contracting Center. We also visited Defense Contract Management Agency personnel to obtain their perspectives on DAU training and to find out about their use of supplementary training. Finally, we visited a nongeneralizable sample of two DCAA locations—the Alabama Branch Office in Huntsville, Alabama, and the Boston Branch Office in Boston, Massachusetts—to obtain the customers’ perspectives on DCAI training and determine the use of supplementary training. We did not examine the appropriateness of the certification training itself nor the content of courses required for certification. We did not assess the efficacy of training provided by supplementary training sources.

To assess the extent to which acquisition training reaches appropriate acquisition personnel, we reviewed DAU and DCAI policies, and we received briefings from DAU and DCAI personnel concerning the determination of training requirements, resource allocation, and scheduling of classes. We reviewed and analyzed the training requirements for all defense acquisition career fields. We collected and analyzed defense acquisition workforce and training data maintained in the AT&L Data Mart system used by DAU for determining course demand and certification status of acquisition workforce members. This provided an understanding of the number of class requests received, class seats scheduled, and students who registered and completed these courses in past fiscal years. We also used these data to analyze the number and reasons for uncertified acquisition workforce personnel. We assessed the reliability of these data by reviewing data query information for specific data requests and interviewed knowledgeable officials who collect and use these data. We intended to focus all analysis of data for fiscal years 2005 through 2009; however, due to data reliability concerns, we limited portions of our analysis to data available for fiscal years 2007 through 2009. We determined that data were sufficiently reliable for the purposes of this report. We conducted interviews with DAU, DCAI, military department, and defense agency representatives who have a role in communicating or analyzing training requirements demand and training resource allocation to gain a fuller understanding of the processes and challenges faced when providing training for the defense acquisition workforce. In addition, we conducted interviews with acquisition workforce members and supervisors to understand the degree to which they are able to enroll in needed acquisition training and challenges they may face in completing this training. We interviewed DAU officials and obtained budget documents to determine DOD’s use of the Defense Acquisition Workforce Development Fund (Section 852 of the National Defense Authorization Act for Fiscal Year 2008) for training and for helping to meet training demand.

To determine the extent to which training recommendations from previous reviews, including the Gansler Commission, have been implemented, we identified previous reviews with training recommendations, and we interviewed and obtained documentation from agency officials on the status of DOD’s implementation of the recommendations. Specifically, for Gansler Commission recommendations, we interviewed Defense Procurement and Acquisition Policy (DPAP) officials to determine the applicability of the training recommendations to the acquisition workforce, and we obtained the Office of the Secretary of Defense’s (OSD) and the Army’s status in implementing the recommendations and supporting documents, including reports detailing the recommendations and action items. We analyzed the supporting documents to assess the status, and, based on our review, we assigned one of the following six status assessments to each of the recommendations. (1) Fully Implemented. The entire wording of the action item has been fulfilled. (2) Partially Implemented. Only a portion of the action has been implemented. When the wording of the action item had multiple parts, if one part or a portion of a part had been implemented (but not all parts), we categorized the action item as “partially implemented.” (3) Not Implemented-Action Taken. No part of the action item has been implemented, but steps have been taken toward the completion of the action item. For example, if legislation had been introduced to address the action but had not been enacted into law, we categorized the action item as “not implemented-action taken.” (4) Not Implemented-No Action. No part of the action item has been completed, and no action has been taken to address the action item. For example, if the action item called for changes in legislation but no legislation has even been proposed, we categorized the action item as “not implemented-no action.” (5) Insufficient Information. Insufficient or conflicting information prevented us from determining the status of the action item. (6) Other. Implementation has occurred or action has been taken that, while not responsive to the letter of the action item, generally was consistent with its purpose. For example, if the action item states that a particular position should be created to coordinate an effort but the coordination is achieved without the creation of the position, we categorized the action item as “other.”

We compared our assessment with OSD’s and the Army’s assessment, and, in making our final determination on implementation status, we provided OSD and Army officials the results of our initial determinations. The officials reviewed these results and provided us with additional, clarifying information that we considered and, when we believed appropriate, used in making our final determination. For the Panel on Contracting Integrity reports, we examined whether DOD had implemented the Panel’s recommendations in 2007 and 2008 by reviewing the 2007, 2008, and 2009 reports. Specifically, we compared the recommended actions from the 2007 report with the reported action in the 2008 report. The same comparative analysis was conducted using the recommended actions from 2008 and the 2009 report. We differentiated between recommendations that specifically mention training from those that did not, as well as recommendations in which training was involved in the implementation of the recommendation. We compared our assessment with the Panel’s assessment. We provided our analysis to DPAP officials to review and provide additional information that we considered in making our final determination. To determine whether DCAA has implemented GAO’s recommendations from a prior report, we interviewed officials at DCAA to understand what actions had been initiated in response to our recommendations.

We conducted this performance audit from December 2009 to September 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Attributes of Effective Training and Development Programs

(a) Training goals are consistent with its overall mission, goals, and culture

Appendix IV: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the individual named above, key contributors to the report were Penny Berrier Augustine, Assistant Director; Johana Ayers; Alezandra Brady; Helena Brink; John Krump; Morgan Delaney Ramaker; Erin Schoening; Angela Thomas; Desiree Thorp; and Tom Twambly.