Human Capital: Opportunities Exist to Further Improve DOD's Methodology for Estimating the Costs of Its Workforces
Released: 2013-09-25T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: DOD must make cost-effective decisions in the use of its military, civilian, and contractor workforces, and CAPE issued guidance that provides a methodology for cost estimates and comparisons among workforces. The conference report accompanying the National Defense Authorization Act for Fiscal Year 2013 mandated that GAO review the cost methodology in Directive-Type Memorandum (DTM) 09-007 or its successor guidance to determine whether they reflect the actual, relevant, and quantifiable costs to taxpayers for work performed by these workforces. This report evaluates the extent to which (1) DOD's methodology reflects the full cost to the taxpayer, and (2) DOD's components incorporated the business rules in the memorandum and successor instruction into workforce mix decisions. GAO compared DOD's cost methodology to guidance from other government entities and interviewed officials from components applying the methodology, as well as other appropriate DOD officials.

section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has improved its methodology for estimating and comparing the full cost to the taxpayer of work performed by military and civilian personnel and contractor support, but the methodology continues to have certain limitations. Best practices state that cost estimating rules should include a common set of standards that minimize conflicts in definitions, but DOD's methodology does not provide guidance for certain costs. For instance, its estimate of service training costs divides total training funding by the number of servicemembers. Using this method yields an average training cost of $6,490 per servicemember in the Army for fiscal year 2012. However, Army data show that training for a general aviation officer can be as high as $93,600 a year, while the training for an enlisted infantryman can be as low as about $4,600 a year. DOD's Cost Assessment and Program Evaluation (CAPE) office has not provided more specific direction on training costs, although some officials have requested it. Additionally, CAPE officials told GAO they did not include Reserve and National Guard personnel in the methodology because usually these personnel are used on a short-term basis. However, a portion of these personnel do serve in a full-time capacity. The Federal Accounting Standards Advisory Board has noted that a cost methodology should include any resources directly or indirectly used to perform work, and DOD relies on Reserve and National Guard personnel, for example, to provide airlift capabilities in support of military operations. Further, CAPE has not yet evaluated certain retirement-related cost elements. A portion of these cost elements may not be appropriate to include because they are not attributable to current military and civilian personnel. Without more specific direction in these areas, it will be more difficult for DOD to have reasonable assurance that its cost estimates and comparisons reflect the full and most accurate cost to the taxpayer of work performed by its various workforces.

DOD components GAO examined generally have incorporated business rules contained in the memorandum and successor instruction into their workforce mix decisions, although DOD officials said opportunities to use the rules have been limited due to budgetary factors and few new or expanded missions. Moreover, implementation challenges exist. Some officials raised questions about the extent to which other officials throughout DOD are aware of a requirement to use the methodology for decisions other than in-sourcing. Further, CAPE recently completed a DOD-wide software tool for implementing its instruction, but at the time of GAO's review, some DOD components had developed their own tools. CAPE officials told GAO that the components' use of its DOD-wide tool will not be required, enforced, or monitored, and that CAPE has not reviewed the components' tools. Best practices state that to be reliable, cost estimates should be derived using a process that produces results that are accurate and can be traced, replicated, and updated. Assessing these tools would enable CAPE to identify the advantages and disadvantages of allowing multiple tools and provide reasonable assurance that cost estimates are reliable. Further, the instruction directs users to a General Services Administration (GSA) website for determining contractor support costs. GAO has reported on limitations of GSA's website such as its reporting of data that do not reflect post-competition prices. Without reliable data sources, DOD components may not be using the most suitable data needed to produce credible cost estimates.

section_title: What GAO Recommends
paragraphs: GAO is recommending that DOD develop further guidance on certain cost elements, such as training; develop business rules for estimating Reserve and National Guard costs; evaluate inclusion or non-inclusion of cost elements related to retirement; assess cost models being used across the department; and reassess sources for contractor data. DOD concurred with two and partially concurred with three of GAO’s recommendations. GAO continues to believe it is important for DOD to fully address the recommendations in order to achieve desired results.

section_title: Letter
paragraphs: To meet its wide array of mission requirements, the Department of  Defense (DOD) maintained a military workforce of over 1.4 million active  duty servicemembers, nearly 850,000 reserve members, and about  800,000 civilian full-time equivalents in fiscal year 2012. Augmenting that  workforce, the department relies on about 710,000 contractor full-time  equivalent personnel. As the federal government confronts growing fiscal  challenges, DOD will compete for resources and face critical workforce  decisions to meet budgetary constraints. Given these constraints, it is  important that the department have an analytically rigorous and balanced  process to estimate the total cost of its military, civilian, and contractor  workforces to inform workforce decisions and make the necessary  program decisions to meet its mission requirements. As reflected in  DOD’s Directive-Type Memorandum (DTM) 09-007 and more recently in  DOD’s Instruction 7041.04, issued on July 3, 2013,to be aware of the full cost of personnel and have a thorough  understanding of the implications of those costs to DOD and the federal  government when developing national security policies and making  program commitments.

GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and  Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). offices of the Under Secretary of Defense (Personnel and Readiness)  (USD(P&R)), the Director of Cost Assessment and Program Evaluation  (CAPE), and DOD’s Office of the Actuary. We also met with the Office of  Management and Budget and other experts and interested parties we  selected based on their work on issues related to the DTM. Specifically,  we met with the Center for Strategic and International Studies, the Center  for Strategic and Budgetary Assessments, the American Federation of  Government Employees, and the Professional Services Council to  discuss their perspectives on the preceding DTM and DOD Instruction.  We also obtained the perspectives of the National Guard and Reserve  community by meeting with the Office of the Secretary of Defense  (Reserve Affairs) and the Reserve Forces Policy Board. To evaluate the  extent to which DOD’s components have incorporated the business rules  identified in the memorandum and instruction into their workforce mix  decisions, we met with knowledgeable officials from 3 to 4 commands  and offices we selected from each of the military departments, and 3  defense agencies, for a total of 13 DOD entities. The results of these  interviews cannot be generalized to all entities across DOD but provided  insights about incorporation of business rules into workforce mix  decisions. We used a non-probability sample, based on size of civilian  workforce as of the end of fiscal year 2012, to make our selections. We  obtained data from each entity on the number of instances they used the  business rules identified in the DTM to inform workforce decisions from  January 2010 through June 2013. In addition, we met with officials in 22  other DOD entities that are involved in guiding the implementation of the  methodology. Although the Marine Corps was not part of our non- probability sample, we met with several Marine Corps commands and  offices to discuss their implementation of the methodology. We also  obtained and evaluated the different software tools developed and  employed by each of the components, defense agencies, and CAPE to  apply the methodology in DTM 09-007 that was finalized in DOD  Instruction 7041.04.

We conducted this performance audit from December 2012 to September  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Further details on our  objectives, scope, and methodology can be found in appendix I.
subsections: section_title: Background
paragraphs: DOD’s DTM 09-007 established business rules, or standard procedures,  to estimate and compare the full cost of military and DOD civilian  personnel and contractor support. These rules were incorporated with  amendments into DOD Instruction 7041.04, which supersedes the DTM.  According to the DTM and the instruction, when developing national  security policies and making program commitments, DOD officials must  be aware of the full costs of personnel and have a thorough  understanding of the implications of those costs to DOD and, on a  broader scale, the federal government. To facilitate this awareness, the  DTM provided and the instruction provides business rules for DOD  officials to estimate the full costs of the defense workforce and contracted  support for tasks supporting planning, defense acquisition, and force  structure. According to the instruction, the Office of the Secretary of  Defense (OSD) and all DOD components are required to use these  business rules when performing an economic analysis to support  workforce decisions, such as determining the workforce mix of new or  expanding mission requirements that are not inherently governmental or  exempt from private-sector performance, and in-sourcing.

Table 1  shows how the DOD components can generally use the business rules  established in DTM 09-007 and in DOD Instruction 7041.04 to estimate  and compare personnel costs and to support workforce mix decisions.

DOD Instruction 7041.04 does not require DOD’s components to make  workforce decisions based on cost alone, but it does require the  components to consider cost in the decision making process when the  function in question is not required by law, regulation, or policy to be  performed by a certain workforce (e.g., inherently governmental or  military essential functions) and other workforce factors are equal. In  other cases, the cost of using personnel to perform work may not be part  of the decision making process. For example, a June 2013 memorandum  from the Assistant Secretary of Defense states that during civilian  furloughs the use of either military personnel or contractors to  compensate for workload resulting from the civilian furloughs is  prohibited.

DOD Instruction 7041.04 states that the full costs of personnel include  labor costs, current and deferred compensation costs paid in cash and in- kind, as well as direct and indirect non-labor costs. For contractor  support, the full cost is the sum of the service contract, the cost of goods,  services and benefits provided in-kind to contractors or reimbursed by  DOD, and the costs to DOD in support of the contract and contract  administration. See table 2 for a description of the full cost of performance  by military and civilian personnel, and contractor support, as defined in  DOD Instruction 7041.04.

Further, the Federal Accounting Standards Advisory Board Handbook  defines full cost as the total amount of resources used to produce the  output. More specifically, the full cost of an output is the sum of (1) the  costs of resources consumed directly or indirectly that contribute to the  output, and (2) the costs of identifiable supporting services provided by  units within the reporting entity and by other reporting entities.
subsections: section_title: Roles and Responsibilities
paragraphs: The Director of CAPE is the principal advisor to the Secretary of Defense  and other senior officials in DOD for independent cost assessment,  program evaluation, and analysis. DOD Instruction 7041.04 states that  CAPE, in collaboration with USD(P&R), and the Under Secretary of  Defense (Comptroller), is responsible for developing a cost model for  DOD-wide application to implement the business rules identified in the  instruction. The instruction also states that CAPE, USD(P&R), the Under  Secretary of Defense (Comptroller), and the heads of the DOD  components (such as the heads of the military departments) and OSD  components, are responsible for using the business rules identified in  the instruction. This includes using the business rules to estimate the full  costs of the defense workforce in support of planning, defense  acquisition, force structure decisions, and when performing an economic  analysis in support of workforce mix decisions.

The DOD Office of the Actuary provides actuarial expertise on all matters  relating to military compensation and benefits, including performing  annual valuations of the military retirement system, education benefits  under the Montgomery G. I. Bill, and health care for the military retired  population.

section_title: Previous GAO Work on Estimating the Full Cost of Personnel
paragraphs: We have previously made recommendations to DOD to develop a  methodology to estimate the full cost of military and civilian personnel. In  our May 2006 report, we found that from fiscal years 2005 through 2007,  the Air Force, the Army, and the Navy collectively converted or planned to convert a total of 5,507 military health care positions to civilian positions.We found at that time that it was unknown whether these conversions  would increase or decrease costs to DOD, primarily because the  methodology each of the departments considered using did not include  the full cost of military personnel. Accordingly, we recommended, and  DOD generally agreed, that the Secretaries of the Air Force, the Army,  and the Navy coordinate with CAPE to develop the full cost for military  personnel and for federal civilian or contract replacement personnel in  assessing whether anticipated costs to hire civilian replacement  personnel will increase costs to DOD for defense health care. In response  to this recommendation, DOD issued DTM 09-007 in January 2010, which  provided a full cost methodology for assessing military, civilian, or  contractor support personnel costs to inform workforce mix decisions,  including military to civilian conversions.

In our February 2008 report on DOD’s efforts to address legislative  requirements to use a full cost methodology to certify and report on  planned conversions of military medical and dental positions to civilian  medical and dental positions, we found that the Navy’s methodology was  the only one that addressed the specific factors identified by the John  Warner National Defense Authorization Act for Fiscal Year 2007 for  positions planned for conversion for fiscal years 2007 and 2008. The Air  Force and the Army relied on composite military rates, instead of using a  full cost methodology. These composite rates did not include all of the  required cost factors, such as training and recruiting costs. Accordingly,  we recommended that DOD, among other things, develop operating  guidance for the military departments to use when justifying future  conversions of military medical and dental positions to civilian positions.  In our recommendation we stated that this guidance should stipulate  requirements to use a consistent full cost methodology for comparing the  cost of military and civilian personnel. Officials in CAPE attributed the  development of DTM 09-007, in part, to these recommendations.

section_title: DOD’s Methodology for Estimating and Comparing the Full Cost to the Taxpayer of Work Performed by Its Workforces is Limited in Certain Aspects
paragraphs: DOD Instruction 7041.04 reflects improvements to DOD’s methodology  for estimating and comparing the full cost to the taxpayer of work  performed by military and civilian personnel and contractor support since  the initial issuance of DTM 09-007, but the instruction is still limited in  certain areas. For example, DOD’s instruction provides limited guidance  on estimating overhead costs and adjusting advertising and recruiting,  and training costs. In addition, CAPE has not established business rules  for estimating the cost of a part of DOD’s total workforce—Reserve and  National Guard personnel. Further, CAPE has not yet evaluated certain  retirement and retiree health benefit cost elements that it is using to  reflect the full cost of currently employed military and civilian personnel.
subsections: section_title: DOD’s Instruction Includes Improvements, but Does Not Provide Guidance on Estimating and Adjusting Certain Cost Elements Related to Overhead, Advertising and Recruiting, and Training Costs
subsections: section_title: Improvements to DOD’s Methodology
paragraphs: DOD Instruction 7041.04 reflects a number of improvements in estimating  certain cost elements in comparison to DTM 09-007, addressing some of  the limitations users of the DTM and the interested parties we met with  identified. While the DTM did not identify a responsible office for  preparing clarifying guidance to assist users in applying the methodology,  the instruction establishes that CAPE, among its other responsibilities, will  be responsible for issuing such guidance. Also, CAPE has expanded the  methodology to address specific elements that users of the DTM  previously identified as missing, such as the cost of foregone taxes, lost  productivity during periods of transition, and some other non-common  costs associated with converting from contract to government  performance. Lastly, CAPE has been developing, refining, and testing a  DOD-wide software tool—the Full Cost of Manpower—that employs the  business rules established in the instruction, and the instruction provides  a link to this tool. However, while CAPE has addressed several of the  limitations users of the preceding DTM and other interested parties  identified, certain limitations still exist.

DOD’s instruction provides limited direction on estimating general and  administrative and overhead costs, and adjusting advertising and  recruiting, and training costs. DOD Instruction 7041.04 states that the  cost elements contained in the instruction can be modified or augmented  in each specific case as necessary, but that the DOD components should  be prepared to support each decision with sufficient justification. Best  practices as reflected in GAO’s Cost Estimating and Assessment Guide  state, however, that establishing ground rules for cost estimating provides  a common set of agreed on estimating standards that provide guidance  and minimize conflicts in definitions.

The instruction directs users to include general and administrative and  overhead costs in cost estimates. These costs include the goods,  services, and benefits that support more than one organization. More  specifically, these costs include a share of supplies and facilities and  professional support services, and a fair share of the recurring costs of  higher-level management and professional support services for  organizations that produce or provide more than one product or service.  The instruction provides a list of direct included for military and civilian personnel, but does not provide data  sources or guidance on how to estimate them. Instead, CAPE identified  subject-matter experts in each of the military departments to serve as  points of contact to address these types of questions and provide  assistance in the future.

DODI 7041.04 defines direct costs as costs that are related directly to the production of  a specific good or the performance of a specific service. Typical direct costs are the  compensation of employees for performance of work, fringe benefit costs, and the costs of  materials consumed or expended in the performance of the work. to contractors through a public-private competition using OMB Circular  A-76. However, both the DOD Inspector General and we have found  that the standard rate of 12 percent of labor costs does not have a sound  analytical basis, which leaves some uncertainty about whether the rate  may be understated or overstated. Our past work acknowledged the  difficulty of obtaining reliable cost data that could provide a sound basis  for an overhead rate, but we concluded that until actual overhead costs  are used to develop a more meaningful standard overhead rate, the  magnitude of savings expected from public-private competitions will be  imprecise and competition decisions could continue to be controversial.

End strength represents the actual number of personnel on board at the end of a fiscal  year. advertising and recruiting costs and added that some functions or  specialties would cost more to recruit and train. Specifically, the users told  us that, for some specialties, the average training rates are too low, while  for other specialties, the rates are too high. The users also said these  costs vary depending on the rank or grade of the personnel, and  depending on whether the specialty requires training or certifications. For  example, the average rate for training in the Army for fiscal year 2012  was $6,490 per servicemember. However, according to Army data, the  amortized cost of training for an officer with a general aviation area of  concentration can range from about $6,500 to $93,600 a year, depending  on rank. In contrast, the amortized cost of training an enlisted member  with an infantryman area of concentration can range from about $4,600 to  $8,000 a year depending on pay grade.

Due to the lack of explanatory guidance on these cost elements, users of  the preceding DTM told us they have developed their own methods for  estimating and adjusting these costs, or have not included the costs in  their estimates. Some officials have requested more developed guidance  on these cost elements, but CAPE did not provide more specific direction  in its recently issued instruction. Rather, as previously mentioned, CAPE  identified subject-matter experts in each of the military departments to  serve as points of contact to address these types of questions. However,  without more developed guidance that establishes a clear set of ground  rules or standards, subject-matter experts in the military departments and  cost estimators must make their own assumptions, which can lead to  inconsistent estimates, as well as hinder DOD’s and congressional  decision makers’ visibility over the costs of accomplishing work by the  different workforces.

section_title: CAPE Has Not Established Business Rules for Estimating Reserve and National Guard Costs
paragraphs: Although Reserve and National Guard personnel are a significant part of  the military workforce, comprising about 38 percent of the military  workforce end strength in fiscal year 2012, CAPE did not establish  business rules for developing cost estimates for these personnel in its  instruction. The Federal Accounting Standards Advisory Board  Handbook states that a cost methodology that captures full costs should  include any resources directly or indirectly used to perform work.  Furthermore, the instruction states that the DOD components should use  the business rules to account for the full cost of personnel when  developing independent cost estimates and pricing units in the force  structure.

Military service officials told us that they currently use or are considering  using Reserve and National Guard personnel to perform work, and that in  the absence of business rules for estimating the cost of these personnel,  some officials said they have generated cost estimates for these  workforces using their own methods. For example, one of the Air Force  commands we interviewed conducted a business case analysis to  evaluate the pros and cons associated with alternative workforce  structures to help them meet their requirements. This analysis included  the consideration of three courses of action: (1) adding additional  contractor support, (2) adding additional civilian and active duty military  personnel or (3) using reservists to provide quick surge capacity. The  command officials said they used the methodology in the preceding DTM  to conduct part of the analysis, and used Air Force guidance for the  reserve component.

DOD’s military workforce consists of both active military and Reserve and  National Guard personnel, and we have previously reported the  importance of DOD employing a strategic approach to managing its total  workforce to achieve its missions.Reserve Forces Policy Board report to the Secretary of Defense  recommended that CAPE establish DOD policy or guidance for  calculating the full cost of the reserve components. The report concluded   In addition, in January 2013, a  that without such a policy, senior leaders within DOD will not have  complete or uniform data on the total costs associated with active and  reserve personnel to make informed workforce decisions. According to  officials in CAPE, determining the best person–civilian or military–to fill a  full-time position is a far different question than determining the best mix  of active and reserve personnel, and that they will work to develop  additional guidance and tools as needed. CAPE added that the Board’s  singular focus on cost obscured the fact that many other factors such as  peacetime and wartime demands, deployment frequency and duration,  and unit readiness are of equal or greater importance. An official in the  Office of the Assistant Secretary of Defense for Reserve Affairs, however,  stated that the reserve components should be included in the instruction  in order for DOD to make effective workforce decisions for its total  workforce.

Accordingly, CAPE added “active duty” to the title of instruction 7041.04 to be clear that  the instruction did not include discussion of Reserve and National Guard personnel costs. to provide engineer units to assist local and state agencies in, among  other things, construction of roads, bridges, and emergency housing.  Without establishing business rules for estimating the cost of Reserve  and National Guard personnel, however, whether as part of the  instruction or in some other venue, DOD cannot create estimates and  comparisons to inform workforce mix and force structure decisions for its  total workforce.

section_title: Portions of Certain Retirement-Related Cost Elements May Not Be Relevant
paragraphs: The instruction directs users to include payments the government makes  into funds to support the retirement and health benefits that current  military and civilian personnel who become eligible for retirement will  receive upon retirement; however, a portion of these payments may not  be relevant to current personnel. Retirement and health benefits are  deferred benefits that current employees will receive in the future. These  deferred benefits, regardless of when the employee receives them, are  attributable to current service by an employee and are to be included in  considering the full costs of a workforce. According to the Federal  Accounting Standards Advisory Board Handbook, however, costs that are  attributable to services rendered in prior years that are attributable to  services previously rendered by current retirees and to past service of  current active civilian and military personnel are sunk costs that should  not be included in determining current costs.

Accordingly, the instruction appropriately includes several cost elements  that capture the actuarial “normal cost” to DOD of future retirement and  health benefits for military and civilian personnel—that is, the costs of  such future benefits that are attributable to a current employee’s work in a   In addition, the instruction includes several cost elements  given year. that represent costs to other federal agencies for deferred benefits. For  contractor support, the cost of retirement and health care are included in  the cost of the contract. See table 3 for a list of cost elements in the  instruction related to retirement, health, and veteran benefits for military  and civilian personnel.

The “normal cost” is the actuarially determined cost of future benefits attributable to  employee service during the fiscal year.

section_title: DOD’s Components Report that They Have Incorporated Business Rules into Their Workforce Mix Decisions, but Implementation Challenges Exist
paragraphs: The DOD components we examined, for the most part, including  commands, offices, and defense agencies, reported to us that they have  incorporated the business rules identified in the DTM and instruction into  their workforce mix decisions. Still, implementation challenges exist.  Some of the DOD components we examined used the business rules for  estimating and comparing the full cost of DOD personnel and contract  support to inform workforce mix decisions, while others reported they  have had limited opportunities to use the business rules. However, some  military service officials said they did not know the extent to which officials  at more local levels were aware of the DTM or used it, and therefore their  reported instances of using the business rules may underestimate the  actual number of instances. In addition, at the time when we met with  these organizations, CAPE had not completed development of a DOD- wide software tool for implementing the instruction and, in the meantime,  the components we examined had developed their own cost tools to  apply the business rules and develop cost estimates. Further, the  instruction that replaced DTM 09-007 directs users to data that may not  contain the most accurate information for determining contractor support  costs.
subsections: section_title: DOD Components Reported Incorporating the Business Rules into Their Workforce Mix Decisions, but Opportunities for Use Have Been Limited
paragraphs: While the DOD components we examined generally reported to us that  they have incorporated the business rules identified in the preceding DTM  and current instruction to create cost estimates in support of workforce  mix decisions, their opportunities to use the business rules have been  limited. Among the 13 DOD components we included in our scope, 8  reported that they used the preceding DTM and provided documentation  showing that cost was a factor in their decision making from January  2010 through June 2013, 4 told us that they had not had the opportunity  to use the DTM but were aware of it, and 1, while it also had not had the  opportunity to use the DTM, reported that it was not aware of it.  According to the DTM and instruction, the policies and procedures  established within it are applicable to all DOD components. Specifically,  the DTM stated and the instruction states that, when developing national  security policies and making program commitments, DOD officials must  be aware of the full costs of personnel and have a thorough  understanding of the implications of those costs to DOD and, on a  broader scale, to the federal government.

All eight of the DOD components we examined that reported that they  had used the DTM to inform workforce mix decisions were commands  within the military services, and a majority of their cost estimates  supported in-sourcing decisions—moving work performed by contractors  to performance by DOD employees. For example, of the 649 instances in  which the commands we examined reported using the DTM from January  2010 through June 2013, 639 or 98 percent were to inform in-sourcing  decisions. Of those 639, the Air Force Materiel Command accounted for  approximately 525 or 82 percent. These commands provided us with  documentation such as decision memoranda, completed cost benefit  analysis spread sheets, and briefing slides given to senior leadership  showing that cost was a factor in their decision-making. Further, with the  exception of one instance in the Naval Supply Systems Command,  officials from DOD components we met told us that, since January 2010,  they have converted neither military to civilian positions nor civilian to  military positions. The Army Installation Management Command and the  Air Force Materiel Command reported a few instances of new or  expanding missions that required the use of the DTM. Table 4 shows the  number of times the commands we examined used the DTM and for what  reason.

Some DOD officials said they were not clear on uses of the DTM beyond  in-sourcing, and others said their reported instances of using the business  rules may underestimate the actual number of reported instances.  Officials within OSD and headquarters elements said they did not know  the extent to which officials throughout the department are aware of the  instruction and the requirement to use the associated business rules. For  example, it was not clear to some Army officials in one office that the  business rules were to be used for costing out decisions beyond in- sourcing. Other officials noted, however, that the business rules identified  in the DTM or references to the DTM have been incorporated into some  service-level policies and procedures to support decisions other than in- sourcing. For example, the Army’s approval form for new service  contracts includes a question asking if the cost of labor for new services  contracts was determined using the business rules. Some military  service officials said they did not know the extent to which the reported  data represented the full degree to which organizations were, in fact,  using the business rules, and therefore their reported instances of using  the business rules may underestimate the actual number of instances.  For example, one official acknowledged to us that it is difficult for his  organization to identify the degree to which the business rules were being  applied at the local levels.

Officials within the agencies and offices that were aware of the DTM but  had not used it told us of several reasons why they did not conduct these  types of workforce cost estimates since the initial issuance of the DTM.  For example, officials said they had not used it, in part, due to issues  related to the current fiscal environment, such as concerns about  anticipated reductions in funds available for contractor support and  limitations on the number of civilian full-time equivalents. Officials also  said they did not use the DTM because their offices do not have direct  access to military personnel and have had no new or expanded missions.

section_title: DOD Components Use Different Tools to Implement the DTM
paragraphs: CAPE recently completed development of a DOD-wide software tool for  implementing DOD Instruction 7041.04. During the time we met with  component organizations, however, the DOD-wide tool was not available,  and in the absence of the tool the components we examined had  developed their own tools to apply the business rules in the DTM to  develop workforce cost estimates. When initially released on January 29,  2010, DTM 09-007 called for CAPE, within one year of its publication, to  develop a cost model for DOD-wide application that employs the business  rules set forth in the DTM. In addition, best practices in GAO’s Cost  Estimating and Assessment Guide state that in order to be reliable, cost  estimates should be derived using an overall process that produces high- quality cost estimates that are comprehensive and accurate and that can  be easily and clearly traced, replicated, and updated.

According to officials in CAPE, they contracted out the development of a  software tool for the required cost model to implement the business rules  and, when DOD issued Instruction 7041.04 on July 3, 2013, CAPE also  released its DOD-wide tool for use across the department. Officials in  CAPE told us that the components’ use of the DOD-wide tool will not be  required, enforced, or monitored. DTM 09-007 stated that CAPE would  oversee compliance with both the DTM and the use of the DOD-wide tool.  When DOD Instruction 7041.04 was issued, however, the requirement for  CAPE to oversee use of its tool was removed.

In the absence of CAPE’s DOD-wide tool, the Air Force modified an  existing software tool originally used to inform public-private competition  decisions, and the Army, the Marine Corps, and the Navy developed tools  using an existing off-the-shelf software program. While the services  developed and use different tools, those tools generally incorporate the  cost elements identified in the DTM and current instruction. The defense  agencies we met with said they had not created tools to implement the  business rules, and officials from these agencies said they would use the  DOD-wide tool when it was made available to them.

According to the instruction, the business rules provide a consistent  approach for all DOD components to estimate the cost of personnel.  Accordingly, the instruction provides a list of potential cost factors  associated with personnel that should be considered in the decision- making process even when personnel costs are not the only factor.  Officials in CAPE stated, however, that they have not reviewed the  services’ tools to ensure they are in compliance with the business rules  and do not plan to review them. An assessment of these various tools  would enable CAPE to identify the advantages and disadvantages of  allowing multiple cost estimation tools. DOD decision makers who then  use these various economic analyses or cost estimates would have  greater assurance that they are using reliable results to make workforce  mix decisions.

section_title: DOD Components May Be Relying on Data Sources That Do Not Contain the Most Accurate Data for Determining Contractor Costs
paragraphs: When estimating contractor support costs for a new or expanding  mission, Instruction 7041.04 provides several options for users to  consider, although these options may not be the most accurate data  sources. For example, the instruction directs the user to begin with the  negotiated price of an existing contract, which DOD and military  department officials told us is their preferred option. If an existing contract  is not available, those officials attempt to find another contract that is  similar to the contactor support for which costs are being estimated. To  facilitate this, most of the services, agencies, and offices we met with  maintain their own database of historical contractor data.

If an existing contract is not available, the instruction directs the user to  either a General Services Administration (GSA) website or the Army’s  online database for contract management and reporting to obtain  contractor support costs for services. We have previously highlighted  limitations with GSA’s data, however, such as the fact that contractors’ published rates on the website do not reflect post-competition prices.The GSA website allows users to search for services, and then identifies  a list of contractors that provide the service along with their rates. The  listed rates, however, may be negotiated with the contractor. Further,  DOD officials we spoke to during this review said that GSA’s website  does not provide targeted data, such as actual contractor rates by  function or geographic location. For example, officials with one command  we spoke to noted their use of a contractor who possesses unique  specialized nuclear weapons related knowledge. According to those  officials, GSA’s website does not contain available data for unique  positions like these. In addition, officials said that GSA’s website provides  too large a range of rates for them to develop realistic estimates. Best  practices from GAO’s Cost Estimating and Assessment Guide state that a  basic characteristic of credible cost estimates is having multiple sources  of suitable, relevant, and available data.

In July 2013, when the DOD Instruction replacing DTM 09-007 was  issued, the Army’s Contractor Manpower Reporting Application was  added as an additional source for contractor data. The Contractor  Manpower Reporting Application is an online database that automates  the Army’s contract management and reporting process for contract  management personnel by allowing users to view contract information,  track contract data, and view various reports based on contract data in  the application. The Contractor Manpower Reporting Application business  process captures information on funding source, contracting vehicle,  organization supported, mission and function performed, labor hours, and  labor costs for contracted efforts, among other things. Currently the  application collects data for the Army, the Air Force, and the Navy on the  number of contractor employees by using direct labor hours and  associated cost data. However, DOD is still developing its department- wide Contractor Manpower Reporting Application system and we  previously found that a number of factors limit the accuracy and  completeness of inventory data.unaware of some of the limitations with the contractor support data  sources provided in the instruction. In the absence of data sources that  are consistent with established practices for developing cost estimates,  DOD components may be using data that do not lead to credible  contractor cost estimates.

section_title: Conclusions
paragraphs: With a total workforce of about 3 million individuals, as well as an  estimated 710,000 contractor full time equivalents, DOD’s investment in  personnel to accomplish its missions constitutes a substantial financial  commitment. In those cases where the department can choose what  workforce it wants to utilize to accomplish tasks, it is crucial that it have  an accurate method for comparing the costs of its civilian, military, and  contractor personnel. DOD has taken important steps to achieve this goal  since our 2008 report on military to civilian conversions for health care  personnel, including the development of DTM 09-007 and its successor  Instruction 7041.04. The department’s current direction for estimating and  comparing the cost of its available workforces, however, could be  improved. For instance, users currently find it difficult to develop  estimates of particular costs such as overhead, advertising and recruiting,  and training, and the services have not received guidance on developing  estimates for Reserve and National Guard personnel. Further, as we  have noted, there is some disagreement within the department about the  inclusion of certain retirement costs in current workforce cost estimates.  More comprehensive guidance in these areas could improve the  components’ ability to make accurate cost comparisons between  workforces.

Similarly, the department currently lacks assurance that workforce cost  estimates and comparisons are consistent across the department, but by  evaluating the different cost estimation tools currently being used by the  components, it could decide on the best course to ensure consistency  and accuracy. Also, current and accurate data on contractor support  costs is critical for the department in making workforce decisions, but due  to limitations in some of the data DOD has identified for making contractor  support cost estimates, components may be relying on data sources that  do not produce intended results. While federal agencies always must  make consistent and cost-effective choices in managing their resources,  this is especially true given the ongoing fiscal challenges that have  imposed budgetary constraints across the federal government and are  likely to continue for some time.

section_title: Recommendations for Executive Action
paragraphs: To improve DOD’s estimates and comparisons of the full cost of its  military, civilian, and contractor workforces, we are making the following  five recommendations to the Secretary of Defense.

To improve DOD’s methodology for estimating and comparing the full  cost of its various workforces, we recommend that the Secretary of  Defense direct the Office of Cost Assessment and Program Evaluation  office to take the following three actions:

Further develop guidance for cost elements that users have identified  as challenging to calculate, such as general and administrative,  overhead, advertising and recruiting, and training;

Develop business rules for estimating the full cost of National Guard  and Reserve personnel; and  In coordination with the department’s Office of the Actuary and  appropriate federal actuarial offices, reevaluate the inclusion and  quantification of pension, retiree health care costs, and other relevant  costs of an actuarial nature and make revisions as appropriate.

To facilitate consistent workforce cost estimates and comparisons, we  recommend that the Secretary of Defense direct the Office of Cost  Assessment and Program Evaluation to assess the advantages and  disadvantages of allowing the continued use of different cost estimation  tools across the department or directing department-wide application of  one tool, and revise its guidance in accordance with the findings of its  analysis.

To improve DOD’s ability to estimate contractor support costs, we  recommend that the Secretary of Defense direct the Office of Cost  Assessment and Program Evaluation, consistent with established  practices for developing credible cost estimates, to research the data  sources it is currently using and reassess its contractor support data  sources for use when determining contractor support costs.

section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD agreed with our  findings that it must make cost-effective decisions in structuring and  shaping its workforce of military personnel, government civilians, and  contracted support. Specifically, DOD concurred with two of our  recommendations and partially concurred with three. DOD’s comments  are reprinted in appendix II. DOD also provided technical comments on  the draft report, which we incorporated as appropriate.

DOD partially concurred with our recommendation to further develop  guidance for the cost elements that users have identified as challenging  to calculate, such as general and administrative, overhead, advertising  and recruiting, and training. In commenting on our report, DOD stated that  it will continue to review the methodology for calculating these cost  elements and issue clarifying guidance where necessary or appropriate.  DOD also stated that it is part of an inter-agency effort that is developing  government-wide cost comparisons and that, if or when government-wide  cost comparison guidance is published, DOD will adjust its own guidance  accordingly. We continue to believe that fully addressing this  recommendation would enhance the development of DOD’s methodology  for estimating and comparing the cost of its workforces.

DOD also partially concurred with our recommendation to develop  business rules for estimating the full cost of National Guard and Reserve  personnel. In its comments, DOD stated that it is assessing the potential  need for reserve manpower costing models, as well as the identification  of questions that reserve costing policies and models would be used to  address. Once it has gained a more thorough understanding of the  questions to be addressed by reserve cost estimates, it will begin work on  guidance as necessary, and development of reserve component costing  models if appropriate. We continue to believe that fully addressing this  recommendation would enhance the development of DOD’s methodology  for estimating and comparing the cost of its workforces and DOD’s ability  to make more informed workforce decisions.

DOD concurred with our recommendation to coordinate with the  department’s Office of the Actuary and appropriate federal actuarial  offices, reevaluate the inclusion and quantification of pension, retiree  health care costs, and other relevant costs of an actuarial nature and  make revisions as appropriate. In its response to our report, DOD stated  that it will work with the Office of the Actuary, and others as necessary, to  reevaluate the inclusion and quantification of these costs elements and,  following evaluation, revisions will be made. We believe such actions, if  implemented effectively, will fully address the intent of the  recommendation.

DOD partially concurred with our recommendation to assess the  advantages and disadvantages of allowing the continued use of different  cost estimation tools across the department or directing department-wide  application of one tool, and revise its guidance in accordance with the  findings of its analysis. We acknowledge, as DOD stated in comments on  our report, that the department is open to assessing the advantages and  disadvantages of allowing the continued use of alternate cost estimation  tools among the various components, and, if necessary, will revise its  guidance based on this assessment. However, in order to satisfy the  intent of this recommendation, DOD needs to take action to fully address  this recommendation. Doing so will facilitate more consistent workforce  cost estimates and comparisons when DOD’s components use the  department’s methodology for estimating and comparing the cost of their  workforces.

Finally, DOD concurred with our recommendation to research the data  sources it is currently using and reassess its contractor support data  sources for use when determining contractor support costs. In its  comments, DOD stated that as the department increases its fidelity into  contractor support costs through the collection of statutorily required  information via the Enterprise-wide Contractor Manpower Reporting  Application in support of the Inventory of Contracts for Services, the  department will modify its guidance accordingly. We believe that this  action, if implemented effectively, will address the intent of the  recommendation.

We are sending copies of this report to the Secretary of Defense, the  Under Secretary of Defense for Personnel and Readiness, the Director of  DOD’s Office of Cost Assessment and Program Evaluation, the Office of  Management and Budget, and appropriate congressional committees. In  addition, this report will also be available at no charge on the GAO Web  site at http://www.gao.gov.

If you or your staff have any questions regarding this report, please  contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for the  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.

section_title: Appendix I: Scope and Methodology
paragraphs: To evaluate the extent to which the Department of Defense’s (DOD)  methodology for estimating and comparing the cost of work performed by  military, civilian, and contractors reflects the full cost to the taxpayer, we  identified each of the cost elements contained in the most recent version  of Directive Type-Memorandum (DTM) 09-007 and DOD Instruction  7041.04 and compared them to best practices in GAO’s Cost Estimating  and Assessment Guide, guidance in the Federal Accounting Standards  Advisory Board Handbook, and the Office of Management and Budget  guidelines for Performance of Commercial Activities. We also reviewed  pertinent DOD and service-specific policies and guidance related to  workforce mix and economic analysis. In addition, we interviewed  knowledgeable officials within the offices of the Under Secretary of  Defense (Personnel and Readiness), the Director of Cost Assessment  and Program Evaluation, DOD’s Office of the Actuary, and the military  services. We also met with the Office of Management and Budget and  other experts and other interested parties we selected based on their  work on issues related to the DTM. Specifically, we met with the Center  for Strategic and International Studies and the Center for Strategic and  Budgetary Assessments, both of which are independent nonprofit policy  research institutes. We also met with the American Federation of  Government Employees, a federal government employee union, and the  Professional Services Council, a trade association of the government  professional and technical services industry. We obtained the perspective  of the National Guard and Reserve community by meeting with the Office  of the Secretary of Defense (Reserve Affairs) and the Reserve Forces  Policy Board. We also reviewed published work by independent research  institutes evaluating the business rules contained in the preceding DTM,  such as those issued by the Center for Strategic and International  Studies, the Project on Government Oversight, and the Reserve Forces  Policy Board.

DOD Instruction 7041.04 was issued on July 3, 2013. Therefore, for a majority of our  review, DTM 09-007 was the most current guidance.

Corps is reported to the Defense Manpower Data Center, we were unable  to align numbers of civilian personnel with specific Marine Corps  commands or offices. Therefore, the Marine Corps was excluded from our  selection criteria and we did not select any Marine Corps commands as  part of our non-probability sample. Once the data for the other  components was compiled, we rank ordered the commands within each  of the military departments and the defense agencies from largest to  smallest based on civilian personnel counts. For each of the military  departments and defense agencies, we divided the rank ordered list of  commands and agencies into three groups: large, medium, and small. We  considered commands and agencies with 10,000 or more full-time civilian  employees as large, commands and agencies with at least 1,000 but less  than 10,000 full-time civilian employees as medium; and commands and  agencies with less than 1,000 full-time civilian employees as small. We  generated a selection number based on the total number of commands  and agencies and counted down the rank ordered list of commands and  agencies to identify those commands and agencies that we would meet  with. This method resulted in one selected command, office, or agency  from each of the categories of large, medium, and small. In addition to  discussing with each of these entities their implementation of the  business rules identified in DTM 09-007, we requested the number of  times they used the business rules to inform workforce decisions (e.g., in- sourcing, workforce conversions, and new or expanded missions) from  January 2010 when the DTM was issued to June 2013. We did not  independently validate the number of uses of the business rules reported  to us by each of the entities or the cost estimates used to inform the  workforce decisions. Table 5 shows the commands and agencies we met  with to determine the extent to which DOD incorporated the business  rules contained in DTM 09-007.

In addition, we met with officials in other DOD entities that are involved in  guiding the implementation of the methodology. Although the Marine  Corps was not part of our non-probability sample, we met with several  Marine Corps commands and offices to discuss their implementation of  the business rules identified in DTM 09-007. We also received a  consolidated Marine Corps response to our questions on the  implementation of the methodology. Table 6 shows the other DOD  organizations we contacted to discuss the department’s efforts to  implement the business rules contained in the preceding DTM.

Further, we received a demonstration by the Office of Cost Assessment  and Program Evaluation of its DOD-wide Full Cost of Manpower tool to  gain an understanding of DOD’s application of the business rules  contained in DTM 09-007. We also requested and obtained copies of the  tools from each of the commands, offices, and defense agencies we met  with to develop an understanding of the software tools they developed to  apply the business rules in the DTM and in the instruction. For the Air  Force we obtained a copy of their software tool, DTM-COMPARE. For the  Army, the Marine Corps, and Navy we obtained copies of the off-the-shelf  software that was programmed to implement the business rules contained  in the preceding DTM. In addition, we attended an Army Installation  Management Command training on their Cost Analysis Workbook tool.  Further, we obtained and reviewed sample documentation from the  military commands that had applied the business rules identified in the  preceding DTM to support workforce mix decisions.

We conducted this performance audit from December 2012 to September  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Defense

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Brenda S. Farrell, (202) 512-3604 or Farrellb@gao.gov.

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, David Moser, Assistant Director;  Timothy Carr, Brian Pegram, Erin Preston, Frank Todisco, Erik Wilkins- McKee, and Michael Willems made key contributions to this report.

section_title: Related GAO Products
paragraphs: Defense Contractors: Information on the Impact of Reducing the Cap on  Employee Compensation Costs. GAO-13-566. Washington, D.C.: June  19, 2013.

Human Capital: Additional Steps Needed to Help Determine the Right  Size and Composition of DOD’s Total Workforce. GAO-13-470.  Washington, D.C.: May 29, 2013.

Defense Acquisitions: Continued Management Attention Needed to  Enhance Use and Review of DOD’s Inventory of Contracted Services.  GAO-13-491. Washington, D.C.: May 23, 2013.

Pension Costs on DOD Contracts: Additional Guidance Needed to  Ensure Costs Are Consistent and Reasonable. GAO-13-158.  Washington, D.C.: January 22, 2013.

Federal Workers: Results of Studies on Federal Pay Varied Due to  Differing Methodologies. GAO-12-564. Washington, D.C.: June 22, 2012.

Defense Workforce: DOD Needs to Better Oversee In-sourcing Data and  Align In-sourcing Efforts with Strategic Workforce Plans. GAO-12-319.  Washington, D.C.: February 9, 2012.

DOD Met Statutory Reporting Requirements on Public-Private  Competitions. GAO-11-923R. Washington, D.C.: September 26, 2011.

Military Personnel: Comparisons between Military and Civilian  Compensation Can Be Useful, but Data Limitations Prevent Exact  Comparison. GAO-10-666T. Washington, D.C.: April 28, 2010.

Military Personnel: Military and Civilian Pay Comparisons Present  Challenges and Are One of Many Tools in Assessing Compensation.  GAO-10-561R. Washington, D.C.: April 1, 2010.

Military Personnel: Guidance Needed for Any Future Conversions of  Military Medical Positions to Civilian Positions. GAO-08-370R.  Washington, D.C.: February 8, 2008.

Military Personnel: Military Departments Need to Ensure That Full Costs  of Converting Military Health Care Positions to Civilian Positions Are  Reported to Congress. GAO-06-642. Washington, D.C.: May 1, 2006.


Question-summary annotations: questions: question: What shortcomings does the DOD’s methodology for estimating taxpayer cost still have?
answer: The Department of Defense (DOD) has improved its methodology for estimating and comparing the full cost to the taxpayer of work performed by military and civilian personnel and contractor support, but the methodology continues to have certain limitations. Best practices state that cost estimating rules should include a common set of standards that minimize conflicts in definitions, but DOD's methodology does not provide guidance for certain costs.
child_questions: question: How do discrepancies in servicemember training costs illuminate this issue?
answer: For instance, its estimate of service training costs divides total training funding by the number of servicemembers. Using this method yields an average training cost of $6,490 per servicemember in the Army for fiscal year 2012. However, Army data show that training for a general aviation officer can be as high as $93,600 a year, while the training for an enlisted infantryman can be as low as about $4,600 a year.
child_questions: question: How has CAPE responded to the request to provide more specific direction on training costs?
answer: DOD's Cost Assessment and Program Evaluation (CAPE) office has not provided more specific direction on training costs, although some officials have requested it.

question: What personnel is not included in CAPE’s methodology?
answer: Additionally, CAPE officials told GAO they did not include Reserve and National Guard personnel in the methodology because usually these personnel are used on a short-term basis.
child_questions: question: Why is this a cause of concern?
answer: However, a portion of these personnel do serve in a full-time capacity.

question: What does the DOD rely on National Guard personnel to do?
answer: The Federal Accounting Standards Advisory Board has noted that a cost methodology should include any resources directly or indirectly used to perform work, and DOD relies on Reserve and National Guard personnel, for example, to provide airlift capabilities in support of military operations.

question: Why may some costs not be appropriate to include in the methodology?
answer: Further, CAPE has not yet evaluated certain retirement-related cost elements. A portion of these cost elements may not be appropriate to include because they are not attributable to current military and civilian personnel.

question: Until more specific direction is given, what will DOD have trouble assuring?
answer: Without more specific direction in these areas, it will be more difficult for DOD to have reasonable assurance that its cost estimates and comparisons reflect the full and most accurate cost to the taxpayer of work performed by its various workforces.

questions: question: What did GAO notice about the DOD components it examined?
answer: DOD components GAO examined generally have incorporated business rules contained in the memorandum and successor instruction into their workforce mix decisions, although DOD officials said opportunities to use the rules have been limited due to budgetary factors and few new or expanded missions.

question: What questions have DOD officials raised?
answer: Some officials raised questions about the extent to which other officials throughout DOD are aware of a requirement to use the methodology for decisions other than in-sourcing.

question: What does best practices state about reliable cost estimates?
answer: Best practices state that to be reliable, cost estimates should be derived using a process that produces results that are accurate and can be traced, replicated, and updated.

question: How would assessing DOD components tools help CAPE?
answer: Assessing these tools would enable CAPE to identify the advantages and disadvantages of allowing multiple tools and provide reasonable assurance that cost estimates are reliable.

question: What issues are on GSA's website?
answer: Further, the instruction directs users to a General Services Administration (GSA) website for determining contractor support costs. GAO has reported on limitations of GSA's website such as its reporting of data that do not reflect post-competition prices.
child_questions: question: What problems may DOD have because of the issues on GSA's website?
answer: Without reliable data sources, DOD components may not be using the most suitable data needed to produce credible cost estimates.

questions: question: What must DOD make?
answer: DOD must make cost-effective decisions in the use of its military, civilian, and contractor workforces, and CAPE issued guidance that provides a methodology for cost estimates and comparisons among workforces.

question: What did the conference report accompanying the National Defense Authorization Act for Fiscal Year 2013 mandate?
answer: The conference report accompanying the National Defense Authorization Act for Fiscal Year 2013 mandated that GAO review the cost methodology in Directive-Type Memorandum (DTM) 09-007 or its successor guidance to determine whether they reflect the actual, relevant, and quantifiable costs to taxpayers for work performed by these workforces.

question: What does this report examine?
answer: This report evaluates the extent to which (1) DOD's methodology reflects the full cost to the taxpayer, and (2) DOD's components incorporated the business rules in the memorandum and successor instruction into workforce mix decisions.
child_questions: question: What did GAO compare for the report?
answer: GAO compared DOD's cost methodology to guidance from other government entities and interviewed officials from components applying the methodology, as well as other appropriate DOD officials.

questions: question: What is GAO recommending to DOD?
answer: GAO is recommending that DOD develop further guidance on certain cost elements, such as training; develop business rules for estimating Reserve and National Guard costs; evaluate inclusion or non-inclusion of cost elements related to retirement; assess cost models being used across the department; and reassess sources for contractor data.
child_questions: question: How did DOD respond to the recommendations?
answer: DOD concurred with two and partially concurred with three of GAO’s recommendations.

question: What does GAO believe is important for DOD?
answer: GAO continues to believe it is important for DOD to fully address the recommendations in order to achieve desired results.