Joint Exercise Program: DOD Needs to Take Steps to Improve the Quality of Funding Data
Published: 2017-02-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Joint Exercise Program is the principal means for combatant commanders to maintain trained and ready forces, exercise contingency and theater security cooperation plans, and conduct joint and multinational training exercises. These exercises are primarily aimed at developing the skills needed by U.S. forces to operate in a joint environment and can also help build partner-nation capacity and strengthen alliances.
House Report 114-102 included a provision for GAO to review DOD's Joint Exercise Program. This report (1) describes guidance DOD has developed for its Joint Exercise Program and DOD's approach to assess the effectiveness of the program and (2) evaluates the extent to which DOD uses two information technology systems to manage the program. GAO observed data in JTIMS and analyzed fiscal years 2014-16 financial data and supporting documentation in the Execution Management System.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has developed a body of guidance for the Joint Exercise Program and has implemented an approach to assess the effectiveness of the program. In addition to the body of guidance for the program, DOD is working to update a key guidance document for military training in accordance with a congressional requirement. DOD's approach to assess the effectiveness of the Joint Exercise Program is aimed at ensuring that its performance measures are specific, measurable, achievable, realistic, and time-phased (commonly referred to as the SMART rubric). The Joint Assessment and Enabling Capability office reviews the performance measures created by the combatant commands against this rubric and provides input and coaching on improving the measures through an ongoing and collaborative process.
DOD uses two key information technology systems—the Joint Training Information Management System (JTIMS) and the Execution Management System—to manage the execution of the Joint Exercise Program, but does not have assurance that funding execution data in the Execution Management System are reliable. JTIMS is the system of record for the Joint Exercise Program that combatant commanders use to plan and manage their joint training exercises. GAO observed significant variation in the type and quality of information entered in JTIMS. Combatant command and Joint Staff officials stated that information in JTIMS lacked consistency in the level of detail provided, sometimes making it difficult to coordinate training with other combatant commands or extract pertinent information about exercises from the system that would be helpful in planning other exercises. Consequently, to help improve the consistency and standardization of information across combatant commands, the Joint Staff published a user guide for JTIMS.
Regarding the Execution Management System, a web-based database DOD uses to track the most recent funding execution data for the Joint Exercise Program, GAO found that DOD does not have assurance that the system produces quality information because supporting documentation is not consistently uploaded into the system and, when it is uploaded, it is not reconcilable to the data entered there. Only U.S. Strategic Command and U.S. Northern Command uploaded supporting documentation for fiscal years 2013-16 as required by the Execution Management System guidance. Reviewing a nongeneralizable sample of uploaded supporting documentation for fiscal years 2014-16, GAO found that the sum of the individual expenditures reported in supporting documentation did not match corresponding total expenditures entered in the system for any of the four combatant commands included in GAO's review. Further, the four combatant commands GAO visited, the Office of the Chairman of the Joint Chiefs of Staff, and the Office of the Assistant Secretary of Defense for Readiness had not implemented effective internal controls similar to those identified in the Standards for Internal Control in the Federal Government to ensure the completeness and accuracy of financial information captured for the Joint Exercise Program. Without such internal controls, DOD and other key decision makers may not have the financial information of sufficient quality to defend the Joint Exercise Program's budget.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD comply with current guidance to upload supporting documentation in the Execution Management System and implement effective internal controls to ensure the completeness and accuracy of financial information. DOD partially concurred with both recommendations, noting existing controls in other related systems of record. GAO believes the recommendations remain valid, as discussed in this report.

section_title: Letter
paragraphs: The Joint Exercise Program is a principal means for combatant  commanders to maintain trained and ready forces, exercise contingency  and theater security cooperation plans, and conduct joint and  multinational training. In fiscal year 2016, as part of this program, the  Department of Defense’s (DOD) combatant commands participated in  more than 150 joint training exercises that ranged in size from small- scale events to major joint and multilateral exercises. For this same  period, funding for these exercises was approximately $600 million.  Although these exercises have the primary purpose of training U.S.  forces, DOD has stated they can also help build partner-nation capacity  and strengthen alliances. Furthermore, joint exercises are designed to  integrate and synchronize interdependent capabilities—such as  intelligence, surveillance, and reconnaissance; electronic warfare; and  special operations forces—across multiple services and commands. DOD  officials stated that participation in joint exercises enables the military  services to build trust and relationships with one another, U.S. allies, and  potential partners while developing the skills necessary to operate in the  joint environment. For example, in fiscal year 2015, U.S. Northern  Command (NORTHCOM) conducted a joint exercise referred to as  Ardent Sentry that trained federal, state, local, and tribal governments  and private and nonprofit organizations on how to respond to a natural  disaster. DOD uses two key information technology systems to manage  its Joint Exercise Program.
House Report 114-102, accompanying a bill for the National Defense  Authorization Act for Fiscal Year 2016, included a provision for us to  review DOD’s Joint Exercise Program. Accordingly, this report (1)  describes guidance DOD has developed for its Joint Exercise Program  and DOD’s approach to assess the effectiveness of the program and (2)  evaluates the extent to which DOD uses two key information systems— the Joint Training Information Management System (JTIMS) and the  Execution Management System–to manage its Joint Exercise Program.
To describe guidance that DOD has developed for its Joint Exercise  Program and DOD’s approach to assess the effectiveness of the  program, we reviewed guidance developed by the Office of the Assistant  Secretary of Defense for Readiness, the Office of the Chairman of the  Joint Chiefs of Staff, and the combatant commands for joint exercises and  other joint training events funded by the Combatant Commanders  Exercise Engagement and Training Transformation account. We  compared guidance issued by the Office of the Assistant Secretary of  Defense for Readiness with that issued by the Office of the Chairman of  the Joint Chiefs of Staff to identify similarities and differences. In addition,  we reviewed guidance on how assessments should be conducted for joint  exercises. Specifically, we reviewed DOD’s approach for making  performance measures specific, measurable, achievable, realistic, and  time-phased (commonly referred to as the SMART rubric) to assess the  return on investment for the Joint Exercise Program. We did not review or  evaluate the quality of any assessments that DOD has conducted for its  joint exercises. Further, we interviewed senior officials from the Office of  the Assistant Secretary of Defense for Readiness, including the Joint  Assessment and Enabling Capability office, the Office of the Chairman of  the Joint Chiefs of Staff, as well as officials from the following four  combatant commands that we visited: NORTHCOM, U.S. Pacific  Command (PACOM), U.S. Strategic Command (STRATCOM), and U.S.  Transportation Command (TRANSCOM). We selected these four  combatant commands for our site visits to achieve a mix of geographical  and functional commands. In addition to using the type of command to  make our selections, we also selected commands based on the funds  that had been apportioned in fiscal year 2016 from the Combatant  Commanders Exercise Engagement and Training Transformation account  to those combatant commands.
To evaluate DOD’s use of JTIMS and the Execution Management  System to manage the Joint Exercise Program, we reviewed guidance  for JTIMS and the Execution Management System; observed data  associated with a nongeneralizable sample of joint exercises maintained  in JTIMS; and reviewed and analyzed a nongeneralizable sample of  cumulative financial data and supporting documentation, if any, entered  by combatant command users in the Execution Management System for  the Joint Exercise Program during fiscal years 2013-16 to examine the  internal controls that were in place. We reviewed supporting  documentation uploaded into the Execution Management System to  determine whether key controls were in place to ensure that the system  maintained quality information, as defined in the Standards for Internal  Control in the Federal Government. Additionally, we spoke with  cognizant officials from the four combatant commands we visited, the  Office of the Chairman of the Joint Chiefs of Staff, and the Office of the  Assistant Secretary of Defense for Readiness about the systems used to  manage DOD’s Joint Exercise Program. We also reviewed the FIAR  plan— DOD’s strategic plan and management tool for guiding,  monitoring, and reporting on the department’s ongoing financial  management improvement efforts and for communicating the  department’s approach to addressing its financial management  weaknesses and achieving financial statement audit readiness— and  guidance. Appendix I discusses our objectives, scope, and methodology  in greater detail.
We conducted this performance audit from June 2015 to February 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOD’s Joint Exercise Program provides an opportunity for combatant  commanders to (1) train to the mission capability requirements described  in the Joint Mission-Essential Task List and (2) support theater or global  security cooperation requirements as directed in theater or in global  campaign plans. All nine of the combatant commands, as well as the four  military services, conduct exercises as a part of the Joint Exercise  Program. The mission for the four combatant commands we visited are  as follows:
NORTHCOM conducts homeland defense, civil support, and security  cooperation to defend and secure the United States and its interests.
PACOM, with assistance from other U.S. government agencies,  protects and defends the United States, its people, and its interests. In  conjunction with its allies and partners, PACOM’s goal is to enhance  stability in the Indo-Asia-Pacific region by promoting security  cooperation, responding to contingencies, deterring aggression, and  when necessary, fighting to win.
STRATCOM conducts global operations in coordination with other  combatant commands, military services, and appropriate U.S.  government agencies to deter and detect strategic attacks against the  United States, its allies, and partners.
TRANSCOM provides a full-spectrum of global mobility solutions and  related enabling capabilities for supported customers’ requirements in  peace and war.
The key players with roles and responsibilities in the Joint Exercise  Program are as follows:
Principal Deputy Assistant Secretary of Defense for Readiness,  whose responsibilities include administering the Combatant  Commanders Exercise Engagement and Training Transformation  account;
Director for Joint Force Development Joint Staff (J7), whose  responsibilities include managing the Combatant Commanders  Exercise Engagement and Training Transformation account and  providing enabling capabilities that support combatant commands’  and the military services’ training;  combatant commands, who develop, publish, and execute command  Joint Training Plans and joint training programs for command staff  and assigned forces; and military services, whose responsibilities include providing trained and  ready forces for joint employment and assignment to combatant  commands.
In fiscal year 2016, the Combatant Commanders Exercise Engagement  and Training Transformation account provided approximately $600 million  dollars to fund more than 150 training events. Funding from this account  covers items such as personnel travel and per diem for planning  conferences and exercise support events, transportation of cargo, airlift,  sealift and port handling, intra-theater transportation for participating  units, consultant advisory and assistance service, equipment and  supplies, and operation and maintenance for training support facilities and  equipment. From fiscal year 2013 through fiscal year 2016, funding for  this account decreased by about $149 million, or by 20 percent, while the  number of exercises conducted remained relatively unchanged (see fig.  1).
DOD officials told us that in part as a result of reduced funding for the  Joint Exercise Program they have at times reduced the scope of  exercises or sought alternative methods, such as relying on organic lift  capabilities of Service components or partnering with another combatant  command to execute exercises. Other factors that could impact the ability  of combatant commands to execute exercises include the availability of  forces; diplomatic (political and military) considerations; and real world  events, such as natural disasters. Though DOD officials stated that these  factors are largely outside of the sphere of combatant commander  influence and therefore are not included in the original planning of the  exercises, officials stated that they use various approaches to try to  mitigate the effect these factors have on their ability to carry out their  respective joint exercise programs. If these factors cannot be mitigated,  the combatant command might cancel a joint exercise, a mitigation  strategy of last resort.
subsections: section_title: DOD Has Developed Guidance for the Joint Exercise Program and Has Implemented an Approach to Assess Program Effectiveness DOD Has Developed a Body of Guidance for the Joint Exercise Program and Is Working to Update a Key Guidance Document That Is Outdated
paragraphs: DOD has developed a body of guidance for the Joint Exercise Program  and is working to update a key outdated guidance document that  identifies overarching roles and responsibilities for military training in  accordance with a congressional requirement in a House Committee on  Armed Services report accompanying the National Defense Authorization  Act for Fiscal Year 2017. DOD-wide guidance, policies, and procedures  addressing various aspects of the Joint Exercise Program are contained  in the following documents:
DODD 1322.18, Military Training, (Jan. 13, 2009), is the overarching  guidance for military training and identifies the roles and  responsibilities for training military individuals; units; DOD civilian  employees; and contractors, among others.
The Program Goals and Objectives document provides guidance for  all programs and activities that utilize funds from the Combatant  Commanders Exercise Engagement and Training Transformation  account.
CJCSI 3500.01H, Joint Training Policy for the Armed Forces of the  United States, (April 25, 2014), establishes guidance for the Joint  Training System—an integrated, requirements-based, four-phased  approach that is used to align a combatant commander’s Joint  Training Strategy with assigned missions to produce trained and  ready individuals, staff, and units. The Joint Training System is used  by combatant commanders to execute the Joint Exercise Program as  shown in figure 2. See appendix II for a more detailed description of  the Joint Training System.
CJCSN 3500.01, 2015-2018 Chairman’s Joint Training Guidance,  (Oct. 30, 2014), provides the Office of the Chairman of the Joint  Chiefs of Staff’s joint training guidance to all DOD components for the  planning, execution, and assessment of joint individual and collective  training for fiscal years 2015 through 2018.
CJCSM 3500.03E, Joint Training Manual for the Armed Forces of the  United States, (April 20, 2015), provides guidance and procedures for  the Joint Training System. Specifically, it focuses on determining joint  training requirements, planning and executing joint training, and  assessing joint training.
CJCSM 3511.01, Joint Training Resources of the Armed Forces of the  United States, (May 26, 2015), provides detailed guidance on joint  funding, joint transportation, and joint training support resources for  joint training exercises.
CJCSI 3150.25F, Joint Lessons Learned Program, (June 26, 2015),  provides guidance for gathering, developing, and disseminating joint  lessons learned for the armed forces for joint training exercises.
Each of the combatant commands we visited had developed their own  implementation guidance, which is consistent with DOD’s guidance for  the Joint Exercise Program.
While DOD has a body of guidance for the Joint Exercise Program,  DODD 1322.18 – key overarching guidance for military training that  identifies roles and responsibilities for training, including the for the Joint  Exercise Program–is outdated. Specifically, this directive assigns  significant roles and responsibilities relevant to the Joint Exercise  Program to U.S. Joint Forces Command, a combatant command that  has not existed since August 2011. For example, according to DODD  1322.18, U.S. Joint Forces Command is responsible for working through  the Office of the Chairman of the Joint Chiefs of Staff to manage joint  force training, accredit joint training programs for designated joint tasks,  and provide Combatant Commanders Exercise Engagement and Training  Transformation funds to support the Joint Exercise Program. According to  subsequent guidance issued in April 2014, the Office of the Chairman of  the Joint Chiefs of Staff was assigned the roles and responsibilities  formerly performed by U.S. Joint Forces Command. Additionally, the  Office of the Assistant Secretary of Defense for Readiness, instead of  U.S. Joint Forces Command, now administers the Combatant  Commanders Exercise Engagement and Training Transformation  account, which funds the Joint Exercise Program.
In House Report 114-537, the House Committee on Armed Services  also noted that DODD 1322.18 is outdated and does not account for  significant organizational changes that have occurred within the  department—specifically, the disestablishment of U.S. Joint Forces  Command and the establishment of the Assistant Secretary of Defense  for Readiness. Consequently, the report directs DOD to update its  guidance and brief the committee on its progress updating the guidance  by December 1, 2016. According to an official from Office of the Assistant  Secretary of Defense for Readiness, the office responsible for DODD  1322.18, the department is aware that the directive is outdated and is  working on updating it but is unsure of when the update process will be  completed. Specifically, according to this DOD official, the department is  working to determine whether the directive can be updated through an  administrative update, which requires less coordination and time to  process than doing so through a total reissuance of guidance. When DOD  completes the update and includes information on current roles and  responsibilities, the key guidance regarding the Joint Exercise Program  should be consistent with other guidance.
section_title: DOD Has Implemented an Approach Intended to Assess the Effectiveness of the Joint Exercise Program
paragraphs: DOD has implemented an approach to assess the return on investment  for the Joint Exercise Program. The Director of the Joint Assessment  and Enabling Capability office stated that officials from that office provide  the combatant commands with guidance for how to develop performance  measures to assess the effectiveness of the Joint Exercise Program.  Specifically, the combatant commands, in conjunction with this office,  develop performance measures using an approach that is aimed at  ensuring the performance measures are specific, measurable,  achievable, realistic, and time-phased (commonly referred to as the  SMART rubric). The Director of the Joint Assessment and Enabling  Capability office reviews the performance measures created by the  combatant commands against the SMART rubric and provides input and  coaching on improving the measures through an ongoing and  collaborative process. See appendix III for a more detailed explanation of  DOD’s approach for assessing individual joint exercises.
The Joint Assessment and Enabling Capability office is working with  individual combatant commands to develop measures to assess the  return on investment of the Joint Exercise Program using the SMART  rubric approach. For example, NORTHCOM officials told us that they are  working with the Joint Assessment and Enabling Capability office to  develop a better method to measure the return on investment for  NORTHCOM joint exercises because the ones they currently use, such  as the number of joint mission-essential tasks in an exercise, do not  reveal any information that would be helpful for decision making. The  officials stated that they are still trying to determine the threshold for the  amount of information that is necessary to measure return on investment  for their joint exercises. Further, officials stated that they were drafting  performance measures to assess return on investment to submit to their  leadership for approval. Additionally, TRANSCOM officials told us that  they too are working with the Joint Assessment and Enabling Capability  office but have not yet determined how to effectively gauge return on  investment for training dollars spent on its exercises.
According to DOD and combatant command officials we interviewed,  readiness is their key performance measure and they have ongoing  efforts to develop more tangible, quantifiable measures to determine the  return on investment for conducting exercises. However, according to  combatant command officials, return on investment is sometimes  intangible and may not be seen immediately. Officials stated that it could  take years to recognize the return on investment for conducting an  exercise. For example, PACOM officials told us that they conducted two  multinational planning exercises and a multinational force standard  operating procedures workshop designed to increase the speed of initial  response forces to an emergent issue and enhance relationships with  partner countries for several years. According to PACOM officials, the  return on that investment, however, was not realized until April 25, 2015,  when a region northwest of Kathmandu, Nepal, was devastated by a 7.8  magnitude earthquake and the after-action report for that earthquake  indicated that PACOM’s exercises were vital in preparing the Nepal Army  for its response.
section_title: DOD Uses Two Key Information Technology Systems to Manage the Joint Exercise Program, but the Data in the System That Tracks the Most Recent Funding Execution Data Does Not Have Required Supporting Documentation and Its Data Quality Is Questionable
paragraphs: DOD uses two key information technology systems—JTIMS and the  Execution Management System—to manage the execution of the Joint  Exercise Program, but DOD does not have assurance that the Execution  Management System produces quality information.
subsections: section_title: DOD Uses JTIMS to Plan and Manage Training and Has Developed a User Guide to Standardize Information
paragraphs: DOD uses JTIMS as the system of record for the Joint Exercise Program  and combatant commanders plan and manage their joint training  exercises through JTIMS. JTIMS automates the management of joint  exercise training data through a web-based system and supports the  application of the four phases of the Joint Training System. Specifically,  JTIMS, which is managed by the Office of the Chairman of the Joint  Chiefs of Staff, is used, among other things, to (1) request and track  forces for joint training exercises, (2) publish the Joint Training Plan, (3)  document and manage joint training programs, and (4) capture the  assessments of exercises. Chairman of the Joint Chiefs of Staff policy  requires the use of JTIMS for a number of fields. For example, guidance  requires, among other things, that the combatant commands enter key  information about a training exercise, such as its objectives, intended  audience (i.e., the joint forces being trained), lessons learned, and  observations on performance, and costs. However, the extent to which  these fields are used, and the quality of the data entered varies by  combatant command. Combatant commands and other DOD entities rely  on the information entered in JTIMS to both conduct their exercises and  participate in exercises sponsored by other combatant commands.
However, during the course of our review, we were informed of and  observed significant variation in the type and quality of information  entered in JTIMS. For example, officials from one combatant command  stated that an exercise description entered by another combatant  command did not provide sufficient detail, therefore making it difficult to  understand the focus of the exercise. In addition, TRANSCOM officials  randomly selected exercises in JTIMS to show us the type of information  entered in the system and we noted that the level of detail provided  sometimes varied significantly by combatant command. Furthermore,  officials from two of the four combatant commands we visited stated that  sometimes the information captured in JTIMS is not useful and could  negatively affect their ability to coordinate training with other combatant  commands or extract pertinent information about exercises from the  system that would be helpful in planning them.
According to an Office of the Chairman of the Joint Chiefs of Staff official,  it is important that combatant commands enter information in JTIMS in a  consistent and standardized manner so that the information is easily  understood and useful for all joint exercise training participants and  planners. The Office of the Chairman of the Joint Chiefs of Staff and  combatant command officials told us that the lack of standardized  information in JTIMS is due to the absence of detailed instructions in  guidance on inputting information into JTIMS. Consequently, to help  improve the consistency and standardization of information across  combatant commands, the Office of the Chairman of the Joint Chiefs of  Staff published a user guide for JTIMS that is intended to mitigate  inconsistencies in the information entered there, standardize the use of  the system across DOD, and improve the overall understanding of the  system. According to this official, the user guide was completed in  October 2016. The Office of the Chairman of the Joint Chiefs Staff plans  to periodically update the user guide to keep pace with joint training policy  updates, JTIMS software upgrades, and joint training enterprise business  rule modifications. In addition to providing step-by-step instructions on  using JTIMS, the guide also provides examples of the type of information  that should be entered in specified fields. Such information should help  improve the overall understanding of and bring consistency to the use of  JTIMS across the combatant commands.
section_title: DOD Uses the Execution Management System to Oversee and Report the Most Recent Funding Execution Data, but Data Quality Has Been Lacking Due to Weaknesses in Internal Controls
paragraphs: DOD uses the Execution Management System, a web-based database,  to track and oversee the most recent execution performance data,  hereafter referred to as data, for the Joint Exercise Program. The  Execution Management System is intended to capture the most recent  data for the Joint Exercise Program. According to officials, it is important  to have accurate and current data in the Execution Management System  because it provides instant status of the over- and underexecution of  funds for the Joint Exercise Program, which is critical to the efficient and  effective execution of the Joint Exercise Program. Moreover, officials from  the Assistant Secretary of Defense for Readiness office stated that data  from this system are used to report how funds are being expended for the  Joint Exercise Program to both DOD decision makers and Congress.
In April 2016, the Office of the Assistant Secretary of Defense for  Readiness issued guidance on the use of the Execution Management  System to the combatant commands. This guidance, referred to as the  Execution Management System Standard Operating Procedure and User  Guide, states that Joint Exercise Program managers are required to (1)  enter the most recent obligation and expenditure amounts for any  transactions funded through the Combatant Commanders Exercise  Engagement and Training Transformation account on a monthly basis  and (2) upload supporting documentation for transactions. The guide also  specifies the type of supporting documents that should be uploaded into  the Execution Management System, such as awarded contracts, invoices,  and travel payments. Prior to issuing guidance in April 2016, an official  responsible for administering the Combatant Commanders Exercise  Engagement and Training Transformation account stated that the  combatant commands were informed of the requirement to upload  supporting documentation into the Execution Management System in  2011. NORTHCOM, STRATCOM, and PACOM officials told us that they  were aware of this requirement prior to April 2016. TRANSCOM officials  initially stated that they were unaware of the requirement; however, our  review of the Execution Management System revealed that they were  uploading supporting documentation for some fiscal years prior to the  guidance being issued.
During our review of the Execution Management System, we found that  the combatant commands we visited had not fully implemented the  guidance and that the quality of information in the system was  questionable. Specifically, we found that:
The Execution Management System is missing supporting  documentation. Based on our review, we found that two— STRATCOM and NORTHCOM—of the four combatant commands we  visited had uploaded supporting documentation, as required by the  Execution Management System guidance, for fiscal years 2013-16. A  third combatant command, TRANSCOM, uploaded supporting  documentation for fiscal years 2013, 2014, and 2016, but did not  upload supporting documentation for fiscal year 2015. The fourth  combatant command, PACOM, did not upload supporting  documentation for fiscal years 2013, 2014, and 2015, but began  uploading supporting documentation in August 2016 for fiscal year  2016 after we informed an official in the Office of the Assistant  Secretary of Defense for Readiness that the command had not been  uploading supporting documentation in accordance with the Execution  Management System guidance. TRANSCOM and PACOM officials  stated that one of the reasons they did not upload supporting  documentation, as required by guidance, was due to the volume of  travel and other related documents generated in executing joint  training exercises. Officials stated that it was overly burdensome to  upload all of these documents. Nonetheless, an official from the Office  of the Assistant Secretary of Defense for Readiness stated the  combatant commands need to do their due diligence in uploading  supporting documentation in order to ensure proper accountability of  Combatant Commanders Exercise Engagement and Training  Transformation funds. This official further stated that efforts are  underway, that includes establishing a new method for how funds are  distributed to stakeholders, to identify approaches that will reduce the  data entry burden at the stakeholder level.
Documentation for expenditures uploaded into the Execution  Management System did not match reported total expenditures  for any of the four combatant commands we visited. Based on our  review of a nongeneralizable sample of supporting documentation  for fiscal years 2014 through 2016 that was uploaded into the  Execution Management System, we found that the sum of the  individual expenditures reported in supporting documentation did not  match the corresponding total expenditures entered in the system for  any of the four combatant commands we visited. According to one  combatant command official familiar with this system, individual  expenditures reported in supporting documents should be  reconcilable to yearly cumulative totals for expenditures. However,  when we attempted to link the sum of individual expenditures reported  in uploaded supporting documentation to total expenditures data  entered into the Execution Management System by combatant  command officials, we were unable to do so for three of the four  combatant commands we visited. For example, in fiscal year 2015,  NORTHCOM uploaded more than 100 documents that supported how  funds were obligated or committed. Our review found that the  uploaded documentation supported approximately $12.7 million in  funds that were committed. However, the figure entered in the  Execution Management System was about $11.9 million. Similarly, in  fiscal year 2014, TRANSCOM supporting documentation showed that  commitments totaled approximately $66.8 million while the figure  entered in the Execution Management System was approximately  $4.6 million. Officials stated that the reason that the supporting  documentation does not match the figures entered in the Execution  Management System is that some supporting documentation had not  been uploaded. Nonetheless, the inability to reconcile supporting  documentation with the expenditures entered in the Execution  Management System undermines the quality of the data in the system  and inhibits DOD decision makers, particularly those in the Office of  the Assistant Secretary of Defense for Readiness, from providing  adequate oversight of how funds are being expended in support of the  Joint Exercise Program goals. Moreover, the inconsistent uploading of the required supporting documentation and difficulty in reconciling  individually reported transactions with cumulative values entered into  the system suggests that weaknesses exist in the Execution  Management System data entry procedures, which impacts the  quality of the data entered in the system. Therefore, it calls into  question the use of the Engagement Management System which,  according to DOD officials, had been established to provide real-time,  accurate information on the execution of Joint Exercise Program  funds to decision makers.
DOD has not implemented key processes to help ensure that the  Execution Management System produces quality information.  Further weakening the quality of the reporting, tracking, and  reconciliation of data recorded in the Execution Management System  is that none of the four combatant commands we visited, the Office of  the Chairman of the Joint Chiefs of Staff, and the Office of the  Assistant Secretary of Defense for Readiness had instituted key  systemic processes to help ensure that the data entered in the  Execution Management System produce quality information—that is,  information that is appropriate, current, complete, accurate,  accessible, and timely. Standards for Internal Control in the Federal  Government states that a variety of control activities should be used  for information systems to support the completeness, accuracy, and  validity of information processing, and the production of quality  information. In addition, management should evaluate information  processing to ensure that it is complete, accurate, and valid. Further,  these standards state that appropriate documentation of transactions  should be readily available for examination. Using these internal  controls could reduce to an acceptable level the risk that a significant  mistake could occur and remain undetected and uncorrected.  Individuals from all four of the combatant commands we visited stated  that only one person at their combatant command was responsible for  entering data into the Execution Management System for their  respective command and that, although they believed their entries  were reliable, no quality assurance oversight was conducted on their  work. An official from the Office of the Assistant Secretary of Defense  for Readiness stated that periodic reviews are conducted on data  entered in the Execution Management System but that these reviews  are mainly focused on the execution rates of funds and not on  whether the data entered produces quality information. Further,  according to an official from the Office of the Chairman of the Joint  Chiefs of Staff, the checks they perform on the data entered in the  Execution Management System are similar to those conducted by the  Office of the Assistant Secretary of Defense for Readiness, in that  they are focused on whether or not monthly expenditures have been  entered into the system in order to ensure that monthly benchmarks  are met and less on whether or not the data entered produces quality  information. According to officials from two of the four combatant  commands we visited, sometimes they receive phone calls from the  Office of the Chairman of the Joint Chiefs of Staff or the Office of the  Assistant Secretary of Defense for Readiness to validate certain data  entries because they seemed erroneous based on an informal review.  However, no officials at the combatant commands we visited, the  Office of the Chairman for the Joint Chiefs of Staff, or the Office of the  Assistant Secretary of Defense for Readiness could demonstrate  systemic processes for ensuring that the Execution Management  System produced quality information. The absence of quality  assurance processes can affect the quality of the information  produced by the system that DOD uses to determine its most recent  execution rates and defend the Joint Exercise Program’s budget.
As previously discussed, the combatant commands are not following  guidance requiring them to upload supporting documentation, and DOD  lacks effective internal controls to help ensure the reliability of the data in  the system. DOD officials acknowledged the issues we identified  regarding inadequate supporting documentation and data reliability within  the Execution Management System. A senior DOD official from the Office  of the Assistant Secretary of Defense, Readiness (Resources) stated that  DOD plans to address these control weaknesses with respect to the  Combatant Commander Exercise Engagement and Training  Transformation account for the Joint Exercise Program as part of its  implementation of DOD’s FIAR Guidance beginning in fiscal year 2018 to  ensure that the account is audit ready. DOD established the FIAR Plan  as its strategic plan and management tool for guiding, monitoring, and  reporting on the department’s ongoing financial management  improvement efforts and for communicating the department’s approach to  addressing its financial management weaknesses and achieving financial  statement audit readiness. To implement the FIAR Plan, the DOD  Comptroller issued the FIAR Guidance, which provides a standard  methodology for DOD components to follow to assess their financial  management processes and controls and to develop and implement  financial improvement plans. These plans, in turn, are intended to provide  a framework for planning, executing, and tracking essential steps and  related supporting documentation needed to achieve auditability. We  believe that if DOD appropriately follows the steps outlined in FIAR  guidance when executing the Combatant Commander Exercise  Engagement and Training Transformation account, it may help improve  the quality of funds execution data from this account and make the  account audit ready.
However, as previously stated, FIAR guidance will not be implemented  with the Joint Exercise Program until fiscal year 2018 and the  effectiveness of the guidance cannot be fully determined until after that  time. In the meanwhile, according to a senior DOD official, DOD plans to  continue using the Execution Management System which is intended to  capture the most recent data for the Joint Exercise Program and inform  management decision-making regarding joint exercise investments.  Without ensuring the required supporting documentation is uploaded and  implementing effective internal controls to ensure that data entered in the  Execution Management System produces quality information, DOD and  other key decision makers may not have the correct financial execution  information to defend the Joint Exercise Program’s budget.
section_title: Conclusions
paragraphs: DOD has developed a body of guidance for the Joint Exercise Program.  In addition, DOD has implemented an approach to develop performance  measures to assess the effectiveness of the Joint Exercise Program.  Further, DOD uses JTIMS and the Execution Management System to  manage the Joint Exercise Program. JTIMS is the system of record for  executing the Joint Exercise Program and the Office of the Chairman of  the Joint Chiefs of Staff officials developed a user guide intended to help  bring more standardization to the system, thereby making the information  more useful to other combatant commands. The Execution Management  System is used to oversee and report on most recent execution  performance data for Joint Exercise Program funding. However, not all of  the combatant commands were following guidance requiring them to  upload supporting documentation, making it difficult for DOD to have  oversight on expenditures for the Joint Exercise Program. Finally, DOD  and the combatant commands lack systemic processes for ensuring that  the Execution Management System produces quality information. Without  ensuring that supporting documentation is uploaded and implementing  effective internal controls to ensure the completeness and accuracy of  financial information captured for the Joint Exercise Program, DOD and  other key decision makers may not have the correct financial information  to defend the Joint Exercise Program’s budget.
section_title: Recommendations for Executive Action
paragraphs: To better ensure quality financial execution information is available to  guide the Joint Exercise Program, we recommend that the Secretary of  Defense direct the Office of the Assistant Secretary of Defense for  Readiness to take the following two actions: direct the combatant commanders to take steps to comply with current  Execution Management System guidance to upload supporting  documentation that is reconcilable to funds executed from the  Combatant Commanders Exercise Engagement and Training  Transformation account; and as the department implements financial improvement plans in  accordance with the FIAR guidance, it should include specific internal  control steps and procedures to address and ensure the  completeness and accuracy of information captured for the Joint  Exercise Program’s Combatant Commanders Exercise Engagement  and Training Transformation account.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for review and comment.  In its  written comments, which are summarized below and reprinted in  appendix IV, DOD partially concurred with both recommendations. DOD  also provided technical comments, which we incorporated as appropriate.
DOD partially concurred with our recommendation to direct the combatant  commanders to take steps to comply with current Execution Management  System guidance to upload supporting documentation that is reconcilable  to funds executed from the Combatant Commanders Exercise  Engagement and Training Transformation account. In its comments, DOD  stated that the Execution Management System is not a system of record  but rather a “desk-side” support tool that relies on manual inputs and  uploads and that the reconciliation of obligation and execution related to  Joint Exercise Program funding occurs elsewhere. DOD further noted that  the Office of the Assistant Secretary of Defense for Readiness (OASD(R))  issued guidance and routinely reinforces the best practices use of the  Execution Management System tool to ensure it produces quality  information. Lastly, DOD noted in its comments that that it may not  continue using the Execution Management System beyond fiscal year  2017.
We recognize that the Execution Management System is not a system of  record. However, as we also note in the report, it is a tool used by DOD to  make decisions regarding the Joint Exercise Program because the  Defense Finance and Accounting Services (DFAS) Accounting Report  Monthly 1002, the system of record, lags behind. Additionally, we  acknowledge that DOD issued guidance for the Execution Management  System in April 2016, but our work found that the guidance was not  routinely reinforced. For example, as we identified in our report, two of the  four combatant commands we visited had not, in fact, uploaded  supporting documentation, as required by the Execution Management  System guidance, for fiscal years 2013-2016. While the Execution  Management System may not be funded beyond fiscal year 2017, we  continue to believe that as long as the Execution Management System  remains in use and for the reasons discussed in the report, combatant  commanders should take the necessary steps to comply with existing  guidance that requires the uploading of supporting documentation into the  Execution Management System so that when DOD managers make  decisions regarding the Joint Exercise Program funding, they use  information from a financial data system that is reconcilable and  auditable.
DOD partially concurred with our recommendation that states that as the  department implements financial improvement plans in accordance with  the FIAR guidance, it should include specific internal control steps and  procedures to address and ensure the completeness and accuracy of  information captured for the Joint Exercise Program’s Combatant  Commanders Exercise Engagement and Training Transformation  account. In its comments, DOD described OASD(R) as having a  supporting role in the execution of FIAR plans, which are implemented by  Washington Headquarters Services and the Office of Secretary of  Defense-Comptroller, and that these agencies provide specific internal  controls, processes and procedures for ensuring completeness and  accuracy of obligation and execution data. DOD also stated that the  Execution Management System is not a component of FIAR and may not  be funded after fiscal year 2017, and that moving toward audit readiness,  necessary steps and procedures will be put into place to strengthen  auditability.
As we stated in the report, the FIAR guidance provides a standard  methodology and framework for assessing and developing a system of  internal controls to achieve auditability. However, as we recommended,  DOD still needs to implement specific internal control steps and  procedures as it implements this guidance to ensure the completeness  and accuracy of the Joint Exercise Program’s Combatant Commanders  Exercise Engagement and Training Transformation account’s financial  information. Further, as we reported the FIAR guidance will not be  implemented in the Joint Exercise Program until fiscal year 2018 and the  effectiveness of the guidance cannot be fully determined until after that  time. Accordingly, we continue to believe that the recommendation  remains valid.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense; the Under Secretary of Defense  for Personnel and Readiness; the Chairman of the Joint Chiefs of Staff;  and the Commanders of U.S. Northern Command, U.S. Pacific  Command, U.S. Strategic Command, and U.S. Transportation Command.  In addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report (1) describes guidance the Department of Defense (DOD) has  developed for its Joint Exercise Program and DOD’s approach to assess  the effectiveness of the program and (2) evaluates the extent to which  DOD uses two key information systems—the Joint Training Information  Management System (JTIMS) and the Execution Management System–to  manage the Joint Exercise Program.
subsections: section_title: Office of the Assistant Secretary of Defense for Readiness
paragraphs: DODD 1322.18, Military Training (January 13, 2009)
section_title: Office of the Chairman of the Joint Chiefs of Staff
paragraphs: CJCSI 3500.01H, Joint Training Policy for the Armed Forces of the  United States (April 25, 2014)
CJCSM 3150.25A, Joint Lessons Learned Program (September 12,  2014)
CJCSN 3500.01, 2015-2018 Chairman’s Joint Training Guidance  (October 30, 2014)
CJCSM 3500.03E, Joint Training Manual for the Armed Forces of the  United States (April 20, 2015)
CJCS Guide 3501, The Joint Training System: A Guide for Senior  Leaders (May 5, 2015)
CJCSM 3511.01H, Joint Training Resources for the Armed Forces of  the United States (May 26, 2015)
CJCSI 3150.25F, Joint Lessons Learned Program (June 26, 2015)
section_title: U.S. Northern Command
paragraphs: NORAD and NORTHCOM, Joint Training System (JTS) Handbook  (May 1, 2013)
NORAD and NORTHCOM Instruction 16-166, Lessons Learned  Program and Corrective Action Program (September 19, 2013)
NORAD and NORTHCOM, JTPs, (December 10, 2015)
section_title: U.S. Pacific Command
paragraphs: PACOM Instruction 0509.1, Joint Lessons Learned and Issue  Resolution Program (April 7, 2010)
PACOM Instruction 0508.12, Joint Training Enterprise in U.S. Pacific  Command (October 15, 2012)
section_title: U.S. Strategic Command
paragraphs: SI 508-09, Exercise Program (May 3, 2013)
SI 509-01, After Action, Issue Solution/Resolution and Lessons  Learned Program, October 14, 2015)
SI 508-03, JTIMS Procedures (November 8, 2015)
section_title: U.S. Transportation Command
paragraphs: USTRANSCOM Pamphlet 38-1, Organization and Functions (May 1,  2008)
USTRANSCOM Instruction 36-13, Training, Education, and  Professional Development Program (May 23, 2013)
USTRANSCOM Instruction 36-36, Joint Training and Education  Program (August 29, 2014)
USTRANSCOM Instruction 10-14, Joint Lessons Learned Program  (November 9, 2015)
We judgmentally selected these combatant commands for our site visits  to achieve a mix of geographical and functional commands, as well as the  funds that had been apportioned to the combatant commands in fiscal  year 2016 from the Combatant Commanders Exercise Engagement and  Training Transformation account, size of command, and location. We  reviewed DOD’s approach to make performance measures specific,  measurable, achievable, realistic, and time-phased (commonly referred to  as the SMART rubric) to assess the return on investment for the Joint  Exercise Program. In addition, we reviewed performance measures  reportedly used to assess the ability of the training audience to  accomplish training objectives for exercises, as well as measures used to  assess the return on investment for conducting an exercise. We also  reviewed performance documentation and information captured in JTIMS,  as well as a nongeneralizable sample of commander summary reports or  after-action reports from seven combatant command joint exercises to  understand the content of these reports. Finally, we interviewed senior  officials from the Office of the Assistant Secretary of Defense for  Readiness, including the Joint Assessment and Enabling Capability  office, the Office of the Chairman of the Joint Chiefs of Staff, as well as  officials from the four selected combatant commands. We did not review  or evaluate the quality of any assessments that DOD has conducted for  its joint exercises. We reviewed the assessments to the extent that it was  possible to ensure that an assessment process existed.
To evaluate DOD’s use of JTIMS and the Execution Management  System to manage the Joint Exercise Program, we reviewed guidance for  JTIMS and the Execution Management System. In addition, we observed  data associated with a nongeneralizable sample of joint exercises  maintained in JTIMS. We also reviewed and analyzed a nongeneralizable  sample of cumulative financial data and supporting documentation, if any,  entered by combatant command users in the Execution Management  System for the Joint Exercise Program during fiscal years 2013-16 to  examine the internal controls that were in place. We reviewed a  nongeneralizable sample of supporting documentation uploaded into the  Execution Management System for fiscal years 2014 through 2016 to  make a determination about compliance with guidance issued by DOD for  the Execution Management System and the Standards for Internal  Control in the Federal Government. Further, we compared individual  transactions reported in the supporting documentation with the  corresponding cumulative data entered into the system. We also  reviewed the FIAR plan— DOD’s strategic plan and management tool for  guiding, monitoring, and reporting on the department’s ongoing financial  management improvement efforts and for communicating the  department’s approach to addressing its financial management  weaknesses and achieving financial statement audit readiness— and  guidance. Additionally, we spoke with cognizant officials from the four  combatant commands we visited, the Office of the Chairman of the Joint  Chiefs of Staff, and the Office of the Assistant Secretary of Defense for  Readiness about the systems used to execute and manage the Joint  Exercise Program. Further, we attended sessions at the 3-day Annual  Review for the Combatant Commanders Exercise Engagement and  Training Transformation Enterprise on the budget for fiscal years 2018  through 2022. We attended sessions that were most pertinent to this  engagement. For example, since the services were not included in the  scope of our review, we did not attend their sessions.
We conducted site visits to collect testimonial and documentary evidence  about DOD’s Joint Exercise Program at the following locations:
Cost Assessment and Program Evaluation Office, Arlington, Virginia
Force Readiness and Training in the Office of the Assistant Secretary  of Defense for Readiness, Arlington, Virginia  Joint Assessment and Enabling Capability office, Alexandria, Virginia  Joint Staff (J7), Arlington, Virginia  Joint Staff (J7) Suffolk, Virginia
U.S. Northern Command, Peterson Air Force Base, Colorado
U.S. Pacific Command, Camp H. M. Smith, Hawaii
U.S. Strategic Command, Offutt Air Force Base, Nebraska
U.S. Transportation Command, Scott Air Force Base, Illinois  We conducted this performance audit from June 2015 to February 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: The Process (the Joint Training System) Used to Conduct Joint Exercises
paragraphs: Guidance from the Office of the Chairman of the Joint Chiefs of Staff  outlines the process that is used by the combatant commands to develop  joint training programs, plan and execute joint training, and assess  training for the Department of Defense’s (DOD) Joint Exercise Program.  This process, referred to in the guidance as the Joint Training System, is  characterized as an integrated, requirements-based, four-phased  methodology used to align the Joint Training Strategy with assigned  missions to produce trained and ready individuals, staff, and units.  According to the guidance, the Joint Training System has four phases  through which the combatant commands execute the Joint Exercise  Program.
Phase I—Requirements. During this phase, an ordered listing of  tasks is developed describing the armed force’s ability to perform  activities or processes that combatant commanders require to execute  their assigned missions. This listing is referred to as the Universal  Joint Task List and it provides a common language to describe  warfighting requirements for combatant commanders. From this list,  the most essential mission capability tasks—mission-essential  tasks—are identified by the combatant commander. Using the  commander’s criteria, mission-essential tasks are prioritized to form  the Joint/Agency Mission-Essential Task List. In addition to combatant  commanders’ priority, key documents pertinent to U.S. national  strategy, such as the Unified Command Plan, Guidance for  Employment of the Force, and other joint doctrine, are analyzed to  determine the most essential mission capability requirements for the  combatant command. The Joint/Agency Mission-Essential Task List  provides the foundation for deriving joint training requirements used to  develop Joint Training Plans and training and exercise inputs to  theater campaign plans. Training requirements are derived from  training proficiency assessments, mission training assessments, and  lessons learned that result from the Phase IV (Assessment) of the  Joint Training System.
Phase II—Plans. The plans phase is initiated by conducting an  assessment of current capability against the Joint Mission-Essential  Task List and relevant lessons learned to identify gaps in training. To  address those gaps, the Joint Training Plan is established and  identifies who is to be trained; what they will be trained in; what the  training objectives are; and when, where, and how the training will  occur. Joint Training Plans, along with training and exercise inputs  into theater campaign plans, are developed, coordinated, and  published in the Joint Training Information Management System  (JTIMS) to identify a commander’s training guidance, audiences,  objectives, events, and support resources, and to identify the  coordination needed to attain the required levels of training  proficiency.
Phase III—Execution. During this phase, events planned in the Joint  Training Plan are conducted and the training audience’s performance  objectives are observed and evaluated. Joint training events are  developed and executed using the five-stage Joint Event Life Cycle  methodology captured and reviewed in JTIMS. Task Performance  Observations—which identify whether the training audience achieved  the stated level of performance to the standards specified in the  training objectives—and the Training Proficiency Evaluations for  each training objective associated with the training event are also  captured in JTIMS. Further, facilitated after-action reports are  developed to highlight potential issues or best practices to support the  assessments in Phase IV (Assessment). Validated observations from  the training event are exported into JTIMS.
Phase IV—Assessment. During this phase, leadership within the  combatant command determines which organizations within the  command are able to perform at the level required to meet the task  standards and which missions the command is trained to accomplish.  Assessments are a commander’s responsibility. To complete Task  Performance Assessments for each task, commanders consider Task  Performance Evaluations, lesson learned, and personal observations  of the joint training exercise. An assessment ranking of trained,  partially trained, or untrained is assigned to each task listed under a  training objective. JTIMS supports the assessment of joint training by  automating the ability of joint organizations to produce Task  Performance Assessments. The Task Performance Assessments are  analyzed to create the mission training assessment that is provided to  a combatant commander on a monthly basis. The mission training  assessment is on how well the command can execute its assigned  missions. These training assessments provide input into the next  training cycle. Lessons learned, after-action reports, and issues  requiring resolution outside of the command are identified during this  phase.
section_title: Appendix III: The Department of Defense Assesses Individual Joint Exercises Using the Joint Training System
paragraphs: Combatant command officials we visited stated that, in accordance with  guidance from the Chairman of the Joint Chiefs of Staff, they used the  Joint Training System as the process for conducting training assessments  of individual joint exercises to determine each command’s overall  readiness to perform command missions. These assessments occur  during Phases III (Execution) and IV (Assessment) of the Joint Training  System. During Phase III, for example, command trainers collect task  performance observations for each training objective identified in the Joint  Training Information Management System (JTIMS). These task  performance observations identify whether the individuals and units  participating in the training exercise achieved the level of performance  stated in standards specified in the training objectives. Training  proficiency evaluations are conducted for each training objective  associated with the exercise. During Phase IV, combatant commanders  consider the proficiency evaluations, as well as after-action and  commander summary reports, to determine a combatant command’s  ability to perform assigned missions at the minimum acceptable level  under a specified set of conditions.
According to an official from the Joint Assessment and Enabling  Capability office, a subordinate office to the Office of the Assistant  Secretary of Defense for Readiness that provides strategic- level  assessments of joint training and joint training enablers throughout the  Department of Defense (DOD), including to combatant commands,  performance measures that are specific, measurable, achievable,  realistic, and time-phased (commonly referred to as the SMART rubric)  are used to assess the Joint Exercise Program. DOD officials stated that  developing performance measures for joint exercises has not been an  easy task and that they are constantly working to improve their  performance measures. Specifically, in an effort to develop, improve, and  provide quality assurance for specific performance measures, the Joint  Assessment and Enabling Capability office works with the combatant  commands to ensure that they are using the right measures to evaluate  the training audience’s ability to perform tasks to specific standards. In  addition, the Joint Assessment and Enabling Capability office hosts  monthly meetings with combatant command stakeholders to discuss  assessment topics, including performance measures. Additionally, the  Joint Assessment and Enabling Capability office hosts at least one  working group meeting at the annual worldwide joint training conference  to conduct face-to-face discussions and reviews of assessment-related  tasks for joint training. According to an annual report from the Joint Staff  Director for Joint Force Development, the Joint Assessment and Enabling  Capability office is available to assist combatant command stakeholders  with assessment-related tasks for the Joint Exercise Program, as  requested.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contact and Staff Appendix V: GAO Contact and Staff Acknowledgments Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Guy A. LoFaro, Assistant Director; Patricia Donahue; Pamela Nicole  Harris; Amie Lesser; Sabrina Streagle; Sonja S. Ware; and Cheryl A.  Weissman made key contributions to this report.
section_title: Related GAO Products
paragraphs: Civil Support: DOD Needs to Clarify Its Roles and Responsibilities for  Defense Support of Civil Authorities during Cyber Incidents. GAO-16-332.  Washington, D.C: April, 4, 2016.
Military Base Realignments and Closures: More Guidance and  Information Needed to Take Advantage of Opportunities to Consolidate  Training. GAO-16-45. Washington, D.C.: February 18, 2016.
Operational Contract Support: Actions Needed to Enhance the Collection,  Integration, and Sharing of Lessons Learned. GAO-15-243. Washington,  D.C.: March 16, 2015.
Defense Headquarters: DOD Needs to Reevaluate Its Approach for  Managing Resources Devoted to the Functional Combatant Commands.  GAO-14-439. Washington, D.C.: June 26, 2014.
Defense Headquarters: DOD Needs to Periodically Review and Improve  Visibility of Combatant Commands’ Resources. GAO-13-293.  Washington, D.C.: May 15, 2013.
Defense Management: Perspectives on the Involvement of the  Combatant Commands in the Development of Joint Requirements.  GAO-11-527R. Washington, D.C.: May 20, 2011.
Homeland Defense: U.S. Northern Command Has a Strong Exercise  Program, but Involvement of Interagency Partners and States Can Be  Improved. GAO-09-849. Washington, D.C.: September 9, 2009.
National Preparedness: FEMA Has Made Progress, but Needs to  Complete and Integrate Planning, Exercise, and Assessment Efforts.  GAO-09-369. Washington, D.C.: April 30, 2009.
Homeland Defense: Steps Have Been Taken to Improve U.S. Northern  Command’s Coordination with States and the National Guard Bureau, but  Gaps Remain. GAO-08-252. Washington, D.C.: April 16, 2008.
Homeland Defense: U.S. Northern Command Has Made Progress but  Needs to Address Force Allocation, Readiness Tracking Gaps, and Other  Issues. GAO-08-251. Washington, D.C.: April 16, 2008.
Homeland Security: Observations on DHS and FEMA Efforts to Prepare  for and Respond to Major and Catastrophic Disasters and Address  Related Recommendations and Legislation. GAO-07-835T. Washington,  D.C.: May 15, 2007.
Military Training: Management Actions Needed to Enhance DOD’s  Investment in the Joint National Training Capability. GAO-06-802:  Washington, D.C.: August 11, 2006.
Military Training: Actions Needed to Enhance DOD’s Program to  Transform Joint Training. GAO-05-548: Washington, D.C.: June 21, 2005.