Joint Exercise Program: DOD Needs to Take Steps to Improve the Quality of Funding Data
Released: 2017-02-24T00:00:00

Letter

Background

DOD’s Joint Exercise Program provides an opportunity for combatant commanders to (1) train to the mission capability requirements described in the Joint Mission-Essential Task List and (2) support theater or global security cooperation requirements as directed in theater or in global campaign plans. All nine of the combatant commands, as well as the four military services, conduct exercises as a part of the Joint Exercise Program. The mission for the four combatant commands we visited are as follows:

NORTHCOM conducts homeland defense, civil support, and security cooperation to defend and secure the United States and its interests.

PACOM, with assistance from other U.S. government agencies, protects and defends the United States, its people, and its interests. In conjunction with its allies and partners, PACOM’s goal is to enhance stability in the Indo-Asia-Pacific region by promoting security cooperation, responding to contingencies, deterring aggression, and when necessary, fighting to win.

STRATCOM conducts global operations in coordination with other combatant commands, military services, and appropriate U.S. government agencies to deter and detect strategic attacks against the United States, its allies, and partners.

TRANSCOM provides a full-spectrum of global mobility solutions and related enabling capabilities for supported customers’ requirements in peace and war.

The key players with roles and responsibilities in the Joint Exercise Program are as follows:

Principal Deputy Assistant Secretary of Defense for Readiness, whose responsibilities include administering the Combatant Commanders Exercise Engagement and Training Transformation account;

Director for Joint Force Development Joint Staff (J7), whose responsibilities include managing the Combatant Commanders Exercise Engagement and Training Transformation account and providing enabling capabilities that support combatant commands’ and the military services’ training; combatant commands, who develop, publish, and execute command Joint Training Plans and joint training programs for command staff and assigned forces; and military services, whose responsibilities include providing trained and ready forces for joint employment and assignment to combatant commands.

In fiscal year 2016, the Combatant Commanders Exercise Engagement and Training Transformation account provided approximately $600 million dollars to fund more than 150 training events. Funding from this account covers items such as personnel travel and per diem for planning conferences and exercise support events, transportation of cargo, airlift, sealift and port handling, intra-theater transportation for participating units, consultant advisory and assistance service, equipment and supplies, and operation and maintenance for training support facilities and equipment. From fiscal year 2013 through fiscal year 2016, funding for this account decreased by about $149 million, or by 20 percent, while the number of exercises conducted remained relatively unchanged (see fig. 1).

DOD officials told us that in part as a result of reduced funding for the Joint Exercise Program they have at times reduced the scope of exercises or sought alternative methods, such as relying on organic lift capabilities of Service components or partnering with another combatant command to execute exercises. Other factors that could impact the ability of combatant commands to execute exercises include the availability of forces; diplomatic (political and military) considerations; and real world events, such as natural disasters. Though DOD officials stated that these factors are largely outside of the sphere of combatant commander influence and therefore are not included in the original planning of the exercises, officials stated that they use various approaches to try to mitigate the effect these factors have on their ability to carry out their respective joint exercise programs. If these factors cannot be mitigated, the combatant command might cancel a joint exercise, a mitigation strategy of last resort.

DOD Has Developed Guidance for the Joint Exercise Program and Has Implemented an Approach to Assess Program Effectiveness DOD Has Developed a Body of Guidance for the Joint Exercise Program and Is Working to Update a Key Guidance Document That Is Outdated

DOD has developed a body of guidance for the Joint Exercise Program and is working to update a key outdated guidance document that identifies overarching roles and responsibilities for military training in accordance with a congressional requirement in a House Committee on Armed Services report accompanying the National Defense Authorization Act for Fiscal Year 2017. DOD-wide guidance, policies, and procedures addressing various aspects of the Joint Exercise Program are contained in the following documents:

DODD 1322.18, Military Training, (Jan. 13, 2009), is the overarching guidance for military training and identifies the roles and responsibilities for training military individuals; units; DOD civilian employees; and contractors, among others.

The Program Goals and Objectives document provides guidance for all programs and activities that utilize funds from the Combatant Commanders Exercise Engagement and Training Transformation account.

CJCSI 3500.01H, Joint Training Policy for the Armed Forces of the United States, (April 25, 2014), establishes guidance for the Joint Training System—an integrated, requirements-based, four-phased approach that is used to align a combatant commander’s Joint Training Strategy with assigned missions to produce trained and ready individuals, staff, and units. The Joint Training System is used by combatant commanders to execute the Joint Exercise Program as shown in figure 2. See appendix II for a more detailed description of the Joint Training System.

CJCSN 3500.01, 2015-2018 Chairman’s Joint Training Guidance, (Oct. 30, 2014), provides the Office of the Chairman of the Joint Chiefs of Staff’s joint training guidance to all DOD components for the planning, execution, and assessment of joint individual and collective training for fiscal years 2015 through 2018.

CJCSM 3500.03E, Joint Training Manual for the Armed Forces of the United States, (April 20, 2015), provides guidance and procedures for the Joint Training System. Specifically, it focuses on determining joint training requirements, planning and executing joint training, and assessing joint training.

CJCSM 3511.01, Joint Training Resources of the Armed Forces of the United States, (May 26, 2015), provides detailed guidance on joint funding, joint transportation, and joint training support resources for joint training exercises.

CJCSI 3150.25F, Joint Lessons Learned Program, (June 26, 2015), provides guidance for gathering, developing, and disseminating joint lessons learned for the armed forces for joint training exercises.

Each of the combatant commands we visited had developed their own implementation guidance, which is consistent with DOD’s guidance for the Joint Exercise Program.

While DOD has a body of guidance for the Joint Exercise Program, DODD 1322.18 – key overarching guidance for military training that identifies roles and responsibilities for training, including the for the Joint Exercise Program–is outdated. Specifically, this directive assigns significant roles and responsibilities relevant to the Joint Exercise Program to U.S. Joint Forces Command, a combatant command that has not existed since August 2011. For example, according to DODD 1322.18, U.S. Joint Forces Command is responsible for working through the Office of the Chairman of the Joint Chiefs of Staff to manage joint force training, accredit joint training programs for designated joint tasks, and provide Combatant Commanders Exercise Engagement and Training Transformation funds to support the Joint Exercise Program. According to subsequent guidance issued in April 2014, the Office of the Chairman of the Joint Chiefs of Staff was assigned the roles and responsibilities formerly performed by U.S. Joint Forces Command. Additionally, the Office of the Assistant Secretary of Defense for Readiness, instead of U.S. Joint Forces Command, now administers the Combatant Commanders Exercise Engagement and Training Transformation account, which funds the Joint Exercise Program.

In House Report 114-537, the House Committee on Armed Services also noted that DODD 1322.18 is outdated and does not account for significant organizational changes that have occurred within the department—specifically, the disestablishment of U.S. Joint Forces Command and the establishment of the Assistant Secretary of Defense for Readiness. Consequently, the report directs DOD to update its guidance and brief the committee on its progress updating the guidance by December 1, 2016. According to an official from Office of the Assistant Secretary of Defense for Readiness, the office responsible for DODD 1322.18, the department is aware that the directive is outdated and is working on updating it but is unsure of when the update process will be completed. Specifically, according to this DOD official, the department is working to determine whether the directive can be updated through an administrative update, which requires less coordination and time to process than doing so through a total reissuance of guidance. When DOD completes the update and includes information on current roles and responsibilities, the key guidance regarding the Joint Exercise Program should be consistent with other guidance.

DOD Has Implemented an Approach Intended to Assess the Effectiveness of the Joint Exercise Program

DOD has implemented an approach to assess the return on investment for the Joint Exercise Program. The Director of the Joint Assessment and Enabling Capability office stated that officials from that office provide the combatant commands with guidance for how to develop performance measures to assess the effectiveness of the Joint Exercise Program. Specifically, the combatant commands, in conjunction with this office, develop performance measures using an approach that is aimed at ensuring the performance measures are specific, measurable, achievable, realistic, and time-phased (commonly referred to as the SMART rubric). The Director of the Joint Assessment and Enabling Capability office reviews the performance measures created by the combatant commands against the SMART rubric and provides input and coaching on improving the measures through an ongoing and collaborative process. See appendix III for a more detailed explanation of DOD’s approach for assessing individual joint exercises.

The Joint Assessment and Enabling Capability office is working with individual combatant commands to develop measures to assess the return on investment of the Joint Exercise Program using the SMART rubric approach. For example, NORTHCOM officials told us that they are working with the Joint Assessment and Enabling Capability office to develop a better method to measure the return on investment for NORTHCOM joint exercises because the ones they currently use, such as the number of joint mission-essential tasks in an exercise, do not reveal any information that would be helpful for decision making. The officials stated that they are still trying to determine the threshold for the amount of information that is necessary to measure return on investment for their joint exercises. Further, officials stated that they were drafting performance measures to assess return on investment to submit to their leadership for approval. Additionally, TRANSCOM officials told us that they too are working with the Joint Assessment and Enabling Capability office but have not yet determined how to effectively gauge return on investment for training dollars spent on its exercises.

According to DOD and combatant command officials we interviewed, readiness is their key performance measure and they have ongoing efforts to develop more tangible, quantifiable measures to determine the return on investment for conducting exercises. However, according to combatant command officials, return on investment is sometimes intangible and may not be seen immediately. Officials stated that it could take years to recognize the return on investment for conducting an exercise. For example, PACOM officials told us that they conducted two multinational planning exercises and a multinational force standard operating procedures workshop designed to increase the speed of initial response forces to an emergent issue and enhance relationships with partner countries for several years. According to PACOM officials, the return on that investment, however, was not realized until April 25, 2015, when a region northwest of Kathmandu, Nepal, was devastated by a 7.8 magnitude earthquake and the after-action report for that earthquake indicated that PACOM’s exercises were vital in preparing the Nepal Army for its response.

DOD Uses Two Key Information Technology Systems to Manage the Joint Exercise Program, but the Data in the System That Tracks the Most Recent Funding Execution Data Does Not Have Required Supporting Documentation and Its Data Quality Is Questionable

DOD uses two key information technology systems—JTIMS and the Execution Management System—to manage the execution of the Joint Exercise Program, but DOD does not have assurance that the Execution Management System produces quality information.

DOD Uses JTIMS to Plan and Manage Training and Has Developed a User Guide to Standardize Information

DOD uses JTIMS as the system of record for the Joint Exercise Program and combatant commanders plan and manage their joint training exercises through JTIMS. JTIMS automates the management of joint exercise training data through a web-based system and supports the application of the four phases of the Joint Training System. Specifically, JTIMS, which is managed by the Office of the Chairman of the Joint Chiefs of Staff, is used, among other things, to (1) request and track forces for joint training exercises, (2) publish the Joint Training Plan, (3) document and manage joint training programs, and (4) capture the assessments of exercises. Chairman of the Joint Chiefs of Staff policy requires the use of JTIMS for a number of fields. For example, guidance requires, among other things, that the combatant commands enter key information about a training exercise, such as its objectives, intended audience (i.e., the joint forces being trained), lessons learned, and observations on performance, and costs. However, the extent to which these fields are used, and the quality of the data entered varies by combatant command. Combatant commands and other DOD entities rely on the information entered in JTIMS to both conduct their exercises and participate in exercises sponsored by other combatant commands.

However, during the course of our review, we were informed of and observed significant variation in the type and quality of information entered in JTIMS. For example, officials from one combatant command stated that an exercise description entered by another combatant command did not provide sufficient detail, therefore making it difficult to understand the focus of the exercise. In addition, TRANSCOM officials randomly selected exercises in JTIMS to show us the type of information entered in the system and we noted that the level of detail provided sometimes varied significantly by combatant command. Furthermore, officials from two of the four combatant commands we visited stated that sometimes the information captured in JTIMS is not useful and could negatively affect their ability to coordinate training with other combatant commands or extract pertinent information about exercises from the system that would be helpful in planning them.

According to an Office of the Chairman of the Joint Chiefs of Staff official, it is important that combatant commands enter information in JTIMS in a consistent and standardized manner so that the information is easily understood and useful for all joint exercise training participants and planners. The Office of the Chairman of the Joint Chiefs of Staff and combatant command officials told us that the lack of standardized information in JTIMS is due to the absence of detailed instructions in guidance on inputting information into JTIMS. Consequently, to help improve the consistency and standardization of information across combatant commands, the Office of the Chairman of the Joint Chiefs of Staff published a user guide for JTIMS that is intended to mitigate inconsistencies in the information entered there, standardize the use of the system across DOD, and improve the overall understanding of the system. According to this official, the user guide was completed in October 2016. The Office of the Chairman of the Joint Chiefs Staff plans to periodically update the user guide to keep pace with joint training policy updates, JTIMS software upgrades, and joint training enterprise business rule modifications. In addition to providing step-by-step instructions on using JTIMS, the guide also provides examples of the type of information that should be entered in specified fields. Such information should help improve the overall understanding of and bring consistency to the use of JTIMS across the combatant commands.

DOD Uses the Execution Management System to Oversee and Report the Most Recent Funding Execution Data, but Data Quality Has Been Lacking Due to Weaknesses in Internal Controls

DOD uses the Execution Management System, a web-based database, to track and oversee the most recent execution performance data, hereafter referred to as data, for the Joint Exercise Program. The Execution Management System is intended to capture the most recent data for the Joint Exercise Program. According to officials, it is important to have accurate and current data in the Execution Management System because it provides instant status of the over- and underexecution of funds for the Joint Exercise Program, which is critical to the efficient and effective execution of the Joint Exercise Program. Moreover, officials from the Assistant Secretary of Defense for Readiness office stated that data from this system are used to report how funds are being expended for the Joint Exercise Program to both DOD decision makers and Congress.

In April 2016, the Office of the Assistant Secretary of Defense for Readiness issued guidance on the use of the Execution Management System to the combatant commands. This guidance, referred to as the Execution Management System Standard Operating Procedure and User Guide, states that Joint Exercise Program managers are required to (1) enter the most recent obligation and expenditure amounts for any transactions funded through the Combatant Commanders Exercise Engagement and Training Transformation account on a monthly basis and (2) upload supporting documentation for transactions. The guide also specifies the type of supporting documents that should be uploaded into the Execution Management System, such as awarded contracts, invoices, and travel payments. Prior to issuing guidance in April 2016, an official responsible for administering the Combatant Commanders Exercise Engagement and Training Transformation account stated that the combatant commands were informed of the requirement to upload supporting documentation into the Execution Management System in 2011. NORTHCOM, STRATCOM, and PACOM officials told us that they were aware of this requirement prior to April 2016. TRANSCOM officials initially stated that they were unaware of the requirement; however, our review of the Execution Management System revealed that they were uploading supporting documentation for some fiscal years prior to the guidance being issued.

During our review of the Execution Management System, we found that the combatant commands we visited had not fully implemented the guidance and that the quality of information in the system was questionable. Specifically, we found that:

The Execution Management System is missing supporting documentation. Based on our review, we found that two— STRATCOM and NORTHCOM—of the four combatant commands we visited had uploaded supporting documentation, as required by the Execution Management System guidance, for fiscal years 2013-16. A third combatant command, TRANSCOM, uploaded supporting documentation for fiscal years 2013, 2014, and 2016, but did not upload supporting documentation for fiscal year 2015. The fourth combatant command, PACOM, did not upload supporting documentation for fiscal years 2013, 2014, and 2015, but began uploading supporting documentation in August 2016 for fiscal year 2016 after we informed an official in the Office of the Assistant Secretary of Defense for Readiness that the command had not been uploading supporting documentation in accordance with the Execution Management System guidance. TRANSCOM and PACOM officials stated that one of the reasons they did not upload supporting documentation, as required by guidance, was due to the volume of travel and other related documents generated in executing joint training exercises. Officials stated that it was overly burdensome to upload all of these documents. Nonetheless, an official from the Office of the Assistant Secretary of Defense for Readiness stated the combatant commands need to do their due diligence in uploading supporting documentation in order to ensure proper accountability of Combatant Commanders Exercise Engagement and Training Transformation funds. This official further stated that efforts are underway, that includes establishing a new method for how funds are distributed to stakeholders, to identify approaches that will reduce the data entry burden at the stakeholder level.

Documentation for expenditures uploaded into the Execution Management System did not match reported total expenditures for any of the four combatant commands we visited. Based on our review of a nongeneralizable sample of supporting documentation for fiscal years 2014 through 2016 that was uploaded into the Execution Management System, we found that the sum of the individual expenditures reported in supporting documentation did not match the corresponding total expenditures entered in the system for any of the four combatant commands we visited. According to one combatant command official familiar with this system, individual expenditures reported in supporting documents should be reconcilable to yearly cumulative totals for expenditures. However, when we attempted to link the sum of individual expenditures reported in uploaded supporting documentation to total expenditures data entered into the Execution Management System by combatant command officials, we were unable to do so for three of the four combatant commands we visited. For example, in fiscal year 2015, NORTHCOM uploaded more than 100 documents that supported how funds were obligated or committed. Our review found that the uploaded documentation supported approximately $12.7 million in funds that were committed. However, the figure entered in the Execution Management System was about $11.9 million. Similarly, in fiscal year 2014, TRANSCOM supporting documentation showed that commitments totaled approximately $66.8 million while the figure entered in the Execution Management System was approximately $4.6 million. Officials stated that the reason that the supporting documentation does not match the figures entered in the Execution Management System is that some supporting documentation had not been uploaded. Nonetheless, the inability to reconcile supporting documentation with the expenditures entered in the Execution Management System undermines the quality of the data in the system and inhibits DOD decision makers, particularly those in the Office of the Assistant Secretary of Defense for Readiness, from providing adequate oversight of how funds are being expended in support of the Joint Exercise Program goals. Moreover, the inconsistent uploading of the required supporting documentation and difficulty in reconciling individually reported transactions with cumulative values entered into the system suggests that weaknesses exist in the Execution Management System data entry procedures, which impacts the quality of the data entered in the system. Therefore, it calls into question the use of the Engagement Management System which, according to DOD officials, had been established to provide real-time, accurate information on the execution of Joint Exercise Program funds to decision makers.

DOD has not implemented key processes to help ensure that the Execution Management System produces quality information. Further weakening the quality of the reporting, tracking, and reconciliation of data recorded in the Execution Management System is that none of the four combatant commands we visited, the Office of the Chairman of the Joint Chiefs of Staff, and the Office of the Assistant Secretary of Defense for Readiness had instituted key systemic processes to help ensure that the data entered in the Execution Management System produce quality information—that is, information that is appropriate, current, complete, accurate, accessible, and timely. Standards for Internal Control in the Federal Government states that a variety of control activities should be used for information systems to support the completeness, accuracy, and validity of information processing, and the production of quality information. In addition, management should evaluate information processing to ensure that it is complete, accurate, and valid. Further, these standards state that appropriate documentation of transactions should be readily available for examination. Using these internal controls could reduce to an acceptable level the risk that a significant mistake could occur and remain undetected and uncorrected. Individuals from all four of the combatant commands we visited stated that only one person at their combatant command was responsible for entering data into the Execution Management System for their respective command and that, although they believed their entries were reliable, no quality assurance oversight was conducted on their work. An official from the Office of the Assistant Secretary of Defense for Readiness stated that periodic reviews are conducted on data entered in the Execution Management System but that these reviews are mainly focused on the execution rates of funds and not on whether the data entered produces quality information. Further, according to an official from the Office of the Chairman of the Joint Chiefs of Staff, the checks they perform on the data entered in the Execution Management System are similar to those conducted by the Office of the Assistant Secretary of Defense for Readiness, in that they are focused on whether or not monthly expenditures have been entered into the system in order to ensure that monthly benchmarks are met and less on whether or not the data entered produces quality information. According to officials from two of the four combatant commands we visited, sometimes they receive phone calls from the Office of the Chairman of the Joint Chiefs of Staff or the Office of the Assistant Secretary of Defense for Readiness to validate certain data entries because they seemed erroneous based on an informal review. However, no officials at the combatant commands we visited, the Office of the Chairman for the Joint Chiefs of Staff, or the Office of the Assistant Secretary of Defense for Readiness could demonstrate systemic processes for ensuring that the Execution Management System produced quality information. The absence of quality assurance processes can affect the quality of the information produced by the system that DOD uses to determine its most recent execution rates and defend the Joint Exercise Program’s budget.

As previously discussed, the combatant commands are not following guidance requiring them to upload supporting documentation, and DOD lacks effective internal controls to help ensure the reliability of the data in the system. DOD officials acknowledged the issues we identified regarding inadequate supporting documentation and data reliability within the Execution Management System. A senior DOD official from the Office of the Assistant Secretary of Defense, Readiness (Resources) stated that DOD plans to address these control weaknesses with respect to the Combatant Commander Exercise Engagement and Training Transformation account for the Joint Exercise Program as part of its implementation of DOD’s FIAR Guidance beginning in fiscal year 2018 to ensure that the account is audit ready. DOD established the FIAR Plan as its strategic plan and management tool for guiding, monitoring, and reporting on the department’s ongoing financial management improvement efforts and for communicating the department’s approach to addressing its financial management weaknesses and achieving financial statement audit readiness. To implement the FIAR Plan, the DOD Comptroller issued the FIAR Guidance, which provides a standard methodology for DOD components to follow to assess their financial management processes and controls and to develop and implement financial improvement plans. These plans, in turn, are intended to provide a framework for planning, executing, and tracking essential steps and related supporting documentation needed to achieve auditability. We believe that if DOD appropriately follows the steps outlined in FIAR guidance when executing the Combatant Commander Exercise Engagement and Training Transformation account, it may help improve the quality of funds execution data from this account and make the account audit ready.

However, as previously stated, FIAR guidance will not be implemented with the Joint Exercise Program until fiscal year 2018 and the effectiveness of the guidance cannot be fully determined until after that time. In the meanwhile, according to a senior DOD official, DOD plans to continue using the Execution Management System which is intended to capture the most recent data for the Joint Exercise Program and inform management decision-making regarding joint exercise investments. Without ensuring the required supporting documentation is uploaded and implementing effective internal controls to ensure that data entered in the Execution Management System produces quality information, DOD and other key decision makers may not have the correct financial execution information to defend the Joint Exercise Program’s budget.

Conclusions

DOD has developed a body of guidance for the Joint Exercise Program. In addition, DOD has implemented an approach to develop performance measures to assess the effectiveness of the Joint Exercise Program. Further, DOD uses JTIMS and the Execution Management System to manage the Joint Exercise Program. JTIMS is the system of record for executing the Joint Exercise Program and the Office of the Chairman of the Joint Chiefs of Staff officials developed a user guide intended to help bring more standardization to the system, thereby making the information more useful to other combatant commands. The Execution Management System is used to oversee and report on most recent execution performance data for Joint Exercise Program funding. However, not all of the combatant commands were following guidance requiring them to upload supporting documentation, making it difficult for DOD to have oversight on expenditures for the Joint Exercise Program. Finally, DOD and the combatant commands lack systemic processes for ensuring that the Execution Management System produces quality information. Without ensuring that supporting documentation is uploaded and implementing effective internal controls to ensure the completeness and accuracy of financial information captured for the Joint Exercise Program, DOD and other key decision makers may not have the correct financial information to defend the Joint Exercise Program’s budget.

Recommendations for Executive Action

To better ensure quality financial execution information is available to guide the Joint Exercise Program, we recommend that the Secretary of Defense direct the Office of the Assistant Secretary of Defense for Readiness to take the following two actions: direct the combatant commanders to take steps to comply with current Execution Management System guidance to upload supporting documentation that is reconcilable to funds executed from the Combatant Commanders Exercise Engagement and Training Transformation account; and as the department implements financial improvement plans in accordance with the FIAR guidance, it should include specific internal control steps and procedures to address and ensure the completeness and accuracy of information captured for the Joint Exercise Program’s Combatant Commanders Exercise Engagement and Training Transformation account.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. In its written comments, which are summarized below and reprinted in appendix IV, DOD partially concurred with both recommendations. DOD also provided technical comments, which we incorporated as appropriate.

DOD partially concurred with our recommendation to direct the combatant commanders to take steps to comply with current Execution Management System guidance to upload supporting documentation that is reconcilable to funds executed from the Combatant Commanders Exercise Engagement and Training Transformation account. In its comments, DOD stated that the Execution Management System is not a system of record but rather a “desk-side” support tool that relies on manual inputs and uploads and that the reconciliation of obligation and execution related to Joint Exercise Program funding occurs elsewhere. DOD further noted that the Office of the Assistant Secretary of Defense for Readiness (OASD(R)) issued guidance and routinely reinforces the best practices use of the Execution Management System tool to ensure it produces quality information. Lastly, DOD noted in its comments that that it may not continue using the Execution Management System beyond fiscal year 2017.

We recognize that the Execution Management System is not a system of record. However, as we also note in the report, it is a tool used by DOD to make decisions regarding the Joint Exercise Program because the Defense Finance and Accounting Services (DFAS) Accounting Report Monthly 1002, the system of record, lags behind. Additionally, we acknowledge that DOD issued guidance for the Execution Management System in April 2016, but our work found that the guidance was not routinely reinforced. For example, as we identified in our report, two of the four combatant commands we visited had not, in fact, uploaded supporting documentation, as required by the Execution Management System guidance, for fiscal years 2013-2016. While the Execution Management System may not be funded beyond fiscal year 2017, we continue to believe that as long as the Execution Management System remains in use and for the reasons discussed in the report, combatant commanders should take the necessary steps to comply with existing guidance that requires the uploading of supporting documentation into the Execution Management System so that when DOD managers make decisions regarding the Joint Exercise Program funding, they use information from a financial data system that is reconcilable and auditable.

DOD partially concurred with our recommendation that states that as the department implements financial improvement plans in accordance with the FIAR guidance, it should include specific internal control steps and procedures to address and ensure the completeness and accuracy of information captured for the Joint Exercise Program’s Combatant Commanders Exercise Engagement and Training Transformation account. In its comments, DOD described OASD(R) as having a supporting role in the execution of FIAR plans, which are implemented by Washington Headquarters Services and the Office of Secretary of Defense-Comptroller, and that these agencies provide specific internal controls, processes and procedures for ensuring completeness and accuracy of obligation and execution data. DOD also stated that the Execution Management System is not a component of FIAR and may not be funded after fiscal year 2017, and that moving toward audit readiness, necessary steps and procedures will be put into place to strengthen auditability.

As we stated in the report, the FIAR guidance provides a standard methodology and framework for assessing and developing a system of internal controls to achieve auditability. However, as we recommended, DOD still needs to implement specific internal control steps and procedures as it implements this guidance to ensure the completeness and accuracy of the Joint Exercise Program’s Combatant Commanders Exercise Engagement and Training Transformation account’s financial information. Further, as we reported the FIAR guidance will not be implemented in the Joint Exercise Program until fiscal year 2018 and the effectiveness of the guidance cannot be fully determined until after that time. Accordingly, we continue to believe that the recommendation remains valid.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense; the Under Secretary of Defense for Personnel and Readiness; the Chairman of the Joint Chiefs of Staff; and the Commanders of U.S. Northern Command, U.S. Pacific Command, U.S. Strategic Command, and U.S. Transportation Command. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

This report (1) describes guidance the Department of Defense (DOD) has developed for its Joint Exercise Program and DOD’s approach to assess the effectiveness of the program and (2) evaluates the extent to which DOD uses two key information systems—the Joint Training Information Management System (JTIMS) and the Execution Management System–to manage the Joint Exercise Program.

Office of the Assistant Secretary of Defense for Readiness

DODD 1322.18, Military Training (January 13, 2009)

Office of the Chairman of the Joint Chiefs of Staff

CJCSI 3500.01H, Joint Training Policy for the Armed Forces of the United States (April 25, 2014)

CJCSM 3150.25A, Joint Lessons Learned Program (September 12, 2014)

CJCSN 3500.01, 2015-2018 Chairman’s Joint Training Guidance (October 30, 2014)

CJCSM 3500.03E, Joint Training Manual for the Armed Forces of the United States (April 20, 2015)

CJCS Guide 3501, The Joint Training System: A Guide for Senior Leaders (May 5, 2015)

CJCSM 3511.01H, Joint Training Resources for the Armed Forces of the United States (May 26, 2015)

CJCSI 3150.25F, Joint Lessons Learned Program (June 26, 2015)

U.S. Northern Command

NORAD and NORTHCOM, Joint Training System (JTS) Handbook (May 1, 2013)

NORAD and NORTHCOM Instruction 16-166, Lessons Learned Program and Corrective Action Program (September 19, 2013)

NORAD and NORTHCOM, JTPs, (December 10, 2015)

U.S. Pacific Command

PACOM Instruction 0509.1, Joint Lessons Learned and Issue Resolution Program (April 7, 2010)

PACOM Instruction 0508.12, Joint Training Enterprise in U.S. Pacific Command (October 15, 2012)

U.S. Strategic Command

SI 508-09, Exercise Program (May 3, 2013)

SI 509-01, After Action, Issue Solution/Resolution and Lessons Learned Program, October 14, 2015)

SI 508-03, JTIMS Procedures (November 8, 2015)

U.S. Transportation Command

USTRANSCOM Pamphlet 38-1, Organization and Functions (May 1, 2008)

USTRANSCOM Instruction 36-13, Training, Education, and Professional Development Program (May 23, 2013)

USTRANSCOM Instruction 36-36, Joint Training and Education Program (August 29, 2014)

USTRANSCOM Instruction 10-14, Joint Lessons Learned Program (November 9, 2015)

We judgmentally selected these combatant commands for our site visits to achieve a mix of geographical and functional commands, as well as the funds that had been apportioned to the combatant commands in fiscal year 2016 from the Combatant Commanders Exercise Engagement and Training Transformation account, size of command, and location. We reviewed DOD’s approach to make performance measures specific, measurable, achievable, realistic, and time-phased (commonly referred to as the SMART rubric) to assess the return on investment for the Joint Exercise Program. In addition, we reviewed performance measures reportedly used to assess the ability of the training audience to accomplish training objectives for exercises, as well as measures used to assess the return on investment for conducting an exercise. We also reviewed performance documentation and information captured in JTIMS, as well as a nongeneralizable sample of commander summary reports or after-action reports from seven combatant command joint exercises to understand the content of these reports. Finally, we interviewed senior officials from the Office of the Assistant Secretary of Defense for Readiness, including the Joint Assessment and Enabling Capability office, the Office of the Chairman of the Joint Chiefs of Staff, as well as officials from the four selected combatant commands. We did not review or evaluate the quality of any assessments that DOD has conducted for its joint exercises. We reviewed the assessments to the extent that it was possible to ensure that an assessment process existed.

To evaluate DOD’s use of JTIMS and the Execution Management System to manage the Joint Exercise Program, we reviewed guidance for JTIMS and the Execution Management System. In addition, we observed data associated with a nongeneralizable sample of joint exercises maintained in JTIMS. We also reviewed and analyzed a nongeneralizable sample of cumulative financial data and supporting documentation, if any, entered by combatant command users in the Execution Management System for the Joint Exercise Program during fiscal years 2013-16 to examine the internal controls that were in place. We reviewed a nongeneralizable sample of supporting documentation uploaded into the Execution Management System for fiscal years 2014 through 2016 to make a determination about compliance with guidance issued by DOD for the Execution Management System and the Standards for Internal Control in the Federal Government. Further, we compared individual transactions reported in the supporting documentation with the corresponding cumulative data entered into the system. We also reviewed the FIAR plan— DOD’s strategic plan and management tool for guiding, monitoring, and reporting on the department’s ongoing financial management improvement efforts and for communicating the department’s approach to addressing its financial management weaknesses and achieving financial statement audit readiness— and guidance. Additionally, we spoke with cognizant officials from the four combatant commands we visited, the Office of the Chairman of the Joint Chiefs of Staff, and the Office of the Assistant Secretary of Defense for Readiness about the systems used to execute and manage the Joint Exercise Program. Further, we attended sessions at the 3-day Annual Review for the Combatant Commanders Exercise Engagement and Training Transformation Enterprise on the budget for fiscal years 2018 through 2022. We attended sessions that were most pertinent to this engagement. For example, since the services were not included in the scope of our review, we did not attend their sessions.

We conducted site visits to collect testimonial and documentary evidence about DOD’s Joint Exercise Program at the following locations:

Cost Assessment and Program Evaluation Office, Arlington, Virginia

Force Readiness and Training in the Office of the Assistant Secretary of Defense for Readiness, Arlington, Virginia Joint Assessment and Enabling Capability office, Alexandria, Virginia Joint Staff (J7), Arlington, Virginia Joint Staff (J7) Suffolk, Virginia

U.S. Northern Command, Peterson Air Force Base, Colorado

U.S. Pacific Command, Camp H. M. Smith, Hawaii

U.S. Strategic Command, Offutt Air Force Base, Nebraska

U.S. Transportation Command, Scott Air Force Base, Illinois We conducted this performance audit from June 2015 to February 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: The Process (the Joint Training System) Used to Conduct Joint Exercises

Guidance from the Office of the Chairman of the Joint Chiefs of Staff outlines the process that is used by the combatant commands to develop joint training programs, plan and execute joint training, and assess training for the Department of Defense’s (DOD) Joint Exercise Program. This process, referred to in the guidance as the Joint Training System, is characterized as an integrated, requirements-based, four-phased methodology used to align the Joint Training Strategy with assigned missions to produce trained and ready individuals, staff, and units. According to the guidance, the Joint Training System has four phases through which the combatant commands execute the Joint Exercise Program.

Phase I—Requirements. During this phase, an ordered listing of tasks is developed describing the armed force’s ability to perform activities or processes that combatant commanders require to execute their assigned missions. This listing is referred to as the Universal Joint Task List and it provides a common language to describe warfighting requirements for combatant commanders. From this list, the most essential mission capability tasks—mission-essential tasks—are identified by the combatant commander. Using the commander’s criteria, mission-essential tasks are prioritized to form the Joint/Agency Mission-Essential Task List. In addition to combatant commanders’ priority, key documents pertinent to U.S. national strategy, such as the Unified Command Plan, Guidance for Employment of the Force, and other joint doctrine, are analyzed to determine the most essential mission capability requirements for the combatant command. The Joint/Agency Mission-Essential Task List provides the foundation for deriving joint training requirements used to develop Joint Training Plans and training and exercise inputs to theater campaign plans. Training requirements are derived from training proficiency assessments, mission training assessments, and lessons learned that result from the Phase IV (Assessment) of the Joint Training System.

Phase II—Plans. The plans phase is initiated by conducting an assessment of current capability against the Joint Mission-Essential Task List and relevant lessons learned to identify gaps in training. To address those gaps, the Joint Training Plan is established and identifies who is to be trained; what they will be trained in; what the training objectives are; and when, where, and how the training will occur. Joint Training Plans, along with training and exercise inputs into theater campaign plans, are developed, coordinated, and published in the Joint Training Information Management System (JTIMS) to identify a commander’s training guidance, audiences, objectives, events, and support resources, and to identify the coordination needed to attain the required levels of training proficiency.

Phase III—Execution. During this phase, events planned in the Joint Training Plan are conducted and the training audience’s performance objectives are observed and evaluated. Joint training events are developed and executed using the five-stage Joint Event Life Cycle methodology captured and reviewed in JTIMS. Task Performance Observations—which identify whether the training audience achieved the stated level of performance to the standards specified in the training objectives—and the Training Proficiency Evaluations for each training objective associated with the training event are also captured in JTIMS. Further, facilitated after-action reports are developed to highlight potential issues or best practices to support the assessments in Phase IV (Assessment). Validated observations from the training event are exported into JTIMS.

Phase IV—Assessment. During this phase, leadership within the combatant command determines which organizations within the command are able to perform at the level required to meet the task standards and which missions the command is trained to accomplish. Assessments are a commander’s responsibility. To complete Task Performance Assessments for each task, commanders consider Task Performance Evaluations, lesson learned, and personal observations of the joint training exercise. An assessment ranking of trained, partially trained, or untrained is assigned to each task listed under a training objective. JTIMS supports the assessment of joint training by automating the ability of joint organizations to produce Task Performance Assessments. The Task Performance Assessments are analyzed to create the mission training assessment that is provided to a combatant commander on a monthly basis. The mission training assessment is on how well the command can execute its assigned missions. These training assessments provide input into the next training cycle. Lessons learned, after-action reports, and issues requiring resolution outside of the command are identified during this phase.

Appendix III: The Department of Defense Assesses Individual Joint Exercises Using the Joint Training System

Combatant command officials we visited stated that, in accordance with guidance from the Chairman of the Joint Chiefs of Staff, they used the Joint Training System as the process for conducting training assessments of individual joint exercises to determine each command’s overall readiness to perform command missions. These assessments occur during Phases III (Execution) and IV (Assessment) of the Joint Training System. During Phase III, for example, command trainers collect task performance observations for each training objective identified in the Joint Training Information Management System (JTIMS). These task performance observations identify whether the individuals and units participating in the training exercise achieved the level of performance stated in standards specified in the training objectives. Training proficiency evaluations are conducted for each training objective associated with the exercise. During Phase IV, combatant commanders consider the proficiency evaluations, as well as after-action and commander summary reports, to determine a combatant command’s ability to perform assigned missions at the minimum acceptable level under a specified set of conditions.

According to an official from the Joint Assessment and Enabling Capability office, a subordinate office to the Office of the Assistant Secretary of Defense for Readiness that provides strategic- level assessments of joint training and joint training enablers throughout the Department of Defense (DOD), including to combatant commands, performance measures that are specific, measurable, achievable, realistic, and time-phased (commonly referred to as the SMART rubric) are used to assess the Joint Exercise Program. DOD officials stated that developing performance measures for joint exercises has not been an easy task and that they are constantly working to improve their performance measures. Specifically, in an effort to develop, improve, and provide quality assurance for specific performance measures, the Joint Assessment and Enabling Capability office works with the combatant commands to ensure that they are using the right measures to evaluate the training audience’s ability to perform tasks to specific standards. In addition, the Joint Assessment and Enabling Capability office hosts monthly meetings with combatant command stakeholders to discuss assessment topics, including performance measures. Additionally, the Joint Assessment and Enabling Capability office hosts at least one working group meeting at the annual worldwide joint training conference to conduct face-to-face discussions and reviews of assessment-related tasks for joint training. According to an annual report from the Joint Staff Director for Joint Force Development, the Joint Assessment and Enabling Capability office is available to assist combatant command stakeholders with assessment-related tasks for the Joint Exercise Program, as requested.

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Staff Appendix V: GAO Contact and Staff Acknowledgments Acknowledgments

GAO Contact

Staff Acknowledgments

Guy A. LoFaro, Assistant Director; Patricia Donahue; Pamela Nicole Harris; Amie Lesser; Sabrina Streagle; Sonja S. Ware; and Cheryl A. Weissman made key contributions to this report.

Related GAO Products

Civil Support: DOD Needs to Clarify Its Roles and Responsibilities for Defense Support of Civil Authorities during Cyber Incidents. GAO-16-332. Washington, D.C: April, 4, 2016.

Military Base Realignments and Closures: More Guidance and Information Needed to Take Advantage of Opportunities to Consolidate Training. GAO-16-45. Washington, D.C.: February 18, 2016.

Operational Contract Support: Actions Needed to Enhance the Collection, Integration, and Sharing of Lessons Learned. GAO-15-243. Washington, D.C.: March 16, 2015.

Defense Headquarters: DOD Needs to Reevaluate Its Approach for Managing Resources Devoted to the Functional Combatant Commands. GAO-14-439. Washington, D.C.: June 26, 2014.

Defense Headquarters: DOD Needs to Periodically Review and Improve Visibility of Combatant Commands’ Resources. GAO-13-293. Washington, D.C.: May 15, 2013.

Defense Management: Perspectives on the Involvement of the Combatant Commands in the Development of Joint Requirements. GAO-11-527R. Washington, D.C.: May 20, 2011.

Homeland Defense: U.S. Northern Command Has a Strong Exercise Program, but Involvement of Interagency Partners and States Can Be Improved. GAO-09-849. Washington, D.C.: September 9, 2009.

National Preparedness: FEMA Has Made Progress, but Needs to Complete and Integrate Planning, Exercise, and Assessment Efforts. GAO-09-369. Washington, D.C.: April 30, 2009.

Homeland Defense: Steps Have Been Taken to Improve U.S. Northern Command’s Coordination with States and the National Guard Bureau, but Gaps Remain. GAO-08-252. Washington, D.C.: April 16, 2008.

Homeland Defense: U.S. Northern Command Has Made Progress but Needs to Address Force Allocation, Readiness Tracking Gaps, and Other Issues. GAO-08-251. Washington, D.C.: April 16, 2008.

Homeland Security: Observations on DHS and FEMA Efforts to Prepare for and Respond to Major and Catastrophic Disasters and Address Related Recommendations and Legislation. GAO-07-835T. Washington, D.C.: May 15, 2007.

Military Training: Management Actions Needed to Enhance DOD’s Investment in the Joint National Training Capability. GAO-06-802: Washington, D.C.: August 11, 2006.

Military Training: Actions Needed to Enhance DOD’s Program to Transform Joint Training. GAO-05-548: Washington, D.C.: June 21, 2005.