Joint Military Education: Actions Needed to Implement DOD Recommendations for Enhancing Leadership Development
Published: 2013-10-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To facilitate unified operations across the services, DOD has provided JPME programs at departmental and service academic institutions for almost 30 years. In July 2012, the Director for Joint Force Development, who reports to the Chairman of the Joint Chiefs of Staff, tasked the MECC to review DOD's joint education objectives and institutions to help ensure that outcomes match requirements for the strategic environment projected for Joint Force 2020. Subsequently, the National Defense Authorization Act for Fiscal Year 2013 mandated GAO to report to Congress on the analytical approach used by the MECC not later than 90 days after the Director submitted the MECC's report to GAO, which the Director did on July 1, 2013. In this report, GAO (1) identifies the purpose of DOD's study of the JPME program, and (2) assesses DOD's methodology used to conduct the Joint Professional Military Education study and its planning for follow-on actions. GAO analyzed the MECC's final report and relevant planning documents, interviewed DOD officials who conducted portions of the study, and reviewed leading practices for evaluating programs by GAO and other entities.
section_title: What GAO Found
paragraphs: The purpose of the Department of Defense's (DOD) study of its Joint Professional Military Education (JPME) program was to identify (1) desired leader attributes as part of the JPME career-long learning experience needed to support DOD's strategic vision and (2) any gaps in the current educational program to facilitate the development of the leaders needed to achieve that vision. Specifically, a Military Education Coordination Council (MECC)--following direction from the Chairman of the Joint Chiefs of Staff and the Director for Joint Force Development--proposed six desired leader attributes, including, for example, the ability to anticipate and respond to surprise and uncertainty, and concluded that the existing institutional structure for providing JPME should be retained. The MECC's gap analysis, however, indicated that in order to support the development of these attributes, a greater emphasis on career-long self-directed learning is also needed, among other things. In line with its findings, the MECC made 21 recommendations to improve the JPME program that address increased accessibility of educational programs, changes to teaching methodologies and assessment mechanisms, and enhanced use of technology.
DOD's methodology generally included leading practices for assessing training programs, but DOD has not yet fully planned for follow-on actions and engaged all stakeholders, nor has it assessed the costs of the MECC's recommendations to provide decision makers with more timely and complete information and help ensure that the study's results are implemented. Specifically, the MECC reviewed other related studies, conducted a gap analysis to identify gaps based on existing and future needs, used the best available data and acknowledged limitations--all practices identified by GAO and other government agencies and research institutions as leading practices for successful evaluations of training and other programs. By contrast, DOD documents state that the results of the study were intended to inform and shape the fall 2013 academic year, but the MECC did not complete its study until June 24, 2013 and provided its report to GAO on July 1, 2013. Further, DOD has not yet identified milestones and timeframes for implementing all of its recommendations. Subsequently, the department developed an update to inform actions for moving forward on its recommendations. DOD identified target dates for completion not later than September 2014 for 9 recommendations, but did not include interim milestones, and has not yet developed target dates for 12 recommendations. In addition, the MECC did not formalize plans to achieve the buy-in of all stakeholders for recommendation implementation. Without this information, it may be difficult for DOD to ensure that stakeholders agree on and are accountable for implementing the recommendations in a timely manner. Finally, the MECC did not analyze the costs or efficiencies associated with implementing its recommendations, but it identified 5 recommendations that could incur additional costs because they require further study. Leading practices such as accounting for program resources enable managers to better manage existing resources and plan for future programmatic needs. Without cost data on the study's recommendations or plans to assess cost in the near term as part of continued efforts to implement the results of the JPME study, decision makers could be hindered in determining the most efficient allocation of departmental resources for JPME.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD establish well-defined timeframes for conducting any follow-on actions and include stakeholders necessary for implementation, and assess the costs of implementing recommendations made in the MECCs recent study of joint professional military education. DOD concurred with both of GAOs recommendations.

section_title: Letter
paragraphs: Professional military education is key to the development of the nation’s  armed forces, and it is the quality of that military education that  distinguishes U.S. forces around the world. As operations over the last  decade have demonstrated, military officers must think critically,  communicate well, conduct themselves with integrity, and lead others in  difficult and often dangerous situations that require forces from across the  military departments. The passage of the Goldwater-Nichols Department  of Defense Reorganization Act in 1986 (Goldwater-Nichols) marked one  of the most significant attempts over the past few decades to coordinate  service efforts.Among other things, the act sought to establish clearer  lines of command and control and improve the ability of the services to  work with each other in truly joint, rather than simply multi-service,  operations. In the wake of Goldwater-Nichols, a panel chaired by  Representative Isaac N. “Ike” Skelton IV in 1989 undertook a  comprehensive review of joint professional military education (JPME) and  made a number of specific recommendations on how the joint reforms of  Goldwater-Nichols should be institutionalized among the services through  officer in-residence education. Since the Skelton Report, Congress has  passed additional legislation affecting the professional military education  system, including a requirement to create a program of JPME, through  which military officers would be educated on joint matters throughout their  careers.  Over time, the Department of Defense (DOD) and others have  conducted studies that, to some extent, reviewed the cumulative effects  of these legislative changes.
The commitment within DOD to JPME demonstrates a major investment,  both of time and money.  Some of the education programs are an  academic year in length, which represents a significant part of an officer’s  career. To support the academic programs, in fiscal year 2012, DOD  spent approximately $47.8 million on its JPME schools within National  Defense University alone.
In July 2012, DOD’s Director for Joint Force Development, expanding on  a white paper signed by the Chairman of the Joint Chiefs of Staff, directed  DOD's Military Education Coordination Council (MECC) to prepare a  report on the value of the JPME program, the educational outcomes  sought, and any changes necessary to strengthen the program’s value to  DOD. According to the director’s memo, this review was necessitated by  changes in the security environment, the fiscal climate, and the  experiences of DOD’s joint force since the passage of Goldwater-Nichols  in 1986. Subsequently, the National Defense Authorization Act (NDAA)  for Fiscal Year 2013 required us to report to Congress on the analytical  approach used by the MECC for that report not later than 90 days after  receiving DOD’s report from the Director for Joint Force Development.Our report (1) identifies the purpose of DOD’s study of the Joint  Professional Military Education program, and (2) assesses DOD’s  methodology used to conduct the Joint Professional Military Education  study and its planning for follow-on actions.
To identify the purpose of DOD’s study of the JPME program, we  reviewed the documents directing the study, the minutes of the MECC  meetings, interim briefing report, and the MECC’s final report, dated June  24, 2013. We also interviewed knowledgeable DOD officials from the  Joint Professional Military Education Division within DOD’s Joint Staff’s  Directorate for Joint Force Development (J7). The Joint Professional  Military Education Division, in conjunction with the MECC, was  responsible for leading DOD’s effort. To assess the methodology DOD  used to conduct the JPME study and its planning for follow-on actions, we  analyzed relevant reports and guidance by GAO, the Office of  Management and Budget, the Office of Personnel Management, and  private sector organizations such as the RAND Corporation, among  others. We identified key practices contained in these documents for  evaluating educational programs and information that were applicable for  the purposes of this engagement. We then compared DOD’s  methodology with these select leading practices for educational program  evaluation. We also reviewed documents identified by DOD officials as  project plans for the study, as well as documentation of the department’s  intended follow-on actions. Again, we interviewed DOD officials from the  Joint Professional Military Education Division within the Directorate for  Joint Force Development. We also met with officials across DOD,  representing institutions such as the National Defense University, the  U.S. Army War College, and the Joint Forces Staff College, among  others, who were identified as Team Leads for the study. Further details  about our scope and methodology can be found in appendix I.
We conducted this performance audit from March through October 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Joint education is provided to some extent at all levels of officer and  enlisted professional military education. The professional development  and career progression for both officers and enlisted personnel through  DOD’s professional military education is a service responsibility;  embedded within the professional military education systems is a  program of JPME that is overseen by the Joint Staff.
For officers, this system is designed to fulfill the educational requirements  for joint officer management as mandated by law. The joint education  program is intended to prepare aspiring military leaders for both  conducting operations in a coherently joint force and thinking their way  through uncertainty. Officer JPME program courses are taught at multiple  sites across the country, including the service Staff and War Colleges and  the National Defense University. See appendix II for a list and map of  DOD’s academic institutions where officer JPME courses are taught.
According to the Officer Professional Military Education Policy  (OPMEP), the JPME program for officers comprises curriculum  components in the five levels of the officer professional military education  system and includes three-statutorily mandated levels of JPME designed  to progressively develop the knowledge, analytical skills, perspectives,  and values that are essential for U.S. officers to function effectively in  joint, interagency, intergovernmental, and multinational operations. For  each of those levels, the emphasis changes to provide instruction in  target areas to enhance leader attributes of attending service members at  their specific rank and tenure.
Within officer PME, the first level of JPME—precommissioning—provides  officer candidate and officer training school, Reserve Officer Training  Corps, and Military Service Academy students with a basic foundation in  defense structure, roles and missions of other military services, the  combatant command structure, and the nature of American military power  and joint warfare. The second level of JPME—primary—provides junior  officers at the O-1 through O-3 ranks with primary education on the  tactical level of war. This level of JPME helps to foster an understanding  of the Joint Task Force combatant command structure and how national  and joint systems support tactical-level operations, among other subjects.
The third level begins the first statutorily-mandated level of JPME  requirements—JPME Phase I. This phase generally focuses on tactical  and operational levels of war, and is typically attended by intermediate- level officers at the O-4 rank, as well as some officers at the O-5 and O-6  ranks. JPME Phase I is incorporated into the curricula of the  intermediate- and senior-level service colleges, as well as other  appropriate educational programs that meet JPME criteria and are  accredited by the Chairman of the Joint Chiefs of Staff. This level of  education provides officers the opportunity to gain better understanding  from a service component perspective of joint force employment at the  operational and tactical levels of war. The subject matter to be covered by  JPME Phase I instruction must, by law, include at the least (1) national  military strategy; (2) joint planning at all levels of war; (3) joint doctrine;   (4) joint command and control; and (5) joint force and joint requirements  development.
The fourth level of JPME provides Phase II of the statutorily-directed  JPME requirements.  JPME Phase II is a follow-on for selected  graduates of service schools and other appropriate education programs  that complements and enhances the JPME Phase I instruction. Phase II  is taught at National Defense University’s National War College and  Dwight D. Eisenhower School for National Security and Resource  Management, the Joint Forces Staff College’s Joint and Combined  Warfighting School and Joint Advanced Warfighting School to both  intermediate- and senior-level (O-5 and O-6) students, and at senior-level  service colleges to senior-level students, and consists of courses on the  operational and strategic levels of war. Phase II helps prepare officers for  high-level policy and command and staff responsibilities, with a focus on  areas such as national security strategy and joint strategic leadership. In  addition to the subjects specified in JPME Phase I, the curriculum for  JPME Phase II must, by law, include: (1) national security strategy;        (2) theater strategy and campaigning; (3) joint planning processes and  systems; and (4) joint, interagency, intergovernmental, and multinational  capabilities and the integration of those capabilities. This phase  completes the educational requirement for joint officer management.
The fifth and final level of instruction is the CAPSTONE course of JPME  for general/flag officers and focuses on the operational and strategic  levels of war for high-level joint, interagency, intergovernmental, and  multinational responsibilities. Specifically, the CAPSTONE course  focuses on (1) the fundamentals of joint doctrine; (2) integrating elements  of national power across military operations to accomplish security and  military strategies; and (3) how joint, interagency, intergovernmental, and  multinational operations support strategic goals and objectives. This level  of JPME is tiered to ensure the progressive and continuous development  of executive level officers.  Figure 1 summarizes the five levels of JPME.
In addition to meeting legislative requirements to provide education on  JPME matters, DOD’s colleges and universities that provide academic  year-long programs for officers are Master’s Degree-granting institutions.  According to the Middle States Commission for Higher Education, the  body that accredits National Defense University, among others,  accreditation is the means of self-regulation and peer review adopted by  the educational community. The accrediting process is intended to  strengthen and sustain the quality and integrity of higher education,  making it worthy of public confidence and minimizing the scope of  external control. Accreditation by the Commission is based on the results  of institutional reviews by peers and colleagues and attests to the  judgment that the institution has met the following criteria:   that it has a mission appropriate to higher education;  that it is guided by well-defined and appropriate goals, including goals  for student learning;  that it has established conditions and procedures under which its  mission and goals can be realized;  that it assesses both institutional effectiveness and student learning  outcomes, and uses the results for improvement;   that it is substantially accomplishing its mission and goals;  that it is organized, staffed, and supported so that it can be expected  to continue to accomplish its mission and goals; and   that it meets the Requirements of Affiliation and accreditation  standards of the Middle States Commission on Higher Education.
DOD also conducts periodic assessments of the three statutorily- mandated levels of officer JPME to ensure that the curricula meet the  prescribed joint educational requirements at each level, and uses the  results of these assessments to update educational policy as appropriate.
For enlisted personnel, the Enlisted Professional Military Education  Policy circulates the policies, procedures, objectives, and  responsibilities for enlisted professional military education and enlisted  JPME. According to that policy, enlisted professional development and  progression through the enlisted military education continuum is a service  responsibility. The initial focus of enlisted professional military education  is military occupational specialty training and education required to  produce enlisted personnel capable of performing assigned tasks.  Beyond occupational specialty training, the Enlisted Professional Military  Education Policy states that all enlisted personnel also should be  exposed to enlisted JPME as they progress through their respective  services’ enlisted professional military education system. In addition, the  policy states that, for some enlisted personnel, more comprehensive joint  education is required to prepare those servicemembers for specific joint  assignments.
Enlisted JPME includes programs that span an enlisted member’s career  and apply to all enlisted personnel. Basic Enlisted JPME addresses  educational guidelines that should be completed by pay grade E-6,  while Career Enlisted JPME addresses educational guidelines for senior  enlisted personnel in grades E-6 or E-7 and above. Beyond these  programs, Senior Enlisted JPME is a stand-alone Web-based course  designed specifically for senior enlisted personnel in pay grades E-6  through E-9 who are serving in or are slated to serve in joint  organizations. DOD’s KEYSTONE course exists to prepare command  senior enlisted leaders for service in flag level joint headquarters or joint  task force organizations. This course is designed for personnel who are  serving in pay grade E-9. The focus of this course is to enable command  senior enlisted leaders to think intuitively from a joint perspective while  serving in their capacity in a general/flag officer joint organization.
DOD also developed a program that, since 2003, has provided JPME  specifically to reserve officers. In response to legislative direction, the  Joint Forces Staff College established a 40-week, blended learning  Advanced Joint Professional Military Education program (AJPME) that  consists of two distance learning periods and two face-to-face periods.  Similar to the active component courses, the Officer Professional Military  Education Policy outlines the requirements for AJPME. The program’s  mission is to educate reserve component officers to plan and execute  joint, interagency, intergovernmental, and multinational operations to  instill a primary commitment to teamwork, attitudes, and perspectives.   AJPME builds on the foundation established by the institutions teaching  JPME Phase I and prepares reserve component officers (O-4 to O-6) for  joint duty assignments.
DOD’s JPME programs are approved by the Chairman of the Joint Chiefs  of Staff.  The Director for Joint Force Development retains responsibility  to support the Chairman of the Joint Chiefs of Staff and the joint  warfighter through joint force development, in order to advance the  operational effectiveness of the current and future joint force.  The Joint  Staff’s Directorate for Joint Force Development (J7) oversees general  policy for the JPME programs. The MECC, which was tasked with  conducting the most recent DOD-initiated study on JPME, serves as an  advisory body to the Director of the Joint Staff on joint education issues.  The purpose of the MECC is to address key educational issues of interest  to the joint education community, promote cooperation and collaboration  among the MECC member institutions, and coordinate joint education  initiatives. Its membership consists of the MECC principals and a  supporting MECC Working Group. These principals include the Director  for Joint Force Development; the Deputy Director of the Joint Staff for  Military Education; the presidents, commandants, and directors of the  joint professional military education colleges and service universities; and  the heads of any other JPME-accredited institutions. Additional  representatives from other commands and organizations may be invited  to participate as is appropriate. The MECC Working Group is comprised  of dean’s level/ O-6 representatives of the MECC principals. The Deputy  Director of the Joint Education and Doctrine Division chairs the MECC  Working Group. Additionally, the Secretary of Defense, service chiefs,  and combatant commanders are invited to send participants to the MECC  meetings to provide feedback to improve the educational process.
section_title: DOD’s Study Was Intended to Identify Leader Attributes, Career-Long Education Goals, and Educational Gaps
paragraphs: The purpose of DOD’s issued study of the JPME program was to identify  desired leader attributes by defining what is needed from the JPME  career-long learning experience to support DOD’s strategic vision, as well  as any gaps in the current educational program to facilitate the  development of the leaders needed to achieve that vision. The original  purpose and objectives for the study were laid out in a July 16, 2012,  white paper from the Chairman of the Joint Chiefs of Staff and a Joint  Staff memorandum issued with the white paper.
In the white paper, the Chairman stated that the study’s purpose was to  support the development of the department’s leaders by fostering the  values, strategic vision, and critical thinking skills needed to lead and  support the development of Joint Force 2020. This white paper also  identified four attributes that the department’s education programs should  develop in its leaders. Those attributes include the following:  the ability to understand the security environment and the  contributions of all elements of national power;   the ability to deal with surprise and uncertainty;  the ability to anticipate and recognize change and lead transitions;  and  the ability to operate on intent through trust, empowerment, and  understanding.
The Chairman’s white paper also stated that other attributes for leader  development would evolve and would need to be aligned with future  operations and incorporated into curricula to help ensure that gaps are  identified and eliminated.
The memorandum, signed by the Director for Joint Force Development,  identified three questions meant to shape the review. The first question  focused on the value of joint education, why it is needed, and how the  military will train and educate its leaders to meet the requirements of Joint  Force 2020. The second question focused on the educational outcomes  sought by the department, specifically with regard to training leaders for  carrying out DOD’s strategic vision, in order to help ensure alignment of  desired leader attributes with the curriculum. The third question built upon  the first two by focusing on any changes needed to strengthen and  achieve the value and outcomes DOD seeks in building leaders for Joint  Force 2020. The memorandum also instructed DOD’s MECC to conduct  the study.
Subsequent to the issuance of the white paper and memorandum, the  MECC met on August 23, 2012, to refine the study’s specific objectives.  According to officials, at the direction of the Director for Joint Force  Development, the MECC removed the requirement to address the value  of joint education and instead focused its efforts on identifying the  intended outcomes of the educational program and actions needed to  achieve the outcomes. The MECC’s discussions also established two  broad objectives for the study—documented in meeting minutes and an  enclosure—based on the direction in the memorandum and further  direction received by the Chairman. The two broad objectives the MECC  established were to:  1.  define the Joint Education Continuum needed to meet the joint  leader development goals for Joint Force 2020, and    2.  determine the gaps that need to be addressed for the Joint Education  enterprise to progress from the current joint leader development  outcomes to producing the future desired joint leader attributes for  Joint Force 2020.
The evolution of the study’s objectives is described below in figure 2.
The MECC’s first objective focused on defining and improving upon both  the officer and enlisted joint education continuums to ensure that they are  in line with the development goals for DOD’s strategic vision. The MECC  identified a need to retain many aspects of the current officer joint  education continuum, as well as a need to incorporate new elements into  the proposed officer joint education continuum that would help to meet  the changing needs of future leaders of Joint Force 2020. For example,  the proposed officer continuum retains the comprehensive approach of  embedding elements of JPME at each level of service-delivered  education with an emphasis on the statutory requirements for JPME  Phases I and II, as well as key milestones from the precommissioning  level of JPME to the general/flag officer rank. In addition to retaining the  existing institutional structure for providing JPME education, however, in  its report the MECC also emphasized the importance of self-directed,  career-long learning and development. This facet of officer development  is intended to convey an expectation that an individual is responsible for  his or her own education and development, thereby inculcating a culture  of lifelong learning. The MECC’s study concluded, on the other hand, that  changes to the enlisted JPME continuum—specifically, between the  Senior Enlisted JPME and KEYSTONE courses— are needed to improve  the depiction of the Chairman’s intent for enlisted professional military  education. Currently, enlisted personnel who complete the Senior  Enlisted JPME course at pay grade E-6 or E-7 are not eligible to attend  the KEYSTONE course until they attain pay grade E-9, potentially leaving  a significant interval in time between participation in the two courses.  Accordingly, the MECC recommended dividing the existing Senior  Enlisted JPME course into two parts. Both parts would be offered prior to  the KEYSTONE course and reduce the interval of time that senior  enlisted servicemembers must wait between phases of enlisted joint  education.
The second objective expanded upon the first, and focused on identifying  and closing gaps in the current JPME curriculum to produce joint leaders  with attributes that are in line with Joint Force 2020. Specifically, the  MECC refined the four attributes proposed in the Chairman’s white paper,  added two additional attributes, and identified a set of subattributes  associated with each of the six desired leader attributes, that the MECC  used to analyze the current capabilities of the department’s joint  education programs. The refined list of desired leader attributes consists  of the abilities to:  1.  Understand the security environment and contributions of all  instruments of national power,  2.  Anticipate and respond to surprise and uncertainty,  3.  Anticipate and recognize change and lead transitions,  4.  Operate on intent through trust, empowerment, and understanding,  5.  Make ethical decisions based on shared values of the profession of  6.  Think critically and strategically in applying joint warfighting principles  and concepts to joint operations.
The MECC then conducted a gap analysis to identify and crosswalk the  desired leader attributes for Joint Force 2020 with the current officer and  enlisted personnel JPME education continuums.
For officers, the MECC’s analysis showed that, while the JPME program  currently addresses the desired leader attribute outcomes at some level  and also meets the intent of the existing requirements identified by the  Skelton Panel, gaps exist where changes are needed to meet the  challenges of Joint Force 2020. These include changes to teaching  methodologies, assessment mechanisms, and other areas in support of  the newly-identified desired leader attributes. Table 1 summarizes where  DOD has identified gaps in its officer JPME curricula associated with the  identified list of desired leader attributes and subattributes.
For enlisted personnel, the MECC determined that a separate and distinct  set of desired leader attributes should be developed for senior enlisted  leaders that will require additional study.  Officials from the Joint Staff  initiated this work on April 8, 2013 and assigned this task to DOD’s  Enlisted Military Education Review Council (EMERC). The EMERC took  up this effort at its August 22, 2013 meeting and proposed a list of six  desired leader attributes for enlisted personnel. DOD officials provided us  with the proposed list of enlisted desired leader attributes on September  16, 2013.
The MECC made a total of 21 recommendations, which collectively  address the study’s objectives and span four categories, including:         (1) desired leader attributes, subattributes, or educational outcomes;      (2) joint education continuums; (3) lifelong learning and advancements in  learning technologies; and (4) faculty quality. See appendix III for a list of  the 21 recommendations. Several of these recommendations call for  additional study by the National Defense University and the services to  support the achievement of leader attributes. Other recommendations  emphasize the importance of strengthening the educational outcomes at  the primary level of joint education, making education programs more  accessible, and using prior learning assessments to tailor education  opportunities to students’ needs, among other things. One  recommendation, in particular, calls for undertaking an approach to  develop a separate and distinct set of desired leader attributes to guide  enlisted joint education. Additionally, the MECC considered whether to  make any legislative proposals, but did not identify the need for any  legislative changes to the joint education enterprise based on the results  of the review. According to officials, the results of the study will be used  ultimately to inform updates and revisions to the Officer Professional  Military Education Policy and the Enlisted Professional Military  Education Policy, which are the primary instructions that distribute the  policies, procedures, objectives, and responsibilities for professional  military education and JPME. Our next section discusses DOD’s plans in  this area.
section_title: Study’s Methodology Generally Included Leading Practices, but DOD Has Not Yet Fully Planned for Follow-on Actions and Engaged All Stakeholders, or Considered Potential Implementation Costs
paragraphs: The methodology DOD used for its study of JPME generally included  leading practices for evaluating strategic training and other programs,  such as reviewing existing literature, using available data, and assessing  skills gaps.  However, the department has not yet fully planned for follow- on actions or fully engaged all stakeholders to help ensure that they are  held accountable for making progress in implementing the study’s  recommendations. Specifically, the department has not taken steps to  ensure that the results of the study will be implemented using an  approach agreed upon by all stakeholders, or that the results will be  implemented in a timely manner. Further, the department has not yet fully  evaluated the potential costs associated with the implementation of the  MECC’s recommendations to provide decision makers with more  complete information and assurance that the recommendations will be  cost-effective.
subsections: section_title: MECC Reviewed Existing Literature, Used Available Data, and Assessed Gaps
paragraphs: According to our 2012 report, Designing Evaluations, a key first step in  designing a program evaluation is to conduct a literature review in order  to understand the program’s history, related policies, and knowledge  base. A review of the relevant policy literature can help focus evaluation  questions on knowledge gaps, identify design and data collection options  used in the past, and provide important context for the requester’s  questions.  Further, according to the RAND Corporation’s Standards for  High-Quality Research and Analysis, a study team should demonstrate  an understanding of other related studies, which should be evident in, for  example, how a problem is formulated and approached. The team should  also take particular care to explain the ways in which its study agrees,  disagrees, or otherwise differs importantly from previous studies.
The MECC’s report states that the initial step in its review was to  determine the JPME enterprise’s effectiveness in meeting its current  requirements.  According to an official, the MECC reviewed many  documents as part of a literature review, but derived its findings from four  specific sources—all issued within the last 5 years—that it determined  were most relevant to the review. Specifically, these sources were:
The 2010 House Armed Services Committee (HASC), Subcommittee  on Oversight and Investigations (O&I), study which provided a  detailed assessment of the state of JPME;
A Decade of War, conducted by the Joint and Coalition Operational  Analysis Division within the Joint Staff (J7);
The Ingenuity Gap: Officer Management for the 21st Century; and
Keeping the Edge: Revitalizing America’s Military Officer Corps.
According to the MECC’s report, these documents collectively provide a  broad look across joint education. The MECC found that the 2010 HASC  report, for example, provided a detailed assessment of the state of JPME,  and concluded that while the overall PME system was basically sound,  some areas needed improvement, such as an increased emphasis on  joint, interagency, intergovernmental, and multinational operations.  Consistent with this finding, the MECC made a recommendation that the  Joint Professional Military Education Division review specific subject  areas for increased emphasis within joint education as part of the process  of revising the officer and enlisted policies. These subject areas include  cyber warfare, interagency and intergovernmental operations, and  information and economic instruments of national power, among others.  Further, these sources suggested that responsibilities for junior officers  will increase and that joint education should be expanded at lower levels.
Similarly, the MECC found that officers are receiving joint education  exposure earlier in their careers, and also recommended that the services  strengthen the instruction of the desired leader attributes at the primary  level of joint education for junior officers. Additionally, these sources  recognized the need for increased emphasis on handling uncertainty and  critical thinking skills. DOD’s study also focused on the ability to anticipate  and respond to surprise and uncertainty, and critical thinking in joint  operations.
RAND’s Standards report further indicates that the data and information  used for a study should be the best available and that a research team  should indicate limitations in the quality of the available data. As part of its  second objective, the MECC analyzed data from joint accreditation visits,  that provided broad areas of assessment, and institutional survey and  outcome assessments, that provided more detailed observations related  to curricula. More specifically, the MECC conducted a review of the  Process for the Accreditation of Joint Education (PAJE) reaffirmation  results and institutional self-studies for the past 10 years. DOD uses  these studies to document compliance with the Officer Professional  Military Education Policy (OPMEP) standards and identify areas for  improvement. These reviews have not identified systemic problems or  required changes of PME over the past 10 years, and have concluded  that PME institutions are successful in meeting policy and curricula  standards, as well as the intent of the Skelton Report. The MECC also  obtained and analyzed existing survey data—from various JPME  institutions—of students, graduates, and supervisors of graduates as an  indirect assessment of the programs to identify common areas and trends  in JPME, as well as outcome assessments such as exams, quizzes, oral  presentations, among other things, from the past 2 years as a more direct  measurement of student learning. Consistent with the aforementioned  RAND Standards report, while the MECC did leverage existing data, the  MECC’s report also presented limitations of the existing survey data that  it used for the study, stemming from variations in survey questions and  measures that the separate JPME institutions used to conduct their own  surveys.
Regarding gap assessments, our 2004 Guide for Assessing Strategic  Training and Development Efforts in the Federal Government highlights  the need for determining the skills and competencies necessary to meet  current and future challenges, and to assess any skill and competency  gaps. According to the MECC’s report, to determine the ability of joint  education to meet the Joint Force 2020 joint leader development  requirements, the MECC conducted a three-phase gap analysis. These  three phases included: (1) refinement of the desired leader attributes,  which included breaking these attributes down into subattributes;            (2) analysis of data provided by 27 officer and enlisted programs across  the JPME system to identify gaps in the JPME program; and (3) gap  analysis and documentation to break down the data into cohorts and rank  the desired leader attributes. Specifically, 27 officer and enlisted  programs, which included the services’ staff and war colleges and  National Defense University, among others, were asked to assess the  current capabilities of their joint education programs to achieve Joint  Force 2020 outcomes by addressing the following three questions:
Are current instructional methods effective, that is, how well do we  Is the curriculum effective, that is, do we teach it?   teach and deliver it?
Are we achieving the educational outcomes, that is, can we measure  it?
According to its report, the MECC relied on the professional opinion of the  academic deans of the JPME schools and directors of other joint  education programs to make determinations about the effectiveness of  their officer and senior enlisted programs based on the 35 subattributes  identified by the MECC. To conduct the gap analysis, the MECC analyzed  the 945 responses it received from the 27 programs and used a stoplight  color coding scheme to summarize by cohort group each program’s  assessment of its current capabilities to achieve the Joint Force 2020  outcomes. The gap analysis results were then coded, scored and  ranked by officer cohort group (precommissioning, intermediate, senior,  CAPSTONE) using four categories to describe the program’s current  capabilities. In turn, this gap analysis helped inform the MECC’s 21  recommendations. While the MECC conducted a similar gap analysis for  its senior enlisted JPME programs, the results of that analysis led the  MECC to conclude that the department needed to further reconsider and  redefine the desired leader attributes for enlisted personnel.
section_title: MECC Has Not Yet Fully Developed Timeframes and Coordinated with All Stakeholders, nor Assessed Costs for Follow- on Action
paragraphs: The MECC has not yet fully developed timeframes and coordinated with  all stakeholders, nor has it assessed the costs associated with  implementing any of the study’s recommendations. As a result, it is not  clear how the department will obtain concurrence among its stakeholders  regarding an approach for moving forward, or how the department will  hold its stakeholders accountable for implementing change. Furthermore,  the department will not know whether the implementation of its  recommendations will be cost effective.
We have previously found that project planning is the basis for  controlling and managing project performance, including managing the  relationship between cost and time. Our 2004 Training Guide highlights  the need for agencies to develop a formal process to help ensure that  strategic and tactical changes are promptly incorporated in training and  development efforts. Specifically, that report advises that agencies  develop plans that describe or outline the way in which the agency  intends to incorporate strategic and tactical changes into its training and  developmental efforts. Our guide also highlights that including important  agency stakeholders in the process can contribute to an open and  continuous exchange of ideas and information, particularly when it comes  to ensuring that strategic and tactical changes are promptly incorporated  into training and developmental efforts.
According to the July 16, 2012 memorandum that directed the MECC  study, the March 1, 2013 deadline for the results of the study was  intended to help shape the fall 2013 academic year. The MECC  completed its report on June 24, 2013, and provided us with a copy of the  final report on July 1, 2013. Subsequently, officials provided us with an  informational paper that identifies an office of primary responsibility and  outlines details on the status of the recommendations. However, this  paper does not fully lay out DOD’s planned actions in a transparent  manner. For nine of the recommendations, the document identifies  potential target dates for completion, one of which is to be completed later  in 2013, and eight of which are to be completed not later than September  2014. For the remaining 12 recommendations, DOD identified a target  date for completion as either “not applicable” or “to be determined”  because most of those recommendations fall under the purview of the  services, the services’ schools, or National Defense University.  Accordingly, officials states that consideration or implementation of any  recommendations that fall under the purview of these stakeholders is at  the discretion of those organizations and, according to officials, the  department cannot identify or impose timeframes for implementing those  recommendations. We recognize that the implementation of several of the  recommendations fall under the purview of the services, the services’  schools, or National Defense University. However, the MECC has not yet  reached out to these entities to identify estimated target dates for  implementation to help guide these efforts. Further, the document does  not identify interim milestones for specific actions between the date of the  report and the target completion dates. The document also states that  completion or implementation of a number of the recommendations is  contingent on revisions to the Enlisted Professional Military Education  Policy or the Officer Professional Military Education Policy, which officials  estimated may take 12-18 months.
Officials did tell us that the Joint Staff’s Joint Professional Military  Education Division staff and members of the MECC are currently in the  beginning stages of incorporating the changes into policy and that the  individual schools may begin to incorporate changes resulting from the  study at any time. However, at the time of our review, the department’s  process for monitoring the implementation of the MECC’s  recommendations was unclear.  Without established milestones and  timeframes for taking follow-on action, and mechanisms for coordinating  with all key officials and stakeholders and holding them accountable,  DOD may not be reasonably assured that the study’s recommendations  will be implemented across the department or that its JPME program will  achieve the potential benefits resulting from these recommendations.
In addition, the department did not include a requirement that the MECC  assess the costs associated with the JPME program or any  recommendations resulting from the study. The MECC’s report  acknowledges that the resource-limited environment facing the  Department of Defense will make it difficult to sustain current practices,  such as maintaining small class sizes and small student-to-faculty ratios,  at the intermediate and senior courses or to consider additional  requirements for the lower-level courses. The report states that  leveraging new learning approaches, such as increased distance learning  options, will not provide an inexpensive or free solution for increased joint  education requirements. However, the MECC did not analyze or  consider either the costs associated with implementing the study’s  recommendations or efficiencies to be derived from implementing the  study’s recommendations. Of the MECC’s 21 recommendations, the  MECC identified five recommendations requiring further study and other  possible investments, which suggests a potential need for additional  funding resources to conduct those studies and implement any findings.  For example, the report included recommendations to (1) conduct a study  to identify and evaluate potential educational tools to achieve certain  desired leader attributes, and (2) consider and explore opportunities to  incentivize and reward lifelong learning. The costs associated with these  recommendations are unclear and officials responsible for the study could  not provide us with cost estimates.
Prior studies of professional military education also did not or were unable  to fully identify the costs associated with the overall JPME program, or  with individual JPME programs. The Skelton Panel, for example, inquired  into the cost per student at each school.  The Office of the Secretary of  Defense provided the panel with raw data produced with different  methodologies by service, and sometimes by school, which resulted in  widely varying costs for roughly similar programs. More recently, the  House Armed Services Subcommittee for Oversight and Investigations  made its own effort to ascertain whether a uniform cost accounting  system existed for DOD’s professional military education system. The  Department provided cost-per-student figures with standardized criteria.  Comparative figures were no longer characterized by such enormous  variations, but there were still a number of unexplained differences in the  data provided by the Office of the Secretary of Defense.
Our 2004 Training Guide states that agencies should strategically target  training and development investments to help ensure that resources are  not wasted on efforts that are irrelevant, duplicative, or ineffective. Our  Training Guide further recommends that agencies should consider the  appropriate level of investment and prioritize funding so that the most  important training needs are addressed first. We have previously  concluded that leading practices such as accounting for program  resources enables managers to better manage existing resources and  plan for future programmatic needs. Further, accounting for program  resources is becoming increasingly important for decision makers within  the department who must make difficult trade-off decisions in a  sequestration budget environment.
According to officials, the MECC omitted any and all cost elements from  the study so as to not limit their assessment of the department’s needs for  educating its leaders for Joint Force 2020 based on resources. Officials  told us that they did not want to eliminate options for change based on the  cost associated with them. However, by not assessing the cost of the  MECC’s recommendations, as well as any efficiencies that may be  achieved through their implementation, DOD may not be in a position to  know if it has the resources to implement them or whether efficiencies will  be achieved, and, without plans to assess cost in the near term as part of  continued efforts to implement the results of the JPME study, decision  makers may not know if implementing their recommendations will be cost  effective.
section_title: Conclusions
paragraphs: The experiences of the last 12 years of operations in Iraq and  Afghanistan have underscored the importance of U.S. military officers’  ability to work jointly across the services and reinforced the need for an  effective joint military education program. DOD’s recently completed  study of JPME is an important step forward as DOD considers how to  adapt its joint military education program to reach the goals it laid forth for  Joint Force 2020. Nonetheless, some questions remain about how well its  findings can be applied to the task of revising the department’s officer and  enlisted JPME policies. Specifically, ensuring transparency and taking  certain actions to ensure this study’s findings and recommendations can  be utilized effectively is important. For instance, without establishing  milestones and timeframes for conducting follow-on actions, and involving  necessary stakeholders and developing an implementation approach that  provides for accountability, the department may not be assured that the  study’s findings and recommendations will be incorporated into the JPME  program at institutions across the department and the services in a timely  manner. Additionally, without reliable information on the costs of  implementing the study’s recommendations, decision makers could be  hindered in determining the most efficient allocation of departmental  resources for JPME. DOD’s JPME program will continue to play a role in  the future development of departmental leaders, and acting to help  ensure the findings and recommendations of this and any future studies  are both utilized and cost-effective could allow the department to make  the best use of the resources it devotes to this program.
section_title: Recommendations for Executive Action
paragraphs: To guide the implementation of actions DOD identified in its study on  JPME, we recommend that the Chairman of the Joint Chiefs of Staff  direct the Director for Joint Force Development to take the following two  actions: establish well-defined timeframes for conducting follow-on actions,  coordinate with all stakeholders, and identify key officials  responsible for implementing the study’s recommendations to help  ensure the usefulness, timeliness, and implementation of any  actions DOD takes in response to the findings and  recommendations contained in its study, and assess the costs of implementing recommendations made and  efficiencies to be derived from the recommendations in order to  implement DOD’s recommendations in a cost-effective manner.
section_title: Agency Comments
paragraphs: In written comments on a draft of this report, DOD concurred with our two  recommendations to guide implementation of the actions that DOD  identified in its review of joint education. DOD’s comments are reprinted  in appendix IV. DOD also provided technical comments on the draft  report, which we incorporated as appropriate.
Regarding our first recommendation that DOD establish well-defined  timeframes for conducting follow-on actions, coordinate with all  stakeholders, and identify key officials responsible for implementing the  study’s recommendations, DOD stated that, as the Joint Staff pursues  implementation of the recommendations in its review, it will establish  timelines, identify offices of primary responsibility, and coordinate with  stakeholders as appropriate. DOD also noted that efforts are already  underway to update the Chairman of the Joint Chief of Staff’s professional  military education policies to ensure that these policy directives are  consistent with the results and recommendations of the recent study.
Regarding our second recommendation that DOD assess the costs of  implementing recommendations made and efficiencies to be derived from  the recommendations, the department again concurred, stating that,  although assessing the costs of its recommendations is beyond the  original scope and purpose in this study, it will consider costs and  efficiencies prior to moving forward with the implementation of any  recommendations. It also stated that a number of recommendations will  be implemented through policy changes which are cost neutral. Further, it  stated that the services will also consider such costs and efficiencies for  recommendations under their purview.
We are sending copies of this report to the appropriate congressional  committees, the Chairman of the Joint Chiefs of Staff, and the Directorate  for Joint Force Development. In addition, this report will also be available  at no charge on the GAO Web site at http://gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To identify the purpose of the Department of Defense’s (DOD) study of  the Joint Professional Military Education (JPME) program, we reviewed  relevant DOD guidance and other documents related to the study,  including the final report dated June 24, 2013, and met with  knowledgeable officials. We reviewed the Chairman of the Joint Chiefs of  Staff’s July 16, 2012 white paper that called for and outlined the general  focus of the study. We also reviewed the Director for Joint Force  Development’s (J7) memorandum that was issued with the white paper  and clarified the overall intent of the study and specific questions to help  frame the study, and directed responsibility of the study to the Military  Education Coordination Council (MECC). We analyzed the MECC’s  meeting minutes and interim briefing report, which provided insight into  how the MECC addressed the questions posed in the Director’s  memorandum, as its approach differed somewhat from the framework  provided in the memorandum. We then compared the aforementioned  guidance documents and interim report findings to the final report to  assess consistency. In addition, we met with knowledgeable officials from  the Joint Professional Military Education Division within DOD’s Joint  Staff’s Directorate for Joint Force Development who, in conjunction with  the MECC, were responsible for guiding DOD’s effort. We also met with  the officers of primary responsibility and/or team leads responsible for  carrying out the various elements of the study. These meetings helped to  ensure our understanding of the purpose and objectives of the study. We  reviewed the final report section that focused primarily on the study’s  findings and recommendations for change to gain an understanding of the  MECC’s findings and recommendations. Further, we met with  knowledgeable officials within the Directorate for Joint Force  Development and the team lead for the findings and recommendations  portion of the study to gain clarification and insight into the major  takeaways from the study, areas in need of change, and any plans for the  way ahead.
To assess the methodology DOD used to conduct the joint professional  military education study and its planning for follow-on actions, we  reviewed the MECC’s final report and supporting documents. We  interviewed DOD officials from the Joint Professional Military Education  Division within the Directorate for Joint Force Development and officials  from across DOD who were identified as officers of primary responsibility  or team leads for specific study tasks associated with the study’s two  objectives. We reviewed relevant reports and guidance by GAO; the  Office of Management and Budget; the Office of Personnel Management;  and private sector organizations such as the RAND Corporation, among  others, to identify select leading practices for evaluating training,  educational, and other programs. The following list includes select  sources we consulted for leading practices, in addition to those listed  throughout the report, and against which we assessed DOD’s JPME  study:
GAO, GAO Schedule Assessment Guide: Best Practices for Project  Schedules, GAO-12-120G (Washington, D.C.: May 2012),
GAO, Designing Evaluations: 2012 Revision, GAO-12-208G  (Washington, D.C.: January 2012),
RAND Corporation, Standards for High-Quality Research and  Analysis (Santa Monica, Calif.: November 2011),
The Office of Personnel Management, Training Evaluation Field  Guide (Washington, D.C.: January 2011),
GAO, Government Auditing Standards: 2007 Revision, GAO-07-162G  (Washington, D.C.: January 2007),
GAO, Human Capital: A Guide for Assessing Strategic Training and  Development Efforts in the Federal Government, GAO-04-546G  (Washington, D.C.: March 2004),
The Office of Management and Budget, Circular A-94, Guidelines and  Discount Rates for Benefit-Cost Analysis of Federal Programs  (Washington, D.C.: Oct. 29, 1992), and
GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.:  March 1991).
We identified key practices contained in these documents for evaluating  educational programs and information that were applicable for the  purposes of this engagement. We then compared DOD’s methodology  with these select leading practices for program evaluation. We also  reviewed documents provided to us and identified by DOD officials as  project plans for the study, as well as documentation of the department’s  intended follow-on actions.
We conducted this performance audit from March through October 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Officer Joint Professional Military Education (JPME) Institutions
subsections: section_title: Department of Defense Institutions
section_title: United States Army
section_title: United States Air Force
section_title: United States Navy
section_title: United States Marine Corps
section_title: Appendix III: Recommendations Made by the Military Education Coordination Council (MECC) in Its Study of Joint Professional Military Education
paragraphs: The MECC made a total of 21 recommendations, which collectively  address the study’s two objectives and span four categories, including:   (1) desired leader attributes, or educational outcomes; (2) joint education  continuums; (3) lifelong learning and advancements in learning  technologies; and (4) faculty quality. Several of these recommendations  call for additional study by the National Defense University and the  services to support the achievement of leader attributes. Other  recommendations emphasize the importance of strengthening the  educational outcomes at the primary level of joint education, making  education programs more accessible, and using prior learning  assessments to tailor education opportunities to students’ needs, among  other things. Table 2 lists the recommendations by category.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Brenda S. Farrell (202) 512-3604 or farrellb@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, David Moser, Assistant  Director; Richard Burkard; Timothy Carr; Gustavo Crosetto; Melissa  Emrey-Arras; Jennifer Madison; Terry Richardson; Jennifer Weber; and  Erik Wilkins-McKee made key contributions to this report.