Military Transformation: Actions Needed to Better Manage DOD's Joint Experimentation Program
Published: 2002-08-29T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) considers the transformation of the U.S. military a strategic imperative to meet the security challenges of the new century. In October 1998, DOD established a joint concept development and experimentation program to provide the engine of change for this transformation. In the nearly 4 years since becoming the executive agent for joint concept development and experimentation, the Joint Forces Command has increased in participation of key DOD stakeholders--the military services, the combatant commands, and other organizations and agencies--in its experimentation activities. The Command has also expanded the participation of federal agencies and departments, academia, the private sector, and some foreign allies. No recommendations flowing from joint experimentation have been approved or implemented. Although the Joint Forces Command issued three recommendations nearly a year ago, they were not approved by the Joint Requirements Oversight Council because of confusion among the Joint Staff and the Joint Forces Command about a proposed change in guidance that required additional data be included when submitting these recommendations. Although DOD has been providing more specific and clearer guidance for joint experimentation, DOD and the Joint Forces Command are missing some key management elements that are generally considered necessary for successful program management.

section_title: Letter
paragraphs: The Department of Defense (DOD) considers the transformation of the U. S. military a strategic imperative to meet the security challenges of the new century. In October 1998, DOD established a joint concept development and experimentation program to provide the engine of change for this transformation. The U.S. Joint Forces Command, the executive agent of the program, carries out experiments—with support from the military services, the combatant commands, and other DOD organizations—to explore and analyze new joint operational and organizational concepts. The experiments can include studies and analyses, conferences, workshops, seminars, war games, and simulated or live field experiments.
On the basis of these experiments, the Joint Forces Command recommends changes aimed at achieving significant improvements in joint capabilities. The Joint Requirements Oversight Council reviews the Command’s recommendations, and the Chairman of the Joint Chiefs of Staff is responsible for approving them. The Council also oversees the implementation of approved recommendations. The cumulative changes are expected to contribute to the way U.S. military forces are transformed. In November 2001, DOD also created an Office of Force Transformation to foster innovation and experimentation and assist the Secretary of Defense to develop guidance and identify strategies and policies for transformation.
Because of the central role DOD’s joint experimentation program plays in military transformation, we reviewed related activities to determine whether changes are needed to improve the program’s ability to focus on future joint operations, military transformation, and the rapid integration of results. Specifically, we assessed the extent to which (1) the Joint Forces Command obtains participation from all relevant sources in developing and conducting its experimentation activities; (2) the recommendations flowing from the joint experimentation process have been approved and implemented; and (3) the necessary management elements are in place to foster program success. We are providing this report to you because of your oversight responsibilities and interest in military transformation issues.
subsections: section_title: Background
paragraphs: The Joint Forces Command, in coordination with the Joint Staff, the services, and other combatant commands and DOD agencies, is responsible for creating and exploring new joint war-fighting concepts, as well as for planning, designing, conducting, and assessing a program of joint experimentation. The Command executed its second large-scale field experiment, Millennium Challenge 2002, this year, and it plans another one in 2004 and others every third year thereafter. These experiments are intended to examine how well the concepts previously explored by the Command in smaller venues will work when applied with the emerging concepts being developed by the services and other combatant commands. For example, Millennium Challenge 2002 tested how well U.S. forces fared against a regional power with a sizable conventional military force and so called “anti-access” capabilities—which can include advanced surface-to-air missiles, antiship missiles and mines, and chemical and biological weapons—and validated the results of earlier experiments to develop the Command’s “rapid decisive” operations concept. The aim of the experiment was to come up with changes that can be made during the current decade. (App. I provides a chronology of major events important to joint experimentation.)
Over the next several years, the Command’s experimentation will focus primarily on two concepts: one to develop a standing joint force headquarters to improve joint command and control; another to conduct more effective joint operations through “rapid decisive” operations. In November 2001, the Chairman of the Joint Chiefs of Staff directed that the Command make development of the prototype headquarters its highest near-term priority. Additionally, the Command will develop a number of other concepts aimed at specialized issues or operational problems to support the two primary concepts.
Joint experimentation is a continuous process that begins with the development of new operational and organizational concepts that have the potential to improve significantly joint operations (see fig. 1). The Joint Forces Command identifies new joint concepts including those developed by other DOD organizations (such as the Joint Staff, services, and combatant commands) and the private sector and tests them in experiments that range from simple (workshops, seminars, war games, and simulations) to complex (large-scale virtual simulations and “live” field experiments). Appendix II provides additional information on joint experimentation program activities.
After analyzing experimentation data, the Command prepares and submits recommendations to the Joint Requirements Oversight Council for review and, ultimately, to the Chairman of the Joint Chiefs of Staff for approval.Before submitting them to the Council, however, the Command submits its recommendations to the Joint Staff for preliminary review and coordination. The recommendations are distributed for review and comment to the Joint Staff directorates, the military services, the combatant commands, and other DOD and federal government organizations. The Council then reviews the recommendations and advises the Chairman of the Joint Chiefs of Staff on whether they should be approved. The changes, if approved, provide the basis for pursuing the capabilities needed to implement a specific operational concept. The Council is also responsible for overseeing the implementation of the recommendations, but it can designate an executive agent, such as the Joint Forces Command, to do so. The Council (or its designated executive agent) is responsible for obtaining the resources needed to implement the recommendations through DOD’s Planning, Programming, and Budgeting System. The Council also assists the Chairman, in coordination with the combatant commands, the services, and other DOD organizations, to identify and assess joint requirements and priorities for current and future military capabilities. The Council considers requirements (and any proposed changes) for joint capabilities across doctrine, organizations, training, materiel, leadership and education, personnel, and facilities.
The Department of the Navy’s budget provides funding to the Joint Forces Command for joint experimentation and other Command missions. In fiscal year 2002, the Command received from the Navy about $103 million for its joint concept development and experimentation program, and it planned to spend about half of this amount for Millennium Challenge 2002. The Command has requested that the Navy provide about $98 million for the program in fiscal year 2003. The Command also provides some funds to the services, the combatant commands, and other DOD organizations for efforts that support its program activities. However, the services fund the operations and support costs of forces participating in joint experimentation. Also, the individual experimentation efforts of the services and the combatant commands are funded from within their own budgets.
section_title: Participation in Joint Experimentation Continues to Evolve
paragraphs: Since it first began joint experimentation, the Joint Forces Command has broadened and deepened the inclusion of other DOD organizations, federal agencies and departments, the private sector, and allies and coalition partners in its process for capturing and identifying new joint ideas and innovations. Organizations participating in joint experimentation are generally satisfied with current opportunities for their ideas to be considered, and many have increased their participation in the program. However, the participation of different stakeholders—the extent of which is determined by the stakeholder—varies considerably and some would like more visits and contacts with the Command. The Command is planning initiatives to increase stakeholder participation in the future, particularly for federal agencies and departments and key allies, but this increase will depend on agency-resource and national-security considerations.
subsections: section_title: Participation Has Grown and Continues to Expand
paragraphs: As the program gradually evolved, the Joint Forces Command solidified a process to involve the military services, the combatant commands, and other DOD organizations in the planning and execution of its joint experimentation activities. Because future joint operations will involve diplomatic, information, and economic actions, as well as military operations, many DOD, federal, and private organizations and governments participate and provide input into the joint experimentation program (see table 1). The Joint Forces Command functions as a facilitator to solicit and coordinate the involvement of these organizations and incorporate their input, as appropriate, into concept development and experimentation activities.
Because the stakeholders determine the extent of their participation in the program, it can vary considerably. However, Joint Forces Command officials stated that participation by the services, the combatant commands, and other DOD organizations has grown steadily since the program was created and continues to grow, as participants become increasingly aware of the strong emphasis that DOD leaders are placing on experimentation. For example, in contrast to the first field experiment in 2000, which had limited involvement by the services, this year’s Millennium Challenge has seen the services more actively involved in early planning, and their individual experiments better coordinated and integrated into the field experiment. Our comparison of participation in the Command’s major field experiment in 2000 with plans for this year’s experiment found a significant increase in the diversity and number of participating organizations and in the number of concepts and initiatives proposed by these organizations. For example, the total number of organizations participating in Millennium Challenge 2002 more than doubled from the prior experiment in 2000 (from 12 to 29 organizations), and the total number of service initiatives increased from 4 to 29.
The Command provides several ways for organizations to participate and provide inputs: they can review program plans and strategies; attend meetings, seminars, and workshops; take part in experimentation activities; and use various communication tools such as E-mail, Internet, and video conferencing. Additionally, the Command obtains input from the various experimentation and research and development organizations of the military services and of some combatant commands and DOD organizations. The Command also considers the results of Advanced Concept Technology Demonstrations efforts, innovations, and recent military operations in developing its program. For example, as a result of its operational experiences in Kosovo, the U.S. European Command identified various joint capability shortfalls in its recent list of Command priorities as a means of guiding the Joint Forces Command in selecting focal areas and activities for experimentation. Further, the Command is taking steps to (1) align its experimentation activities with the schedules of major service and combatant command exercises and (2) adjust its program to allow for earlier consideration of new concepts proposed by the services and the combatant commands in the input process. These adjustments would improve synchronization of experiments with the availability of forces and the training schedules of the services and the combatant commands, allow for greater involvement of these entities in the process, and increase the likelihood that joint requirements are sufficiently considered early in the development of concepts.
Participating organizations also provide input during the annual preparation of two key joint experimentation-program documents: the Chairman of the Joint Chiefs of Staff’s guidance on joint experimentation and the Joint Forces Command’s Joint Concept Development and Experimentation Campaign Plan (see fig. 2). Each year the Chairman provides guidance to the Joint Forces Command to use in developing its Campaign Plan for joint concept development and experimentation. The basis for the Chairman’s guidance is derived from several sources, including strategy and planning documents, studies, and other assessments. Additionally, key DOD stakeholders, including the Chairman’s Joint Warfighting Capability Assessment teams and the Joint Requirements Oversight Council, provide input to the Joint Staff to use in developing the Chairman’s guidance. The Joint Forces Command uses this guidance, with additional input from DOD stakeholders, in preparing its Campaign Plan, which is the primary vehicle for synchronizing its joint experimentation activities and coordinating resources. The Command also solicits and considers input for the Campaign Plan from some other federal agencies and departments, academia, private sector, and allies. After review and endorsement by the combatant commands, the services, and the Joint Requirements Oversight Council, the Chairman approves the Campaign Plan.
section_title: Stakeholders Generally Satisfied but Point to Some Possible Improvements
paragraphs: Officials at the military services, the combatant commands, and other DOD organizations we talked with said they were generally satisfied with the opportunities for input provided by the Joint Forces Command. At the same time, DOD stakeholders have taken various actions to increase their participation. Some, however, would like more contacts and communication with the Command. The Command is responding with some initiatives.
Each service, the Joint Staff, the U.S. Special Operations Command, the U.S. Space Command, as well as some DOD and federal agencies (such as the National Imagery and Mapping Agency and the National Security Agency) have assigned liaison officers at the Joint Forces Command.However, officials at the Central, Pacific, and Southern Commands stated that their staffing levels currently do not allow them to devote personnel in this role. Combatant command officials indicated that the frequency and number of meetings, conferences, and other events held at the Joint Forces Command often make it difficult for their organizations to attend. The officials believe that as a result, the views and positions of their organizations are not always fully captured in some discussions and deliberations. Some of the combatant commands have or are planning to establish their own joint experimentation offices. Officials from the Pacific and Special Operations Commands stated that although their respective joint experimentation offices are largely focused on supporting their own experimentation efforts, the offices provide a cadre of staff who can better coordinate and participate more consistently in the Joint Forces Command’s joint experimentation program. For example, Pacific Command officials said that their own experimentation efforts to improve the command of joint operations over the past few years have contributed to joint experimentation by providing significant insights for the Joint Forces Command’s development of the standing joint-force headquarters concept. Central Command and Southern Command officials said their Commands have plans to establish similar offices soon.
While satisfied with their participation and their ability to provide input into the program, officials at some combatant commands believe that a number of things could be done to improve the program, assuming resources are available. They believe that the Joint Forces Command could increase its visits to and participation in combatant-command activities. Some of the officials also believe that if the Joint Forces Command assigned liaison officers to their commands, the Command would obtain first-hand knowledge and a better appreciation of the various commands’ individual requirements. These officials believe that such a presence at their commands would demonstrate the Joint Forces Command’s commitment to joint experimentation and would allow for interaction with staff throughout their commands. The Joint Forces Command does not favor doing this because of the cost and the difficulty in providing the staff necessary to fulfill this role. Officials at the Pacific, Central, and Southern Commands also believe that some level of funding should be provided to the combatant commands for their use in supporting individual command and the Joint Forces Command experimentation efforts. Combatant command officials stated that currently, funds from other command activities must be diverted to support these efforts.
Out of concern about the need to improve communications and participation in joint experimentation planning, the Joint Forces Command is planning some initiatives such as the following: It plans to create a virtual planning-center site for joint experimentation on its Intranet to provide DOD stakeholders with easily accessible weekly updates to information on planned experiments; participants; goals and objectives; and ongoing experimentation by the Joint Forces Command, the services, the combatant commands, and DOD agencies. It plans to develop the requirements for the site during fall 2002 and to initiate the project soon after. It established Project Alpha—a “think-tank” group—in early 2002 to provide another source of input and outputs. The project will interface with researchers throughout DOD, Department of Energy national laboratories, private industry, and academia to find cutting-edge technologies for inclusion in service and joint experimentation. This relationship will provide an opportunity for the Joint Forces Command to leverage the work of these organizations and similarly, for these organizations to gain a better understanding of and include their work in the joint experimentation program.
section_title: Expanding Interagency and Foreign Participation in Experimentation Events Is Being Considered
paragraphs: As the joint experimentation program matured, participation by non-DOD federal agencies and departments gradually increased. Participation, however, depends upon the agencies’ desire to be involved and their available resources. Lack of involvement could lead to missed opportunities. And participation by allies and coalition partners has been limited by security concerns.
The Joint Forces Command’s input process allows individual federal agencies and departments, such as the Departments of State and Justice, to participate in joint experimentation events as they choose. Interagency participation is improving, according to Command officials. For example, federal agencies and departments are participating in Millennium Challenge 2002 to assist the Command in developing its standing joint- force headquarters concept.
However, resource and staffing constraints prevent some agencies and departments from taking part in experiments. For example, according to a Joint Forces Command official, the Department of Transportation and the Central Intelligence Agency decided not to send representatives to Millennium Challenge 2002 because of staffing constraints. Not only could non-DOD agencies provide important insights and contributions to joint operations, but also some important opportunities could be missed if these agencies do not consistently participate in joint experimentation events.
While federal agencies and departments are beginning to increase their role in joint experimentation, several service and combatant command officials we spoke with believe that greater involvement is needed because of the role these organizations are likely to have in future joint operations. For example, these non-DOD federal agencies and departments would provide support (economic, diplomatic, and information actions) to U.S. military forces in their conduct of operations aimed at defeating an adversary’s war-making capabilities—support that is critical to implementation of the Joint Forces Command’s rapid decisive operations concept.
Several DOD (service, combatant command, Office of the Secretary of Defense, and other DOD organizations) officials we spoke with believe that the Joint Forces Command should explore ways to boost the participation and involvement of allies and coalition partners in joint experimentation. Joint Forces Command officials agree and believe that such cooperation would foster a better understanding of allied perspectives, allow the Command to leverage concept development work, expand available capabilities, and facilitate the development of multinational capabilities. The Command recently created a multinational concept-development and experimentation site on its Intranet to facilitate the involvement of allies and coalition partners in joint experimentation. However, some DOD officials believe that the Joint Forces Command should do more because future U.S. military operations will likely be conducted with other countries. The officials stress that other nations’ military personnel should be included in experiments to develop new operational concepts, if these concepts are to be successful. Joint Forces Command officials pointed out, however, that the participation and involvement of other countries are often constrained by restrictions on access to sensitive security information. For example, North Atlantic Treaty Organization countries only participated as observers in Millennium Challenge 2002 because of security information restrictions. The Command, however, plans to develop ways to better handle these restrictions to allow greater participation by other nations in its next major field experiment in 2004.
section_title: No Recommendations for Change Have Been Approved or Implemented
paragraphs: Nearly 4 years after the program was established, only three recommendations have flowed from the joint experimentation program, and none of them have been approved. Confusion about proposed changes in guidance regarding the information required for submitting these recommendations has partly delayed their approval. At the time we concluded our review, official guidance on what information should accompany joint experimentation recommendations had not been approved. In addition, several DOD officials expressed concern that the process used to review and approve recommendations, the same as that used for major acquisition programs, may not be the most appropriate for a program whose aim is to integrate changes quickly. However, the officials could not pinpoint any specific impasses in the approval process. The DOD officials are also concerned about potential delays in the integration of new concepts because of the lengthy DOD resource allocation process.
subsections: section_title: Concerns Raised about the Delays in Approving Recommendations
paragraphs: The Joint Forces Command submitted one recommendation to the Chairman of the Joint Chiefs of Staff in August 2001 and two more in November 2001 (see table 2). At the time we ended our review, none of the recommendations had been approved. The recommendations to improve the planning and decision-making capabilities of joint forces and provide better training for personnel conducting theater missile defense operations were based on analyses of results of experiments carried out in the first 3 years of joint experimentation. Inputs included two major experiments: Millennium Challenge 2000 (live field experiment in August-September 2000) and the Unified Vision 2001 (virtual simulation experiment in May 2001). The first recommendation was submitted for review just 3 months after the end of the last experiment.
According to a Joint Staff official, however, approval of the recommendations has been delayed because Joint Forces Command and Joint Staff officials were confused about proposed changes in guidance. In May 2001, the Joint Requirements Oversight Council proposed new guidance, which would require that information on costs and timelines be included in joint experimentation recommendations. Prior guidance did not require such information. Although the recommendations went through preliminary review by the Joint Staff, the omission was not caught until the recommendations were to be scheduled for review by the Joint Requirements Oversight Council. Joint Forces Command officials told us that they were not aware of the change in guidance until that time. When we ended our review, Joint Forces Command officials were working with the Joint Staff to assess how much data could be prepared and when. Command officials said that the recommendations will be resubmitted in fall 2002 together with other recommendations emerging from Millennium Challenge 2002. As a result, no recommendations have yet been reviewed or approved. Also, at the time we ended our review, the draft guidance on joint experimentation recommendations had not been approved and issued. This guidance will become especially important because joint experimentation is expected to produce new recommendations more rapidly as the program matures.
The requirement for costs and timeline data is consistent with that of recommendations for major weapon-system-acquisition programs. However, joint experimentation officials at the Joint Forces Command believe that requiring this type of information on joint-experimentation recommendations may not be appropriate because (1) these recommendations are generally intended to convince decision makers to develop particular joint capabilities, not specific weapon systems; (2) the new requirement may slow the preparation of future recommendations; and (3) it will be difficult to provide accurate estimates of costs and timelines for recommendations that span further into the future. It is too early to determine whether these concerns are valid.
section_title: Concerns Raised about the Process for Funding Implementation of Recommendations
paragraphs: Some DOD officials were also concerned that the system currently used to allocate resources to implement joint-experimentation recommendations—DOD’s Planning, Programming, and Budgeting System—may not be the most efficient because it usually takes a long time to review, approve, and provide funding in future budgets. A recommendation approved in 2002, for example, would not be incorporated into DOD’s budget until 2004 or even later. This delay could result in missed opportunities for more rapid implementation.
A Joint Staff official told us that the Joint Staff and the Joint Forces Command recently adjusted the timing of events to better align the joint experimentation process with the Planning, Programming, and Budgeting System. Additionally, DOD established a special fund for the Joint Forces Command to use as a temporary funding source to speed up the implementation of certain critical or time-sensitive recommendations. This source will provide early funding for implementation until funding is provided through DOD’s Planning, Programming, and Budgeting System. However, Joint Forces Command and other DOD officials believe other ways to implement new joint capabilities within the framework of existing budget and oversight practices may need to be considered.
section_title: Some Key Management Elements Are Missing
paragraphs: DOD has been providing more specific and clearer guidance on its goals, expectations, and priorities for the joint experimentation program. Nevertheless, the management of joint experimentation is missing a number of key elements that are necessary for program success: some roles and responsibilities have not yet been defined; current performance measures are not adequate to assess progress; and the Joint Forces Command lacks strategic planning tools for the program.
subsections: section_title: Guidance More Clearly Defined
paragraphs: DOD officials stated that the joint experimentation program had difficulty in its first years because guidance was evolving and was not specific: DOD’s transformation goals were not adequately linked to transformation efforts, and roles and responsibilities were not clearly defined. Over time, the Secretary of Defense and the Chairman of the Joint Chiefs of Staff have provided more specific guidance on the goals and expectations for joint experimentation and its contribution to DOD’s transformation efforts. Guidance for joint experimentation has evolved gradually over the program’s nearly 4-year life span, partly because of shifting defense priorities and lack of clarity about the roles of various DOD stakeholders. Roles and responsibilities have also matured with the program.
The Secretary of Defense’s 2001 Quadrennial Defense Review Reportestablished six transformation goals, which include improving U.S. capabilities to defend the homeland and other bases of operations, denying enemies sanctuary, and conducting effective information operations. According to DOD officials, the Secretary of Defense’s most recent planning guidance tasked the Joint Forces Command to focus its experimentation on developing new joint operational concepts for these goals. To begin meeting these goals, the Chairman has also provided the Joint Forces Command with clarifying guidance that identified specific areas for the Command to include in its experimentation, such as the development of a standing joint-force headquarters concept and of a prototype to strengthen the conduct of joint operations. The Command has reflected this new guidance in its latest Joint Concept Development and Experimentation Campaign Plan. Additionally, the Secretary of Defense reassigned the Command’s geographic responsibilities to focus it more clearly on its remaining missions, particularly transformation and joint experimentation.
DOD officials at both headquarters and the field believe that the recent guidance begins to provide a better framework for the Joint Forces Command to establish and focus its joint experimentation efforts. Some officials, however, believe that future guidance should further clarify the link between joint experimentation and DOD priorities and the required resources necessary to support joint experimentation. DOD, in its comments to a draft of this report, stated that it expects the Transformation Planning Guidance—currently being prepared by the Office of the Secretary of Defense—will establish the requirements necessary to link experimentation to changes in the force.
section_title: Role of Office of Force Transformation Is Not Clear
paragraphs: While roles and responsibilities for DOD organizations are now broadly defined, the new DOD Office of Force Transformation’s role in joint experimentation and its relationship to other stakeholders have not yet been clearly established. The Office’s charter or terms of reference have not been released. DOD plans to issue a directive later this year that will include a charter and description of the Office’s authorities and responsibilities. However, there is still uncertainty about the extent of authority and involvement the Office will have in the joint experimentation program and the Office’s ability to link the program with DOD’s overall transformation efforts. Joint Forces Command and other DOD officials consider having a transformation advocate in the Office of the Secretary of Defense as a beneficial link between the Joint Forces Command’s, the services’, and the combatant commands’ joint experimentation programs and DOD’s overall transformation agenda.
According to DOD’s 2001 Quadrennial Defense Review Report, the Office of Force Transformation, created in November 2001, is to play a role in fostering innovation and experimentation and should have an important responsibility for monitoring joint experimentation and for providing the Secretary of Defense with policy recommendations.
An Office of Force Transformation official told us that the Office will be an advocate for transformation and will help develop guidance and make recommendations on transformation issues to the Secretary of Defense (the Office provided comments on the Secretary’s annual planning guidance and developed instructions for the services on preparing their first transformation road maps).
The Office has also decided to take a cautious approach in carrying out its mission because of possible resistance from other DOD organizations, the same official said. The Office plans to offer its assistance to DOD organizations in their transformation efforts and attempt to influence their thinking on key issues, rather than asserting itself directly into their efforts, for example by funding military use of existing private-sector technology to act as a surrogate for evaluating possible concepts, uses, and designs.
Joint Forces Command officials stated that as of May 2002, they had had only limited discussions with the Office and had not established any working agreements on how the Office would participate in the joint experimentation program.
The Office of Force Transformation has only recently assembled its staff and is beginning to plan its work and establish contacts within DOD and with other organizations. The Office’s budget for fiscal years 2002 and 2003 is about $18 million and $35 million, respectively.
section_title: Current Performance Measures Are Not a Useful Assessment Tool
paragraphs: DOD’s performance measures (or metrics) for assessing joint experimentation—by measuring only the number of experiments carried out—do not provide a meaningful assessment of the program’s contribution toward meeting its performance goal for military transformation because they are only quantitative. Consistent with good management practices and in order to effectuate the purposes of the Government Performance and Results Act of 1993, federal agencies devise results-oriented metrics that provide an assessment of outcomes or the results of programs as measured by the difference they make. In its fiscal year 2000 performance report, the most recent it has issued, DOD described the performance indicators for the joint experimentation program in terms of the number of experiments conducted against a target goal for the prior, current, and following fiscal years. In fiscal year 2000, DOD exceeded its target number of experiments and did not project any shortfalls in meeting its target in the next fiscal year. Although this measure does provide a quantitative assessment of experimental activity, it does not provide a meaningful method for assessing how joint experimentation is helping to advance military transformation.
An Office of the Secretary of Defense official stated that DOD recognizes that better performance measures are needed for assessing how joint experimentation advances transformation and for two other metrics currently used to assess its military transformation goal. The official stated that developing such measures is a challenge because joint experimentation does not easily lend itself to traditional measurement methods. For example, most programs consider a failure as a negative event, but in joint experimentation, a failure can be considered as a success if it provides insights or information that is helpful in evaluating new concepts or the use of new technologies.
An Office of the Secretary of Defense official told us that the RAND Corporation and the Institute for Defense Analyses recently completed studies to identify possible performance measures for assessing the progress of transformation. DOD is evaluating them and is preparing the Transformation Planning Guidance to provide more specific information on the priorities, roles, and responsibilities for executing its transformation strategy. The same official stated that the new guidance will include a discussion of the types of performance measures needed for assessing transformation progress or will assign an organization to determine them. In either case, measures will still need to be developed and implemented. DOD plans to issue the new guidance later in 2002 but has not determined how new performance measures would be incorporated into its annual performance report.
section_title: Joint Forces Command’s Program Lacks Strategic Planning Tools
paragraphs: The Joint Forces Command has not developed the strategic planning tools—a strategic plan, an associated performance plan, and performance- reporting tools—for assessing the performance of the joint experimentation program. Strategic planning is essential for this type of program, especially considering its magnitude and complexity and its potential implications for military transformation. Such planning provides an essential foundation for defining what an organization seeks to accomplish, identifies the strategy it will use to achieve desired results, and then determines—through measurement—how well it is succeeding in reaching results-oriented goals and achieving objectives. Developing strategic-planning tools for the joint experimentation program would also be consistent with the principles set forth in the Government Performance and Results Act of 1993, which is the primary legislative framework for strategic planning in the federal government.
The Joint Forces Command prepares an annual Joint Concept Development and Experimentation Campaign Plan that broadly describes the key goals of its program, the strategy for achieving these goals, and the planned activities. However, a February 2002 progress report, prepared by the Joint Forces Command’s Joint Experimentation Directorate, on the development of the Directorate’s performance management system indicated that one-fourth of those organizations providing feedback on the Campaign Plan believed that the Plan lacks specificity in terms of the program’s goals and objectives and an associated action plan that outlines the activities to be carried out in order to achieve those goals. Officials we spoke with at the military services, the combatant commands, and the Joint Forces Command all cited the need for more specific and clearer goals, objectives, and performance measures for the program.
In the progress report, the Command acknowledged the benefits of strategic planning and the use of this management tool to align its organizational structure, processes, and budget to support the achievement of missions and goals. The report proposed that the Command develop a strategic plan, possibly by modifying its annual Campaign Plan, and subsequently prepare a performance plan and a performance report. Command officials indicated that the basic requirements of a strategic plan could be incorporated into the Campaign Plan, but they were unsure, if such an approach were taken, whether the changes could be made before the annual Campaign Plan is issued later this year. Similarly, the Joint Forces Command has had difficulty in developing specific performance measures for joint experimentation. A Command official stated that the Command has tried to leverage the performance measures developed by other organizations like itself, but found that there is widespread awareness throughout the research and development community, both within and outside DOD, that such measures are needed but do not exist. Additionally, a Joint Forces Command official stated that whatever metrics the Command develops must be linked to its mission-essential tasks for joint experimentation and that the Command is currently developing these tasks. At the time we ended our review, the Command had identified six broad areas for which specific metrics need to be developed. These included quality of life, customer relationships, and experimentation process management.
section_title: Conclusions
paragraphs: After nearly 4 years, the Joint Forces Command’s process for obtaining inputs for the development and execution of DOD’s joint experimentation program has become more inclusive. However, questions continue about whether the program is the successful engine for change envisioned when it was established. Since the program’s inception, only three recommendations have flowed from experimentation activities and their review, approval, and implementation have been delayed from confusion over a change in guidance that required additional information be included in the recommendations. As a result, no recommendations for change have been approved or implemented to date. To the extent that the draft guidance on what should be submitted with joint experimentation recommendations can be officially approved and issued, future recommendations could be submitted for approval and implementation more quickly. Underscoring the need to finalize the guidance is the anticipated recommendations to be made after this year’s major field experiment, Millennium Challenge 2002.
The lack of strategic planning for joint experimentation deprives the Joint Forces Command of necessary tools to effectively manage its program. Implementation of strategic planning at the Joint Forces Command would create a recurring and continuous cycle of planning, program execution, and reporting and establish a process by which the Command could measure the effectiveness of its activities as well as a means to assess the contributions of those activities to the operational goals and mission of the program. Such planning could also provide a tool—one that is currently missing—to identify strengths and weaknesses in the development and execution of the program and a reference document for the effective oversight and management of the program.
Performance measures developed under the Command’s strategic planning could provide the standard for assessing other experimentation efforts throughout DOD, which are also lacking such metrics.
The lack of a meaningful performance measure for assessing the contribution of the joint experimentation program to advance DOD’s transformation agenda limits the usefulness and benefit of this management tool to assist congressional and DOD leaders in their decision-making responsibilities. Establishing a “meaningful” joint experimentation performance measure for its annual performance report would provide congressional and DOD leadership a better assessment of the program’s contribution and progress toward advancing transformation. Such a metric would also be consistent with the intent of the Results Act to improve the accountability of federal programs for achieving program results. Because the role and relationships of the Secretary of Defense’s new Office of Force Transformation have not yet been clarified, the Secretary may not be effectively using this office in DOD’s transformation efforts. This office, if given sufficient authority, could provide the Secretary with a civilian oversight function to foster and monitor the joint experimentation program to ensure that it is properly supported and provided resources to advance the DOD’s overall transformation agenda. Rectifying these shortcomings is critical in view of the importance that DOD has placed on joint experimentation to identify the future concepts and capabilities for maintaining U.S. military superiority.
section_title: Recommendations for Executive Action
paragraphs: To improve the management of DOD’s joint experimentation program, we recommend that the Secretary of Defense direct the Chairman of the Joint Chiefs of Staff to   approve and issue guidance that clearly defines the information required to accompany joint experimentation recommendations for the Joint Requirements Oversight Council’s review and approval and require the Commander in Chief of the U.S. Joint Forces Command to develop strategic planning tools to use in managing and periodically assessing the progress of its joint experimentation program.
We further recommend that the Secretary of Defense   develop both quantitative and qualitative performance measures for joint experimentation in DOD’s annual performance report to provide a better assessment of the program’s contribution to advancing military transformation and   clarify the role of the Office of Force Transformation and its relationship to the Chairman of the Joint Chiefs of Staff, the Joint Forces Command, and other key DOD stakeholders in DOD’s joint experimentation program.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments from DOD on a draft of this report, which are included in their entirety as appendix III. DOD agreed with our recommendations and indicated that it expects that a forthcoming Transformation Planning Guidance and subsequent guidance will be responsive to them by clarifying roles and missions across DOD, implementing recommendations for changes, and establishing clear objectives. We believe such strategic guidance from the Secretary of Defense could provide a significant mechanism for better linking and clarifying the importance of the joint experimentation program with DOD’s transformation agenda. DOD also provided technical comments to the draft that were incorporated in the report where appropriate.
section_title: Scope and Methodology
paragraphs: To determine the extent to which the Joint Forces Command obtains input from stakeholders and other relevant sources in developing and conducting its joint experimentation activities, we reviewed an array of documents providing information about participants in joint experimentation, including guidance and other policy documents, position papers, fact sheets, reports, and studies of the military services, the combatant commands, the Joint Staff, and other DOD organizations. We also reviewed Joint Forces Command plans and reports. Additionally, we made extensive use of information available on public and DOD Internet web sites. To assess the change in participation by various stakeholders over time, we compared the differences in the numbers of participating organizations and initiatives provided by these organizations between the Joint Forces Command’s first two major field experiments in 2000 and 2002 (Millennium Challenge 2000 and Millennium Challenge 2002). We conducted discussions with officials at five combatant commands, the Joint Staff, the military services, and other DOD organizations, such as the Joint Advanced Warfighting Program and the Defense Advanced Research Projects Agency. Appendix IV lists the principal organizations and offices where we performed work. At the Joint Forces Command, we discussed with joint experimentation officials the process for soliciting and incorporating inputs for joint experimentation from the military services and the combatant commands. We also attended conferences and other sessions hosted by the Joint Forces Command to observe and learn about joint experimentation participants and their contributions and coordination. For example, we attended sessions for the Command’s preparation of its annual Joint Concept Development and Experimentation Campaign Plan and planning for this year’s Millennium Challenge experiment. With officials from each of the services and the combatant commands, we discussed perceptions of the effectiveness of coordination and participation in joint experimentation. We also obtained observations about participants’ involvement from several defense experts who track joint experimentation and military transformation. Although we did not include a specific assessment of the individual experimentation efforts of the services and combatant commands, we did discuss with service and command officials how their efforts were coordinated and integrated into joint experimentation. We also did not determine the extent that individual inputs obtained from various participating organizations were considered and incorporated into the joint experimentation program.
To determine the extent to which recommendations flowing from the joint experimentation process have been approved and implemented, we reviewed and analyzed data that tracked the progress of the first three joint experimentation recommendations submitted by the Joint Forces Command. We also obtained and analyzed relevant guidance and held discussions with Joint Staff, Joint Forces Command, and Office of the Secretary of Defense officials on the Joint Requirements Oversight Council process for reviewing and approving joint experimentation recommendations. We also discussed issues relating to implementation of joint experimentation recommendations through DOD’s Planning, Programming, and Budgeting System.
To assess whether key management elements, such as policy, organization, and resources, were in place for the program, we conducted a comprehensive review of current legislative, policy, planning, and guidance documents and reports and studies. We used the principles laid out in the Government Performance and Results Act of 1993 as an additional benchmark for assessing the adequacy of performance measures established for the program and of tools used to manage the program. We also discussed the status and evolution of joint experimentation oversight and management, including office roles and responsibilities and joint experimentation metrics, with officials at the Joint Forces Command, the Joint Staff, the services, the combatant commands, the Office of the Secretary of Defense, the Office of Force Transformation, and other DOD organizations. Several defense experts who follow joint experimentation and military transformation discussed with us joint experimentation oversight and management and gave us their impressions regarding current joint experimentation management practices.
Our review was conducted from October 2001 through May 2002 in accordance with generally accepted government auditing standards.
We are sending copies of this report to interested congressional committees, the Secretary of Defense, the Chairman of the Joint Chiefs of Staff, and the Commander in Chief, U.S. Joint Forces Command. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.
Please contact Richard G. Payne at (757) 552-8119 if you or your staff have any questions concerning this report. Key contacts and contributors to this report are listed in appendix V.
section_title: Appendix I: Chronology of Key Events Important to Joint Experimentation
subsections: section_title: Event Chairman of the Joint Chiefs of Staff issued Joint Vision 2010.
paragraphs: DOD’s Report of the Quadrennial Defense Review issued.
Secretary of Defense designated Commander in Chief, U.S. Joint Forces Command, as executive agent for joint experimentation. Joint Advanced Warfighting Program established.
Relevance to joint experimentation This vision of future war fighting provides a conceptual template for the Department of Defense’s (DOD) transformation efforts across all elements of the armed forces. Report discussed the importance of preparing for future national security challenges. It concluded that DOD needed to institutionalize innovative investigations, such as war-fighting experiments, to ensure future concepts and capabilities are successfully integrated into the forces in a timely manner. The Secretary of Defense tasked the Joint Forces Command to design and conduct joint war-fighting experimentation to explore, demonstrate, and evaluate joint war-fighting concepts and capabilities. DOD established the program at the Institute for Defense Analyses to serve as a catalyst for achieving the objectives of Joint Vision 2010 (and later Joint Vision 2020). To that end, the program is to develop and explore breakthrough operational concepts and capabilities that support DOD’s transformation goals. Joint concept development and experimentation program initiated.
Joint Forces Command assumed responsibility as the executive agent for joint experimentation. Joint Advanced Warfighting Program conducted the first joint experiment for Joint Forces Command.
An experiment—J9901—that investigated approaches for attacking critical mobile targets. Experiment allowed the Joint Forces Command to begin its learning process on how to conduct joint experimentation. Report proposed several recommendations to promote military transformation.
Report of the Defense Science Board Task Force on DOD Warfighting Transformation issued. Chairman of the Joint Chiefs of Staff issued Joint Vision 2020. Millennium Challenge 2000 major field experiment conducted. Chairman of the Joint Chiefs of Staff issued updated Joint Vision Implementation Master Plan.
Transformation Study Report: Transforming Military Operational Capabilities issued.
Joint Forces Command conducted Unified Vision 2001 experiment.
Secretary of Defense’s planning guidance issued.
DOD’s Quadrennial Defense Review Report issued.
Updated vision statement described the joint war-fighting capabilities required through 2020. The first major field experiment coordinated by the Joint Forces Command among the services and other stakeholders. Guidance described the process for generation, coordination, approval, and implementation of recommendations emerging from joint experimentation and defined the roles and responsibilities of DOD stakeholders. Study conducted for the Secretary of Defense to identify capabilities needed by U.S. forces to meet the twenty-first century security environment. Made several recommendations directed at improving joint experimentation. A major joint experiment—largely modeling and simulation— conducted to refine and explore several war-fighting concepts, such as “rapid decisive” operations. Required studies by defense agencies and the Joint Staff to develop transformation road maps and a standing-joint-force headquarters prototype. The report established priorities and identified major goals for transforming the Armed Forces to meet future challenges. It called for new operational concepts, advanced technological capabilities, and an increased emphasis on joint organizations, experimentation, and training.
Event Chairman of the Joint Chiefs of Staff issued joint experimentation guidance.
Office of Force Transformation established. Unified Command Plan 2002 issued.
Secretary of Defense’s planning guidance issued.
Joint Forces Command conducted Millennium Challenge 2002.
Relevance to joint experimentation The guidance directed the Joint Forces Command to focus its near- term experimentation on developing a standing joint force headquarters prototype. Office assists the Secretary of Defense in identifying strategy and policy, and developing guidance for transformation. Plan reduced the number of missions assigned to the Joint Forces Command to allow the Command to devote more attention to its remaining missions such as joint experimentation. The guidance directed the Joint Forces Command to develop new joint concepts that focus on the six transformation goals set forth in the 2001 Quadrennial Defense Review Report. Second major field experiment conducted to culminate a series of experiments to assess “how” to do rapid decisive operations in this decade.
section_title: Appendix II: Additional Information on Joint Concept Development and Experimentation Activities
paragraphs: The Joint Forces Command uses various types of assessment activities to develop, refine, and validate joint concepts and associated capabilities. As shown in figure 3, the Command begins to move through the five joint concept development phases by conducting workshops, seminars, and war games to develop information and identify possible areas to explore in developing new concepts and associated capabilities and then uses simulated or live experiment events to confirm, refute, or modify them. These activities vary in scale and frequency, but each activity becomes larger and more complex. They can involve a small group of retired flag officers and academics, up to 100 planners, operators, and technology experts, or several thousand in the field. Near the end of the process, the Command will conduct a large-scale simulation experiment (such as Unified Vision 2001), followed by a major field experiment (such as Millennium Challenge 2002). The process continuously repeats itself to identify additional new concepts and capabilities.
Table 3 provides additional information about the characteristics, scale, and frequency of these and other associated activities and experiments.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Organizations and Offices Contacted
subsections: section_title: Washington, D.C., Area
paragraphs: Office of the Secretary of Defense, Program Analysis and Evaluation   Office of the Under Secretary of Defense for Policy   Office of the Under Secretary of Defense for Acquisition, Technology,   Joint Advanced Warfighting Program   Defense Advanced Research Project Agency   Office of Force Transformation   Operational Plans and Interoperability Directorate   Joint Vision and Transformation Division   Command, Control, Communications, and Computers Directorate   Force Structure, Resources, and Assessment Directorate   Directorate of Training   Directorate of Integration   Directorate for Strategy, Concepts, and Doctrine   Office of the Deputy Chief of Naval Operations for Warfare   Marine Corps Combat Development Command Department of the Air Force   Booz Allen Hamilton   The Carlyle Group   Center for Strategic and Budgetary Assessments   Hicks & Associates, Inc.
section_title: Honolulu, Hawaii, Area
section_title: Miami, Fla., Area
section_title: Norfolk, Va., Area
section_title: Tampa, Fla., Area
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the individuals named above, Carol R. Schuster, Mark J. Wielgoszynski, John R. Beauchamp, Kimberley A. Ebner, Lauren S. Johnson, and Stefano Petrucci made key contributions to this report.