Military Training: Cost-Effective Development of Simulations Presents Significant Challenges
Published: 1995-11-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed the Department of Defense's (DOD) development of the Joint Simulation System (JSIMS), focusing on whether DOD: (1) is progressing with its development of JSIMS; and (2) decisions to improve the Aggregate Level Simulation Protocol (ALSP) Confederation are cost-effective.
section_title: What GAO Found
paragraphs: GAO found that: (1) JSIMS has not progressed beyond the conceptual stage due to internal disagreements within DOD; (2) further JSIMS development is contingent on the availability of about $416 million in funding; (3) DOD plans to spend at least $40 million through 1999 to improve ALSP before replacing it with JSIMS, but it is unclear whether that approach is cost-effective; (4) funding availability depends on how the services prioritize their contributions to JSIMS and ALSP; (5) the cost of ALSP improvements could increase because the planned improvements are service-specific and there is also no ALSP Central Funding; and (6) management of ALSP improvements is fragmented because DOD is not ensuring that the services will complete them and that the improvements are cost-effective.

section_title: Letter
paragraphs: Since 1988, the Department of Defense (DOD) has been grappling with how to provide simulations that realistically portray joint warfare operations for training. To help meet this training need, the Defense Advanced Research Project Agency developed the Aggregate Level Simulation Protocol (ALSP). The ALSP technique provides a means for multiple service and agency models to communicate with each other. Although ALSP is a technological achievement, the existing warfare capabilities of the individual models is limited, and therefore, the problem of providing a valid joint training environment remains. Overcoming these limitations would require extensive improvements to the ALSP Confederation.
Because of these limitations and the cost and complexity of the ALSP Confederation, the military services; the Joint Staff; and the Office of the Secretary of Defense’s Director, Defense Research and Engineering, initiated the development of a comprehensive system that would take the place of the ALSP Confederation. The new system—the Joint Simulation System (JSIMS)—is scheduled to reach initial operational capability by 1999 and full operational capability by 2003.
Until JSIMS is operational, DOD plans to continue making improvements to the ALSP Confederation. Because DOD has repeatedly acknowledged a need to improve joint training through cost-effective simulations, we initiated a review to determine whether (1) DOD is progressing with its development of JSIMS and (2) DOD’s decisions to improve the ALSP Confederation are cost-effective. We conducted this review under our basic legislative responsibilities and are addressing this report to you because we believe it will be of interest to your committees.
subsections: section_title: Background
paragraphs: Because the models that comprise the ALSP Confederation were built in the early 1980s to meet service-specific needs, they lack the ability to simulate many aspects of joint warfare, including operations other than war, strategic mobility, space, intelligence, and logistics capabilities. These models also lack the capability to represent many combat interactions, such as ground to ship. Because of the existing shortfalls of the services’ individual models, the ALSP Confederation can only fully support 2 of 25 identified CINC and service joint training requirements. This also may be the result of the fact that requirements for application of the ALSP technology were solicited from the CINCs and services only after development of the technology.
The services have long recognized the technical and training shortfalls of their respective models for accurately portraying joint operations. The Army’s Corps Battle Simulation is a ground maneuver training simulation used in exercises for commanders and battle staffs. The Army model lacks the capability to simulate weather information, the terrain of the battlefield, and ground-to-ground combat interactions with the Marine Corps model. The Air Force’s Air Warfare Simulation used to support air operations has limited capability to simulate electronic warfare, reconnaissance and surveillance play, and space capabilities. The Navy’s sanctioned training model, the Enhanced Naval Wargaming System, operates on a hardware system that cannot interface with ALSP. The Navy has been modifying this model for acceptance into the confederation since 1993. Navy officials were unable to elaborate on the joint training benefits that would be achieved from these modifications. According to service modeling and simulation officials and after-action reports, the Research, Evaluation, and Systems Analysis Simulation—a naval analytical model—has been used successfully in the current ALSP Confederation. In 1994, the Marine Corps introduced a new amphibious operations simulation, the Marine Air Ground Task Force Tactical Warfare Simulation, into the ALSP Confederation.
The Office of the Secretary of Defense created the Defense Modeling and Simulation Office to serve as the focal point for modeling and simulation under the Director, Defense Research and Engineering. The DOD Executive Council for Modeling and Simulation, chaired by the Director, Defense Research and Engineering, advises and assists the Under Secretary of Defense for Acquisition and Technology in modeling and acquisition decisions. The JSIMS program is a jointly managed DOD program with the Air Force providing acquisition oversight. The JSIMS Joint Program Office, under the Air Force Program Executive Officer for Combat Support Systems, has been designated as an acquisition activity for JSIMS. The Army’s Simulation, Training and Instrumentation Command is the executive agent for the day-to-day management of the ALSP Confederation.
section_title: JSIMS Proceeds at a Deliberate Pace
paragraphs: The development of JSIMS is already a year behind schedule and a clear, consistent definition of JSIMS is still evolving. According to the June 1994 joint memorandum of agreement, a clear definition was due of what constitutes JSIMS within 4 months of the signing of the memorandum. Also due was a detailed plan of action in the form of a JSIMS Joint Program Office charter and JSIMS master plan delineating duties, responsibilities, mission, scope, and strategies for implementing JSIMS. However, lack of agreement among the services as to what JSIMS entails has delayed approval of the charter and the plan. The services have different interpretations of the memorandum of agreement. The low end of expectations is a set of standards and protocols that would allow interoperability for the services’ next generation of simulations. The high end of expectations is a “super model” in which JSIMS would describe all of the objects, such as aircraft, for all of the services and determine all warfare functions.
During July 1995, the Acting Assistant Secretary of the Air Force for Acquisition approved milestone 0 for the JSIMS program, which authorizes proceeding into the concept exploration and definition phase of the acquisition cycle. At that time, the JSIMS Joint Program Office stated that JSIMS would comprise (1) a core element of common functions, such as terrain and weather effects and (2) warfare functions, such as air, ground, and naval combat, and logistics. Common core development would be the responsibility of the JSIMS Joint Program Office while warfare function development will be the responsibility of designated executive agents. The executive agents will develop a joint representation of their warfare area that would then be integrated with the JSIMS core. The Army is the executive agent for land warfare, the Air Force for air and space warfare, and the Navy for sea warfare. The Marine Corps’ missions will be included throughout these executive agents’ warfare representations.
Further, the 1994 memorandum of agreement stated that JSIMS should also be adaptable to other modeling and simulation applications, such as analysis and testing. However, in February 1995, the Deputy Secretary of Defense directed the Director, Program Analysis and Evaluation, to initiate and lead development of a new joint analysis model called the Joint Warfare System (JWARS). Program Analysis and Evaluation officials informed us that they believed improvements to DOD’s analytical capability needed to be made now and they could not afford to wait for JSIMS to become a reality. The JSIMS’ focus is now solely on providing a simulation environment for joint task force training. Coordination between the JSIMS and JWARS programs is being worked out.
Currently, the major stumbling block for JSIMS is how to fund the $416 million program since there is no central funding line for the program. Some military service officials have expressed concerns about the piecemeal approach of funding JSIMS. As of July 1995, the JSIMS’s core element was estimated to cost about $154 million. Under the provisions of the joint memorandum of agreement, the Army, the Air Force, and the Navy have each agreed to provide 30 percent of this cost. The Marine Corps will provide 10 percent of the cost. In addition to the $154 million, the executive agents will incur additional costs, currently estimated at a total of $262 million, to develop simulations for their specific warfare functions.
The problem with this approach is that if a service believes that improving its own core competencies has a higher priority to fund than its responsibilities for JSIMS, that function for JSIMS may not be developed in concert with the other required components. The military services are proceeding to develop the next generation of simulations that will better address their specific mission or core requirements. The services are also responsible for ensuring that these simulations are able to function within the JSIMS’ domain. The Army’s program, Warfighters’ Simulation 2000, is estimated to cost about $200 million and be operational by 2000. The Air Force is developing the National Air and Space Warfare Model that is estimated to cost about $103 million and be fully operational by 2003. The Navy is developing an analytical simulation, the Naval Simulation System, at an initial estimated cost between $15 million and $25 million that could be enhanced at an additional cost of about $47 million to function in a training capacity. Unless decisive management is exercised, these service efforts may outpace JSIMS’ core development and require additional modifications to operate in the JSIMS’ domain.
According to DOD officials, several recent events have occurred that demonstrate the JSIMS program is moving forward. First, on July 14, 1995, the Director, Defense Research and Engineering, chaired the first JSIMS Senior Review Board at which the members agreed to provide their share of the JSIMS Joint Program Office permanent staff. Second, the Under Secretary of Defense for Acquisition and Technology signed a memorandum on August 8, 1995, calling on DOD components to formally adopt a proposed division of funding and personnel requirements. Third, the Deputy Secretary of Defense endorsed the establishment of a joint core funding line with the services providing both their share of core funding and personnel to staff the JSIMS Joint Program Office. In addition, the Director, Defense Research and Engineering, and the Joint Staff are to provide a share of funding for the JSIMS core program. However, we note that these actions have not been formalized.
section_title: DOD’s Expenditures on the ALSP Confederation May Not Be Cost-Effective
paragraphs: Concurrent with the development of JSIMS, DOD has decided to make improvements to the ALSP Confederation, the last of which is expected to be in place in 1999—at the same time that JSIMS should reach initial operational capability. According to the ALSP Master Plan, the improvements are intended to respond to the identified CINC and service training requirements and include additional capabilities such as strategic mobility and ground-to-ground combat interactions between models. Even though officials from the Office of the Secretary of Defense’s Director, Defense Research and Engineering, the Defense Modeling and Simulation Office, and the Army Simulation, Training, and Instrumentation Command told us that the total cost of these improvements will not be significantly high, none of these offices was able to provide comprehensive cost estimates. We identified about $40 million that DOD plans to spend for ALSP Confederation improvements through fiscal year 1999. However, because this money may be directed toward service-specific improvements rather than joint improvements, the cost could be higher.
As is the case with JSIMS, there is no central funding line for the ALSP Confederation improvements. Consequently, DOD’s ability to achieve all of the improvements that it seeks is dependent on funding from the individual military services, agencies, or CINCs. However, to date the Office of the Under Secretary for Acquisition and Technology has not provided the management to ensure that all significant components of the ALSP improvements will be completed. Consequently, management of the improvements has been fragmented and it is questionable whether the improvement plan is cost-effective. For example, the Army has decided not to fund ALSP improvements to its ground warfare model, which is a primary component of the ALSP Confederation. The Army is proceeding to develop its new training model, Warfighters’ Simulation 2000. The Army has already awarded contracts for the new model’s development. The impact of the Army’s decision not to fund ground warfare improvements on other confederation model improvement efforts or future training requirements is unknown.
In contrast, the Air Force is spending about $7 million to consolidate two versions of its air warfare model and plans to enter the combined model into the ALSP Confederation in 1997. The consolidation effort will result in combining the best features of the two versions, as well as preventing future duplicative efforts. The Navy has been spending nearly $2 million annually to replace its current confederation model by fiscal year 1997. Navy officials, however, could not specify how the replacement model would improve the confederation’s joint training capability. The U.S. Transportation Command and the U.S. Space Command are each modifying models for inclusion into future confederations that would expand the ALSP Confederation’s capability.
section_title: Recommendations
paragraphs: To help ensure the total development of JSIMS, we recommend that the Secretary of Defense establish a joint funding line for the core development of JSIMS and direct the Secretaries of the Army, the Navy, and the Air Force to establish funding lines for their respective executive agent JSIMS responsibilities regarding warfare function development. Further, we recommend that the Secretary of Defense require the Under Secretary for Acquisition and Technology to assume a stronger management role to resolve simulation issues by defining JSIMS and developing a definitive plan of action and developing a transition strategy to phase out ALSP and phase in JSIMS. This strategy should be based upon cost estimates associated with modifying, expanding, and testing the ALSP Confederation to decide which improvements to the ALSP Confederation provide benefits that are cost-effective.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of our report, DOD generally agreed with our findings and recommendations (see app. I). The Department said that it recognizes the shortcomings of today’s joint training simulations and is committed to developing more cost-effective capabilities. In response to our recommendations, DOD said that it has taken action to establish a joint funding line for the JSIMS core and to ensure service support for their respective combat representations. DOD stated that a plan to phase out ALSP and phase in JSIMS will be developed based on both technical considerations provided by the Under Secretary of Defense for Acquisition and Technology and operational considerations provided by the services and CINCs.
However, DOD did not agree with our assessment of the status of the JSIMS program. The Department does not believe that the JSIMS program has been stalled. DOD said that (1) it deliberately established ambitious milestones in the JSIMS memorandum of agreement to serve as an action to move the project along; (2) the JSIMS project has moved from a general consensus agreement, through stand-up of a transitional JSIMS Joint Program Office, to the formation of a permanent Joint Program Office; (3) a systems definition for JSIMS was developed in an April 1995 meeting; (4) the JSIMS Operational Requirements Document is in final review; and (5) the program officially entered the Concept Exploration and Definition phase when it attained milestone 0 status during July 1995.
Our assessment of the status of JSIMS is based upon documentation provided to us during our review. The various management groups responsible for development of JSIMS have conducted numerous meetings in an effort to bring about a consensus of what JSIMS constitutes. However, we believe that JSIMS has been stalled at a fundamental level as evidenced by the minimal progress since the signing of the June 1994 memorandum of agreement. At the conclusion of our review, there were indications that the program might be progressing. However, no actions had been finalized. A permanent charter for the Joint Program Office as called for by October 1994 is still not established. The JSIMS Operational Requirements Document is still not approved. According to documents presented at the July 1995 JSIMS Senior Review Board meeting, the estimated cost to develop JSIMS core and warfare functions is $416 million. The JSIMS core without the warfare functions will not achieve DOD’s joint training objectives. Therefore, we believe it is important to identify all development costs.
DOD said that it could not substantiate the $40 million we identified that the services are planning to spend on ALSP improvements. DOD stated that $6.1 million is currently budgeted for ALSP core support through fiscal year 1999. DOD acknowledged that all other funding for modifications in the ALSP models is provided by the services or CINCs but could not substantiate this figure. The $40 million figure was derived from documents and discussions held with service budget officials and is subject to change depending upon the services’ priorities for spending.
section_title: Scope and Methodology
paragraphs: To determine whether DOD is progressing with its development of JSIMS, we interviewed knowledgeable officials from the Defense Modeling and Simulation Office, Washington, D.C.; the Joint Staff, Washington, D.C.; the Joint Warfighting Center, Fort Monroe, Virginia; the JSIMS Joint Program Office, Orlando, Florida; and the services’ modeling and simulation management offices in Washington, D.C. In addition, we interviewed the Director, Defense Research and Engineering, Office of the Secretary of Defense. We reviewed the draft DOD Modeling and Simulation Master Plan; the Executive Council for Modeling and Simulation meeting minutes; and DOD, Joint Staff, and service modeling and simulation policies. In addition, we reviewed related Defense Science Board and DOD Inspector General reports.
To determine whether DOD’s decisions to improve the ALSP Confederation are cost-effective, we interviewed modeling and simulation officials at the Simulation, Training, and Instrumentation Command, Orlando, Florida; the Warrior Preparation Center, Einsiedlerhof Air Station, Germany; the Joint Training Analysis and Simulation Center, Suffolk, Virginia; and the National Simulation Center, Fort Leavenworth, Kansas. We reviewed numerous documents on the ALSP Confederation. We discussed the costs of simulation improvements with each of the service model’s proponents.
We conducted our work between January 1995 and August 1995 in accordance with generally accepted government auditing standards.
We are sending copies of this report to the Chairmen, Senate and House Committees on Appropriations, Senate Committee on Armed Services, and House Committee on National Security; the Secretary of Defense; the Deputy Secretary of Defense; the Under Secretary of Defense for Acquisition and Technology; the Director, Defense Research and Engineering; and the Secretaries of the Army, the Navy, and the Air Force. We will make copies available to others on request.
Please contact me at (202) 512-5140 if you or your staff have any questions concerning this report. Major contributors to this report were  Charles J. Bonanno, Brenda S. Farrell, Raymond G. Bickert, and Colin L. Chambers.
section_title: Comments From the Department of Defense
paragraphs: The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.