Chemical Weapons: Lessons Learned Program Generally Effective but Could Be Improved and Expanded
Released: 2002-09-10T00:00:00

Letter

Background

In 1985, Congress required the Department of the Defense to destroy the U.S. stockpile of chemical agents and munitions and to establish an organization within the Army to manage the agent destruction program. Later, Congress also directed DOD to research and develop technological alternatives to incineration for disposing of chemical agents and munitions. These activities evolved into the Chem-Demil Program. The Chem-Demil Program includes the Chemical Stockpile Emergency Preparedness Program, created in 1988, to enhance the emergency management and response capabilities of communities near the storage sites in case of an accident. The Nonstockpile Chemical Materiel Product was added in 1993 to destroy any chemical weapons or materiel not included in the stockpile disposal program.

The Chemical Stockpile Disposal Project has or plans to use incineration to destroy chemical agents at five sites: Johnston Atoll in the Pacific Ocean; Anniston, Alabama; Pine Bluff, Arkansas; Umatilla, Oregon; and Tooele, Utah. Tooele is the only site with a facility currently operating. The three other stateside facilities are scheduled to begin operations in fiscal years 2002-2003. The Johnston Atoll facility has finished destroying its stockpile and is being closed. The Alternative Technologies and Approaches Project will use non-incineration methods (such as agent neutralization by chemical treatment) to destroy agents in bulk containers at Newport, Indiana, and Aberdeen, Maryland. The Assembled Chemical Weapons Assessment Program is also researching alternative methods to destroy agents in weapons at Pueblo, Colorado, and Blue Grass, Kentucky.

The Office of the Secretary of Defense and the Department of the Army share management roles and responsibilities in the Chem-Demil Program. The Program Manager of the Assembled Chemical Weapons Assessment Program reports to the Under Secretary of Defense for Acquisition, Technology, and Logistics. Thus, it is independent of the Program Manager for Chemical Demilitarization, who reports to the Assistant Secretary of the Army (Installations and Environment).

In 1997, the United States ratified the Chemical Weapons Convention, a treaty committing member nations to dispose of selected chemical agents and materiel by April 29, 2007. In September 2001, the Army updated the life cycle cost estimate for the Chem-Demil Program from $15 billion to $24 billion. The new cost estimate extended the agent destruction schedule at four of the eight stateside sites beyond the initial target date of April 2007. Despite setbacks experienced at Johnston Atoll, Tooele, Utah, and Umatilla, Oregon, among others, the incineration program has successfully destroyed over 25 percent of the original stockpile (see table 1).

The Lessons Learned Program was created in part because many different contractors were involved in the incineration program, and a system was needed to collect and preserve the institutional knowledge and acquired experience. The program is intended to identify, capture, evaluate, store, and share (implement) lessons learned during the different phases of the chemical stockpile demilitarization process. It collects two different kinds of lessons: “design” lessons covering engineering and technical processes and “programmatic” lessons involving management, quality assurance, emergency response, and public outreach. As criteria for assessing the knowledge management processes used by the Lessons Learned Program, we identified four of a number of federal organizations that practice knowledge management and operate lessons learned programs. In making our selections, we reviewed literature and spoke with knowledge management experts to find organizations recognized for their ability to share lessons or effectively manage knowledge. We identified the following organizations: the Center for Army Lessons Learned, the Department of Energy, the U.S. Army Corps of Engineers, and the Federal Transit Authority (for more details, see appendix II).

There are two levels of authority involved in developing lessons learned from proposed engineering changes. A Configuration Control Board composed of headquarters staff in the Office of the Program Manager for Chemical Demilitarization has authority to approve, reject, or defer engineering change proposals that involve costs above a set limit or affecting multiple sites. The Field Configuration Control Boards have authority over changes at their sites involving lower costs. In September 2001, the Lessons Review Team (consisting of headquarters staff) was established to screen all lessons and engineering changes and provide the information needed to determine which lessons require a response from sites. For more information on the lessons learned process, see appendix III.

Lessons Learned Program Has Made Positive Contributions but Needs Improvement

The Lessons Learned Program has made valuable contributions in support of the Chemical Stockpile Disposal Project’s efforts to safely destroy the chemical stockpile. It has generally operated consistently with knowledge management principles and lessons sharing best practices and has successfully captured and shared thousands of lessons. However, the program does not apply or incorporate all knowledge management principles and lessons sharing best practices. For example, the program does not provide needed guidance for senior managers; it does not have formal a validation procedure to determine whether a problem has been fixed; and the database of lessons learned needs improvement.

Important Program Contributions

The Lessons Learned Program has contributed to the Chem-Demil Program’s goal of destroying the chemical weapons stockpile while promoting safety, maintaining schedule, and saving or avoiding costs. We found that the Chem-Demil Program’s management, through its leadership, encourages headquarters, field staff, and contractor personnel in the incineration program to use the Lessons Learned Program. It has provided funding and has established processes to capture, evaluate, store, and share lessons. It is committed to continuous improvement and has provided the technology needed to support the lessons learned process. Finally, it fosters a culture in which knowledge sharing is an important element of day-to-day operations.

While it is difficult to quantify the benefits of each lesson, available data indicate that lessons learned have generally helped avoid on-the-job injuries (by using government-furnished-approved tools that are better suited to specific tasks), reduce costs (by improving the containers used to transport weapons), or maintain schedules (by improving the design of a socket to disassemble weapons). We also found that lessons from accidental releases of chemical agents at Johnston Atoll and Tooele, Utah, were implemented at other incineration sites under construction, thus incorporating improvements into the design of those new facilities.

Program Lacks Guidance to Support Managers’ Decision Making

The Lessons Learned Program does not have guidance explaining how senior managers (at headquarters) should use it in support of their decision making process. Specifically, there is no guidance that defines the procedures to be followed when an alternative to a lesson is chosen or when a lesson is not implemented. Lessons learned guidance for another federal government agency recommends that lessons be used to optimize management decision making and to interact with other management tools such as reviews, investigations, root-cause analyses, and priorities.

We reviewed documentation of lessons learned from incidents at the Johnston Atoll and Tooele, Utah facilities, and found that three other facilities—Anniston, Umatilla, and Pine Bluff—had not implemented a lesson that had evolved from problems with pipes in the pollution abatement systems. The Tooele site had used a superior and more expensive material (hastelloy) to fix their problem than the material used at the other sites. Headquarters decided not to implement the lesson at the three sites primarily because it would have involved higher initial costs. This decision ultimately caused serious safety concerns, higher costs, and delayed the schedule. In February 2002, pipes at Anniston had failures similar to those experienced at the first two sites. This raised safety concerns and resulted in a 4-week delay to replace the pipes with hastelloy. It is too early to determine whether the material used at the Umatilla and Pine Bluff sites will have the same problems. Although they need flexibility to manage the program, senior managers also need guidance to help make decisions that allow them to consider the potential impact of not implementing lessons learned. This process would include safety and risk analyses that can provide criteria should they decide not to adopt a lesson learned.

Program Lacks a Procedure to Validate Lessons Implemented

There is no formal procedure to ensure that the lessons or corrective actions that have been implemented have fully addressed a deficiency. Chem-Demil Program guidance for engineering change proposals does require that changes be tracked and reported after implementation, but there is no similar requirement in the guidance for the Lessons Learned Program (which includes programmatic lessons). Both contractor and incineration project officials also confirmed that there are no procedures for monitoring the effectiveness of corrective actions. As a result, a problem could reoccur and affect safety and costs.

As shown in figure 1, the Lessons Learned Program process does not contain the final validation stage (dashed line), which most knowledge management systems and Army guidance consider as a necessary step. As we previously reported, Army guidance states that lessons learned programs should have a means for testing or validating whether a corrective action has resolved a deficiency. The standard issued for another federal lessons learned program indicates that analyses should be made to evaluate improvements or to identify positive or negative trends. The standard also states that corrective actions associated with lessons learned should be evaluated for effect and prioritized. Without such a validation procedure in the architecture of the Lessons Learned Program, there is little assurance that problems have been resolved, and the possibility of repeating past mistakes remains.

Database Is Difficult to Use, Lessons Are Not Prioritized

The lessons learned database includes about 3,400 issues, 3,055 engineering change proposals, and 2,198 lessons. But it is not easy to obtain fast and ready access to relevant information. Furthermore, the lessons in the database are not prioritized, making it difficult to identify which lessons are most important and which need to be verified and validated.

It is important that an organization employ appropriate technology to support the participants of a lessons learned program. Having a technology be available does not automatically guarantee its use or acceptance. According to lessons sharing best practices, the goal of technology is to (1) match a solution to users’ needs, (2) establish a simple content structure so that items may be found easily and retrieved quickly, and (3) deliver only relevant information from all possible sources. According to database users we interviewed and surveyed, it is difficult to find lessons because the search tool requires very specific key words or phrases, involves multiple menus, and does not link lessons to specific events. As a result, some users are reluctant to use the database and thus may not benefit from it when making decisions that affect the program.Many users who responded to our survey stated that they experienced difficulties in searching the database, and some we interviewed described specific problems with searches. One described the database as “frustrating.” We tested the search tool and also had difficulty finding lessons linked to specific incidents.

Users we interviewed made a number of suggestions to improve the Lessons Learned Program’s database, including improving the search capability, organizing by subject matter, ranking or prioritizing lessons, creating links to other documents, providing a Web-based link to the database, periodically purging redundant data, and making access screens more user-friendly.

Furthermore, because the database does not prioritize lessons, managers may be unaware of some important areas or issues that need to be monitored or lessons that need to be reviewed and validated. By contrast, lessons learned processes used by the selected federal agencies include periodic reviews of the usefulness of lessons and the archiving of information that is no longer pertinent or necessary. The processes also include prioritizing lessons by risk, immediacy, and urgency. In 1998, the Army Audit Agency recommended that the database be purged or archived of obsolete items and that current and future lessons be prioritized. In September 2001, the Chem-Demil Program created a Lessons Review Team to begin identifying “critical” lessons (those requiring a response). But the team is not prioritizing lessons.

Some Knowledge Management Principles Are Not Applied

Several other areas also did not adhere to knowledge management principles and lessons sharing best practices. For example, the Chem- Demil Program’s management plan does not explain how the Lessons Learned Program is to achieve its goals or define performance measures to assess effectiveness. Knowledge management principles stress the importance of leaders articulating how knowledge sharing will be used to support organizational goals. Furthermore, the Chem-Demil Program does not provide incentives to encourage involvement in the Lessons Learned Program. Lessons sharing best practices and knowledge management principles prescribe developing and using performance measures to determine the effectiveness of a program. In addition, the Lessons Learned Program currently surveys employees after workshops to measure their satisfaction; however, these surveys are not sufficient to assess the overall effectiveness of the program. The program is attempting to identify ways to measure the cost and benefits derived from lessons learned. Knowledge management principles also encourage using performance evaluation, compensation, awards, and recognition as incentives for participation in lessons learned programs. The lack of incentives in the Lessons Learned Program may lead to missed opportunities for the identification and sharing of lessons learned.

Sharing of Lessons Learned Could Be Expanded

The Lessons Learned Program has shared thousands of lessons among the five incineration sites through the different phases of construction, testing, and destruction of chemical agents. However, as the Chem-Demil Program evolved through the 1990s, and as the components using alternative technologies were added, the scope of the Lessons Learned Program did not expand to share lessons with the new components (see app. IV for a history of the Chem-Demil Program’s evolution). The Lessons Learned Program remained primarily focused on the five incineration sites. At the same time, each stockpile destruction component developed its own separate lessons learned, but without any program wide policies or procedures in place to ensure coordination or sharing of information across components. We reported in May 2000 that effective management of the Chem-Demil Program was being hindered by a complex organizational structure and ineffective coordination. This has created barriers to sharing.

Today, the four sites that are likely to use alternative technologies are not full participants in the lessons learned effort: The Assembled Chemical Weapons Assessment Program does not fully participate in the lessons learned process or activities. In at least one instance, the Assembled Chemical Weapons Assessment Program requested (from the Program Manager for Chemical Demilitarization), a package of data including lessons on the pollution abatement system filters, mustard thaw, and cost estimates. The data were eventually provided, but they were too late to be used during a DOD cost data review. This lack of access forced the program to submit incomplete cost data for the review because it was unable to obtain information from the incineration project in a timely manner.

The Alternative Technologies and Approaches Project does have access to the Lessons Learned Program’s database, and it plans to develop its own separate database that it will share with the Lessons Learned Program only at “key milestones.” The project’s information, however, could be very valuable to other components of the Chem- Demil Program, especially the Assembled Chemical Weapons Program, which also researches alternative technologies. This plan could lead to lost opportunities and duplication of efforts.

Many of the lessons learned by the incineration project could be used by the other components of the Chem-Demil Program to promote safe, cost- effective, and on-time operations. Many of the technical processes (storing, transporting, unloading, and disassembling weapons) and programmatic processes (regulatory compliance, management, public relations practices) used by the Chemical Stockpile Disposal Project are very similar to those used by the other programs. This is also the case for processes used to develop operating destruction, or throughput, rates and cost and schedule projections. In fact, the majority of processes at incineration facilities are the same as those used by the Assembled Chemical Weapons Assessment Program and the Alternative Technologies and Approaches Project. Under these circumstances, promoting a culture of knowledge sharing would enable all components to capture and use organizational knowledge.

Furthermore, there is the possibility that the Pueblo, Colorado, site (and possibly the Blue Grass, Kentucky, site) now managed by the Assembled Chemical Weapons Assessment Program, which now reports to a DOD office, may be transferred to the Army’s Chem-Demil Program. If this transfer of responsibilities does take place, it would be important for the two programs to be already sharing information fully and seamlessly. Even if the transfer does not take place, knowledge management principles and lessons sharing best practices both dictate that components of the same program should share information, especially if they all have a common goal.

Conclusions

The Lessons Learned Program has made important contributions to the safe destruction of the nation’s stockpile of chemical weapons. We found that the program generally adheres to knowledge management principles and lessons sharing best practices. However, the program’s full potential has not been realized. The program needs guidance to help senior managers make decisions that allow them to weigh the potential impact of not implementing lessons learned. This guidance would be a set of procedures, including safety and risk analyses, to be followed before deciding to counter a lesson learned. Without such guidance, decision makers, in at least one case, chose lower cost over safety and schedule, ultimately at the expense of all three. Also, the Lessons Learned Program lacks procedures to validate the effectiveness of implemented lessons. The lack of a validation step partially defeats the purpose of the lessons learned process, which relies on the confirmed effectiveness of solutions emerging from knowledge and experience. If the effectiveness of a lesson cannot be validated over time, problems may emerge again, with a negative impact on safety, costs, and schedule.

Further, the information in the lessons learned database is not easily accessible or prioritized. These drawbacks have frustrated users and may discourage them from using the database. This could lead to wrong or misinformed decisions that could affect safety. In addition, there is no overarching coordination or sharing of information across all the components of the Chem-Demil Program, which grew and evolved over time without policies or procedures to ensure that knowledge would be captured and communicated fully. As a result, fragmented or duplicative efforts continue today, and the Assembled Chemical Weapons Assessment Program in particular lacks access to important data maintained by the Chemical Stockpile Disposal Project and the Alternative Technologies and Approaches Project. In the case of the Chem-Demil Program, the absence of policies and procedures promoting and facilitating the broadest dissemination of lessons learned places the safety, cost effectiveness, and schedule of the chemical weapons destruction at risk.

Recommendations for Executive Action

To improve the effectiveness and usefulness of the Chemical Demilitarization Program’s Lessons Learned Program, we recommend that the Secretary of Defense direct the Secretary of the Army to develop guidance to assist managers in their decision making when making exceptions to lessons learned, develop procedures to validate, monitor, and prioritize the lessons learned to ensure corrective actions fully address deficiencies identified as the most significant, and improve the organizational structure of the database so that users may easily find information and develop criteria to prioritize lessons in the database.

We also recommend that the Secretary of Defense direct the Secretary of the Army to develop policies and procedures for capturing and sharing lessons on an ongoing basis with the Alternative Technology and Approaches Project and in consultation with the Under Secretary of Defense (Acquisition, Technology, and Logistics) develop policies and procedures for capturing and sharing lessons on an ongoing basis with the Assembled Chemical Weapons Assessment Program.

Agency Comments and Our Evaluation

The Army concurred with our five recommendations and provided explanatory comments for each one. However, these comments do not address the full intent of our recommendations. With regard to our recommendation that it provide guidance to assist managers when deciding to make an exception to a lesson, the Army stated that the Lessons Review Team has guidance for characterizing the severity level of lessons learned. However, as our report clearly points out, this guidance is for site officials and is insufficient in assisting senior managers at headquarters on important decisions involving costly lessons that could potentially impact several sites. We believe that good management practices require that senior managers make decisions based on risk, safety, and cost analyses and that guidance should be developed to support this decision-making process as we recommended.

In concurring with our recommendation to develop procedures to ensure corrective actions fully address deficiencies, the Army stated that it is initiating an effort whereby the system’s contractors will be responsible for validating, monitoring, and prioritizing lessons. The Army’s Lessons Learned Program currently does not validate the results of corrective actions. Contracting this important function will require monitoring by the Chem-Demil program to ensure that validation is properly conducted as we recommended.

The Army stated that it has improved the Lessons Learned database to make it easier to locate information. Converting the database to an Internet-based program should also improve its accessibility and utility. Although these actions address some users’ concerns, the Army needs to address all related user issues identified in our report in order to improve the benefits of the database.

The Army concurred with our recommendation to develop policies and procedures to capture and share lessons with the two alternative technology programs. It stated that progress had been made toward sharing lessons between the Alternative Technologies and Approaches Project and the Lessons Learned Program at key milestones. The Army also said it has shared the lessons database with the Assembled Chemical Weapons Assessment Program. However, the Army should require, as we recommended, that policies and procedures for capturing and sharing lessons on an ongoing basis be established, instead of sharing at key milestones and on a one-way basis. This approach would ensure that both alternative technology programs fully participate in the Lessons Learned Program and that the database is constantly enriched to enhance safety, cost, and schedule based decisions for all components of the Chem-Demil program.

The Army’s comments are printed in appendix V. The Army also provided technical comments, which we incorporated where appropriate.

We are sending copies of this report to interested congressional committees, the Secretaries of Defense and of the Army; the Assistant Secretary of the Army (Installations and Environment); the Under Secretary of Defense (Acquisitions, Logistics, and Technology); the Director, Federal Emergency Management Agency; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov Please contact me at (202) 512-6020 if you or your staff have any questions regarding this report. Key contributors to this report were Donald Snyder, Bonita Oden, Pamela Valentine, Steve Boyles, and Stefano Petrucci.

Appendix I: Information on the Incineration Process and Incidents at Three Sites

There have been three releases of agent from operating incineration facilities and one incident during construction that have generated several lessons learned. The incineration process and the releases and construction incidents are described below.

The Army’s Baseline Incineration Process

A baseline incineration process uses a reverse-assembly procedure that drains the chemical agent from the weapons and containers and takes apart the weapons in the reverse order of assembly. Once disassembled, the chemical agent and weapon parts are incinerated in separate furnaces and the gaseous and solid waste is treated in a separate process. Liquid brine resulting from the treatment of exhaust gases in the pollution abatement system is dried to reduce the volume and transported to a commercial hazardous waste management facility.

The path to weapons disposal, in general includes six major steps. 1. Chemical weapons are stored in earth-covered, concrete-and steel buildings called igloos. These igloos are guarded and monitored for any signs of leaking weapons by the U.S. Soldier and Biological Chemical Command. 2. Chemical weapons are taken from the igloos and transported to a disposal plant in sealed on-site containers by the U.S. Soldier and Biological Chemical Command. The sealed containers are resistant to fire and impact. 3. When the on-site containers arrive at the disposal plant, workers check them for leaking weapons before opening them. Chem-Demil crews then load the weapons onto conveyors that carry the weapons through the disposal process. When the weapons are loaded onto the conveyor, the U.S. Soldier and Biological Chemical Command no longer has responsibility for them. 4. From this point on, workers manage the disposal process from an enclosed control room using advanced robotics, computer technology, and video monitoring equipment. Automatic, robotic equipment drains the chemical agent from the weapon and takes the weapons apart in explosive proof rooms. 5. Once dismantled and drained, the individual weapon parts travel to different furnaces in the plant, each designed for a specific purpose. The liquid incinerator destroys the chemical agent, the deactivation furnace destroys explosive materials, and the metal parts furnace heats shell casings and other heavy metal parts to destroy any remaining agent contamination. 6. The pollution abatement system cleans the air before it is released into the environment.

Agent Release at Tooele

The Tooele Chemical Agent Disposal Facility (Tooele plant) is located on Deseret Chemical Depot in Tooele, Utah. The facility is designed to dispose of 44.5 percent of the nation’s original stockpile of chemical weapons. Tooele plant is the first chemical weapons disposal facility built within the continental United States. Construction of the Tooele plant began in October 1989 and disposal operations began in August 1996. Operations at Tooele plant should be completed in 2008. The Tooele plant incorporates systems originally tested and used at the Chemical Agent Munitions Disposal System, also located at the depot. These systems were first used on an industrial scale at the Army’s Johnston Atoll Chemical Agent Disposal System (Johnston Atoll plant) in the Pacific Ocean. The Johnston Atoll plant was the first integrated facility built to dispose of chemical weapons.

The sequence of events described in table 3 is based on documents from the Utah Department of Environmental Quality—Division of Solid and Hazardous Waste, U.S. Army Safety Center, Department of Health and Human Services—Centers for Disease Control and Prevention, and a program contractor. On May 8, 2000, the day shift was processing rockets in the deactivation furnace system. The deactivation furnace system lower tipping gate (used to control the feed of munitions to the furnace) did not close properly and munitions/agent processing was terminated. Workers in protective gear began to clean and repair the gate and a strainer. A bag from the strainer, contaminated with GB (nerve) agent, was left on top of the gate. This is believed to be the source of the agent that was released. Vapors were drawn from the bag through the furnace system.

During the initial attempt to re-light the afterburners following the cleaning procedure, the agent monitoring equipment alarmed. During a second attempt to re-light these burners another agent monitor alarmed. In summary, a small amount of agent escaped through the common stack during attempts to relight the furnace. (See table 2.)

The several corrective actions taken were based on 105 investigation findings involving operations, training, and equipment. Lessons learned from this incident include (1) modifying feed chute clean out procedures, (2) providing operator refresher training, (3) installing a deactivation furnace remote operated valve to isolate the deactivation furnace during afterburner re-lights, and (4) redesigning deactivation furnace feed chute.

Agent Releases at Johnston Atoll

In addition to reviewing lessons from Tooele incidents, we were briefed on two incidents that occurred at Johnston Atoll, and we reviewed relevant investigation reports for these incidents. Both incidents resulted in corrective actions and generated several lessons learned.

On March 22, 1994, the liquid agent gun purge process began. The next day workers dressed in protective gear removed the liquid agent gun, and three lines had to be disconnected and capped (sealed). These three lines to the liquid agent gun are the atomizing air, fuel oil, and the agent line. During the disconnecting of the agent line, the liquid incineration room agent monitoring system alarmed. Also, the agent monitors in the common stack began to alarm. Operators turned off the induction fan to divert room air out through plant exhaust to the carbon filters.

Lessons learned from this incident include (1) replacing the fuel oil purge system flow meter with an instrument that could be read in the control room; an investigation found that the flow meter on the agent purge line was not functioning (2) directing room air away from the pollution abatement system to prevent contaminated air from escaping through the duct work without going through the furnace and (3) counseling workers on the importance of following approved standard operating procedures.

On December 8, 1990, a laboratory analysis confirmed emission of chemical agent from the common stack following a purging (flushing) of the agent line. It was determined that the probable cause of the release was that a quantity of agent GB (nerve) leaked from the agent gun or feed line into the primary chamber of the liquid incineration furnace, and the agent was swept downstream by the induced draft fan (used to draw air through the plant) while the furnace was in a cool-down cycle. It appears that the agent that leaked into the incinerator and ultimately discharged to the atmosphere was from either valves in the agent feed line to the primary chamber that were not totally sealed or the agent remained in the agent line after it was purged and was aspirated into the incinerator and subsequently the atmosphere. During the incident, and due to a malfunctioning agent-sampling probe, the agent-monitoring equipment in the common stack did not detect agent.

Lessons learned from this incident include (1) improving the process to purge (flush) chemical agent from the feed line by adding a fuel oil purge and increasing the purge cycle to ensure a complete purge, (2) modifying the alarm system in the common stack to provide redundancy and test the alarms more frequently, and (3) closing all four valves after the agent line is purged and process activities involving the liquid incineration feed system when the furnace is cooling down to the charcoal filters.

Construction Incident at Umatilla

On September 15, 1999, more than 30 construction workers were affected by an irritating vapor in the air while working in the munitions demilitarization building. This incident caused many workers to experience respiratory irritation, sending them to the local hospital where they were examined and released. Later that day, all construction work stopped and approximately 800 contracted workers were sent home. Investigations and analyses lead to the determination that chemical agent was not involved; instead this was determined to be a construction incident.

As construction progressed, the building became a “closed-in” area and may not have been adequately ventilated. The building ventilation system was not designed to control contaminants during construction; it was only intended to control a release of chemical agent when construction was complete and operations had begun. The release of 800 contracted- construction workers without informing them of the situation that no chemical agents were involved, coupled with the slow release of information to the press, eventually heightened public concern.

Lessons learned from this incident include (1) enhancing local ventilation in the munitions demilitarization building, (2) establishing and posting evacuation routes and response procedures throughout the site, (3) installing a temporary public address system at the construction site, and (4) ensuring there is adequate communications between the site and any off-site facilities particularly in the event of an incident.

Agent Exposure at Tooele

On July 15, 2002, at the time we were drafting this report, an individual working at the incineration facility in Tooele, Utah, experienced a confirmed accidental chemical agent exposure. This individual was performing maintenance on an agent purge line valve in the liquid incinerator room and was exposed to residual agent present in the agent purge line. The worker exhibited symptoms of chemical agent exposure. Although the Army, DOD Inspector General, and the facility’s contractor are conducting investigations looking into the events associated with the accidental exposure, it is too early to report on lessons resulting from this incident. The Program Manager for Chemical Demilitarization is awaiting the investigation reports and will incorporate the corrective actions into lessons learned. According to the Army, agent operations will not commence until all corrective actions have been taken and the plant is deemed safe to operate.

Appendix II: Scope and Methodology

To assess the Lessons Learned Program, we reviewed literature on the principles of knowledge management and our previous reports on lessons sharing best practices.

To assess the leadership of the Lessons Learned Program, we interviewed Chem-Demil Program managers, personnel, and the contractor staff who manage the Lessons Learned Program. We also reviewed management documents describing the program and we conducted 30 structured interviews with the Chem-Demil Program’s managers (headquarters and field level) and systems contractor staff at three sites (Aberdeen, Maryland; Anniston, Alabama; and Tooele, Utah) to determine how clearly management articulated its expectations about using lessons learned. We did not select a statistical sample of database users; therefore, our survey results cannot be generalized to all Lessons Learned Program database users.

To describe the lessons learned process, we reviewed documentation relevant to the lessons learned process. We also interviewed personnel from the office of the Program Manager for Chemical Demilitarization, the Anniston, Alabama, site, and the contractor responsible for managing the Lessons Learned Program.

To learn how technology supports the Lessons Learned Program, we reviewed the lessons learned process and identified the methods used to gather, consolidate, and share information with stakeholders. We also asked the staff we surveyed how effectively does the program’s technology tools support the lessons learned process.

To determine whether the Chem-Demil Program fosters a culture of knowledge sharing and use, we talked to program managers for each Chem-Demil Program components, headquarters staff, and personnel from the lessons learned contractor staff to determine how lessons are shared and whether employees are encouraged to participate in the program. We also asked the staff we surveyed how frequently they submitted information to the program, whether they used the lessons, and whether there were incentives to encourage participation.

To determine whether lessons learned contributed to the goals of the destruction program; we documented and reviewed several important lessons that program staff identified. We also traced several lessons from incidents at Johnston Atoll and Tooele to verify that they had been shared and implemented at the Anniston facility. We used unverified Army data to assess whether the Lessons Learned Program achieved its aim of reducing or avoiding unnecessary costs. To determine if the Lessons Learned Program process conforms to other programs’ lessons sharing processes we identified four of a number of federal organizations that practice knowledge management and operate lessons learned programs. In making our selections, we reviewed literature and spoke with knowledge management experts to find organizations recognized for their ability to share lessons learned or effectively manage knowledge. We obtained information from the Center for Army Lessons Learned, the Department of Energy, the U.S. Army Corps of Engineers, and the Federal Transit Authority. We interviewed representatives from each organization about the processes they used for identifying, collecting, disseminating, implementing, and validating lessons learned information. We reviewed their lessons learned program guidance to compare and contrast their practices with the incineration project’s Lessons Learned Program process. We also interviewed an expert familiar with the program about the management of the lessons learned process. To assess the search, linkage, and prioritization of the database, we obtained documentation and interviewed the contractor staff about the information in the database. We tested the search feature of the database, including accessing menus, keyword and category listings, and analyzed several lessons learned we had obtained from our searches. We obtained opinions from the staff we surveyed on the effectiveness of the lessons learned database and their suggested areas of improvement. The respondents included managers and others with an average of 9 years experience in the Chem-Demil Program. The staff we surveyed routinely search the database for lessons learned information. We did not select a statistical sample of database users; therefore, our survey results cannot be generalized to all Lessons Learned Program database users.

To assess the extent to which lessons learned have been shared, we interviewed the Program Manager for Chemical Demilitarization and the contractor responsible for operating the Lessons Learned Program. We also attended status briefings for each Chem-Demil component. We focused our work primarily on the stockpile destruction projects/programs. We conducted interviews with officials from the Alternative Technologies and Approaches Project, the Assembled Chemical Weapons Assessment Program, and the Chemical Stockpile Disposal Project to gather evidence on the commonality the alternative technology components have with the incineration program and the extent to which they share lessons learned information. To determine whether each component participated in the Lessons Learned Program by either sharing or receiving lessons learned information, we reviewed workshop minutes from calendar years 2000 and 2001.

To describe the incidents at three sites, we attended briefings on the incidents provided by officials from the incineration program, and reviewed incident investigation reports and entries in the Lessons Learned database. We identified key lessons from these sources and toured the Anniston Chemical Disposal Facility, to determine whether lessons learned had been shared and implemented. During our visit, we observed that several lessons from the Tooele incident, among others, were implemented.

Appendix III: Lessons Learned Process

The Lessons Learned Program was established to collect and share lessons learned within the incineration program. The Programmatic Lessons Learned Program uses various methods to identify, review, document, and disseminate lessons learned information among government and contractor personnel. The program uses facilitated workshops to introduce lessons and also takes lessons from engineering change proposals. The Lessons Review Team reviews issues and determines specific lessons to be implemented. These issues, engineering changes, and lessons are stored in a database.

The program uses five distinct steps to develop lessons learned, as shown in figure 2.

Issues are raised through topics submitted to workshops (meetings of headquarters and site personnel), critical document reviews (of changes to program documents), engineering change proposals (technical changes at one or more sites), quick reacts (immediate action), and express submittals (information from a site.)

Experts review issues to determine if a change should be initiated in a workshop, an assessment (a study to support a management recommendation for change), engineering change proposal review process (a team at each site reviews changes at other sites), and directed actions (requests for information on actions a site has taken.)

Lessons are identified from workshops, assessment reports, and the lessons review team (headquarters activity to segregate lessons into response required or not required.)

Issues and lessons are stored in the database.

Lessons are then shared with stakeholders, including contractor personnel, through access to the database, technical bulletins (a quarterly publication with information of general interest to multiple sites), programmatic planning documents (containing policies, guidelines, management approaches, and minimum requirements), and site document comparisons (new documents with baseline documents.)

Four primary elements of these steps are discussed below.

Lessons Learned Program Facilitated Workshops

Facilitated workshops are the primary method for introducing lessons learned into the Lessons Learned Program. Facilitated workshops are meetings that offer an environment conducive for site and headquarters personnel to speak openly about experiences. The intent of the workshops is to allow program personnel familiar with particular subjects to hold detailed discussions of issues relative to specific subjects. All issues discussed in the workshops are entered into the database and later reviewed to determine if the issues should become lessons learned.

The facilitated workshop process begins with a memorandum that requests site personnel from the Lessons Learned Program team to identify topics they want to discuss in workshops. These topics are generally divided into three basic categories: (1) valuable information provided to other sites, (2) challenging issues and discussion of issues with other sites in anticipation of possible recommendations, and (3) general topics to discuss different approaches to a problem. After each workshop, a feedback survey is sent to participants to determine user satisfaction with workshops.

Engineering Change Proposal Review Process

Engineering change proposals are the primary method of approving and documenting design changes at the sites. Members of the Configuration Control Board and the Field Configuration Control Boards are responsible for reviewing and approving engineering change proposals within certain dollar limits. The Configuration Control Board, consisting of members from headquarters, is also responsible for managing changes to items or products identified for configuration control, such as facilities and equipment in order to maintain or enhance reliability, safety, standardization, performance, or operability. Each Field Configuration Control Board consists of members from a site, and is responsible for controlling engineering changes during construction, systemization, operations, and closure of facilities. Engineering change proposals are discussed during bi-weekly teleconferences where the sites can ask the originating site questions about the proposed engineering change.

The Field Configuration Board is responsible for approving engineering change proposals with an estimated cost of $200,000 or less. The Configuration Control Board is responsible for approving proposals with an estimated cost of $200,001 to $750,000. Proposals over $750,000 are sent to the Project Manager for Chemical Stockpile Disposal for approval. After approval, the engineering change proposals are reviewed and input into the database and sent to the Lesson Review Team as part of the review process.

Engineering changes are the primary source of design-related lessons learned. Engineering change proposals are approved changes in the design or performance of an item, a system or a facility. Such changes require change or revision to specifications, engineering drawing, and/or supporting documents. Consequently, the Program Manager for Chemical Demilitarization developed a review process as a method to capture these lessons in the Lessons Learned Program. The purpose of the Engineering Change Proposal Review Process is to provide Chemical Demilitarization sites with more control over lessons learned decisions and incorporate lessons learned sharing under the Lessons Learned Program. Additionally, the review process is structured to allow each site the opportunity to review engineering changes being implemented at other sites and consider the applicability to their site. The review team consists of members from the sites, the Program Manager’s office, the Lessons Learned Program team, and the U.S. Army Corps of Engineers.

Lessons Review Team

The Lessons Review Team, established in September 2001, is responsible for reviewing issues discussed in facilitated workshops to determine their potential impact and to determine if a specific site action is required. Additionally, the review team reviews engineering change proposals to determine if they are design-related lessons learned.

Issues are considered “lessons learned” when they have programmatic interest and significant impact on safety, environmental protection, or plant operations. The Lessons Review Team designates lessons learned as mandatory, “response required,” and “response not required.” A lesson is mandatory if the method of implementation has been or is directed from the Program Manager for Chemical Demilitarization headquarters. A lesson that is characterized as “response required” for means that the given site must provide information to the home office on the action taken to address the lesson. “Response not required” means that the site is not required to provide information to the headquarters on the action that the site has taken.

For mandatory lessons, the Lesson Review Team decision makers provide specific guidance for implementation of lessons. Technical support staff on the team conducts lesson reviews and provides recommendations to the decision maker regarding lessons. A team member is responsible for initial review of lessons and recommended designation, distribution of materials before the meetings, and facilitation of the meetings.

Lessons Learned Program Database

The Lessons Learned Program database is a repository for (1) issues generated from facilitated workshops, (2) engineering change proposals, (3) critical document reviews, (4) quick react/advisory system and other lessons learned process data, and (5) programmatic and design lessons learned. As of April 2002, the database contained 3,400 issues, 7,630 directed action, and 3,055 engineering change proposals.

The database was developed as a stand-alone program allowing users to employ search utilities or category trees to retrieve lessons. The program opens to the main screen, which consists of a search, categories, and lessons screens. The lessons screen is a search mechanism that utilizes a “drop down menu” enabling users to locate lessons by selecting categories or subcategories to narrow the search for lessons in a specific area. To summarize information and identify lessons in the lessons learned database, the database contains background information to support each lesson. The background information provides a condensed history, as well as the status of each lesson at the Chemical Demilitarization site.

Appendix IV: Chemical Demilitarization Program Management Developments, 1997-2001

The Departments of Defense and the Army made several changes to the management structure of the Chem-Demil Program, principally in response to congressional legislation. Originally the Program Manager for Chemical Demilitarization reported directly to the Assistant Secretary of the Army (Installations and Environment), who also oversees storage of the chemical weapons stockpile. The U.S. Soldier and Biological Chemical Command manages the stockpile. The Command also manages the loading, delivery, and unloading of chemical weapons at the destruction facility. After the estimated cost of the program reached a certain dollar amount, as required by statute, the Army formally designated it a major defense acquisition program. To manage this program in the Army acquisition chain, it was then transferred to the Assistant Secretary of the Army (Acquisition, Technology, and Logistics). The Program Manager for Chemical Demilitarization continued executing the program. In 1997, the Chemical Stockpile Emergency Preparedness Program was removed from the Program Manager for Chemical Demilitarization and transferred back to the Assistant Secretary of the Army (Installations and Environment) where it is currently managed by the U.S. Soldier and Biological Chemical Command. Also in 1997, the Army and the Federal Emergency Management Agency signed a new memorandum of agreement to better manage the on- and off-post emergency response activities, respectively.

In the 1997 Defense Appropriations Act (sec. 8065), Congress required the Assembled Chemical Weapons Assessment Program be independent of the Program Manager for Chemical Demilitarization and report directly to the Under Secretary of Defense (Acquisition and Technology). The purpose of this legislation was to separate the pilot program from the baseline incineration activities. Achievement of this goal also meant that two program offices would share responsibilities associated with disposal activities in Kentucky and Colorado. However, the pilot program’s legislation does not specifically state whether or not the Program Manager for Chemical Demilitarization will manage the assessment program once the development of technology evaluation criteria, the technology assessment, the demonstration, and pilot phases end.

In May 2000, we reported on the fragmented management structure and the inadequate coordination and communication within the Chem-Demil Program. We recommended that the Army should clarify the management roles and responsibility of program participants and establish procedures to improve coordination among the program’s various components. The Army, in December 2001, transferred the Chemical Demilitarization Program to the Assistant Secretary of the Army (Installations and Environment), bringing all components of the program, except the Assembled Chemical Weapons Program, under a single Army manager, as shown in figure 3. Another significant management change occurred in April 2002 when the Program Manager for Chemical Demilitarization retired after holding this position for the past 5 years.

Appendix V: Comments from the Department of the Army