Gulf War Illnesses: DOD's Conclusions About U.S. Troops' Exposure Cannot Be Adequately Supported
Released: 2004-06-01T00:00:00

Letter

Scope and Methodology

To determine the validity of DOD’s conclusion—that U.S. troops’ exposures to chemical warfare agents were as DOD estimates suggested— based on its plume-modeling analysis, we examined the meteorological and dispersion models DOD used to model chemical warfare agent releases from the U.S. demolition of Khamisiyah and Coalition bombings of Al Muthanna, Muhammadiyat, and other sites in Iraq during the Gulf War deployment period. We evaluated the basis for the technical and operational assumptions DOD made in (1) conducting the modeling for the bombing and demolition of Iraqi sites and (2) estimating the specific data and information used in the modeling, relating to source term, meteorological conditions, and other key parameters. We also evaluated the efforts of the CIA and DOD to collect and develop data on source term and other key parameters used in the modeling efforts.

We interviewed DOD and CIA modelers and officials involved with the modeling and obtained documents and reports from DOD’s Deployment Health Support Directorate. We also interviewed and received documents from DOE officials who were involved with the modeling at LLNL. In addition, we interviewed officials and obtained documents from the Institute for Defense Analyses (IDA) concerning the IDA expert panel assessment of CIA’s modeling of Khamisiyah. We also interviewed U.S. Army officials at Dugway Proving Ground, Utah, to determine how chemical warfare agents might have been released during the Khamisiyah pit area demolitions. Finally, we interviewed officials at the U.S. Army Center for Health Promotion and Preventive Medicine, to determine how specific troop unit exposures were identified, and officials of the United Nations Monitoring, Verification, and Inspection Commission (UNMOVIC), to obtain information on source term data from the United Nations Special Commission’s (UNSCOM) analyses and inspections of the Khamisiyah, Al Muthanna, Muhammadiyat, and other sites.

To determine the validity of DOD’s and the Department of Veterans Affairs’ (VA) conclusions—based on epidemiological studies—that there was no association between Khamisiyah exposure and the rates of hospitalization or mortality, we reviewed published epidemiological studies in which hospitalization and mortality among exposed and nonexposed U.S. troops were analyzed. We also interviewed the study authors and researchers and examined the Gulf War population databases provided to the researchers by DOD in support of these studies. We interviewed Veterans Benefits Administration officials and obtained documents and reports on their analyses of DOD’s population databases.

We did not examine whether plume modeling data were being used by VA to determine eligibility for treatment or compensation.

In an effort to identify the total costs associated with modeling and related analyses of chemical warfare agent releases during the Gulf War; we interviewed relevant officials and collected cost data from various DOD agencies and DOD contractors who supported the modeling efforts.

To determine the extent of British troops’ exposure to chemical warfare agent-related releases during the Gulf War, we interviewed British Ministry of Defense (MOD) officials in London and at Porton Down, and reviewed U.K. Ministry of Defense reports concerning the potential effects of exposure to chemical warfare agent-related releases on British forces.

We conducted our work from May 2002 through May 2004 in accordance with generally accepted government auditing standards.

Background

According to the CIA, modeling is the art and science of using interconnected mathematical equations to predict the activities of an actual event. In this case, modeling was used to determine the direction and extent of the plume from chemical warfare agents. In environmental hazard modeling, simulations recreate or predict the size and path (that is, the direction) of the plume, including the potential hazard area, and potential exposure levels are generated.

Information for Modeling

In addition to identifying the appropriate event to model, modeling requires several components of accurate information: the characteristics or properties of the material that was released and its rate of release (for example, quantity and purity; the vapor pressure; the temperature at which the material burns; particle size; and persistency and toxicity); temporal information (for example, whether chemical agent was initially released during daylight hours, when it might rapidly disperse into the surface air, or at night, when a different set of breakdown and dispersion characteristics would pertain, depending on terrain, plume height, and rate of agent degradation); data that drive meteorological models during the modeled period (for example, temperature, humidity, barometric pressure, dew point, wind velocity and direction at varying altitudes, and other related measures of weather conditions); data from global weather models, to simulate large-scale weather patterns, and from regional and local weather models, to simulate the weather in the area of the chemical agent release and throughout the area of dispersion; and information on the potentially exposed populations, animals, crops, and other assets that may be affected by the agent’s release.

Types of Models Used

Various plumes during the 1991 Gulf War were estimated using global- scale meteorological models, such as the National Centers for Environmental Prediction Global Data Assimilation System (GDAS) and the Naval Operational Global Atmospheric Prediction System (NOGAPS). Regional and local weather models were also used, including the Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS), the Operational Multiscale Environmental Model with Grid Adaptivity (OMEGA), and the Mesoscale Model Version 5 (MM5).

Transport and diffusion models were also used during the 1991 Persian Gulf War plume simulation efforts. These models estimate both the path of a plume and the degree of potential hazard posed by the chemical warfare agents. Dispersion models used during the Gulf War included the Hazard Prediction and Assessment Capability (HPAC) along with its component, the Second-order Closure Integrated Puff (SCIPUFF) model; the Vapor, Liquid, and Solid Tracking (VLSTRACK) model; the Non- Uniform Simple Surface Evaporation (NUSSE) model; and the Atmospheric Dispersion by Particle-in-Cell (ADPIC) model.

DOD’s Conclusion about U.S. Troops’ Exposure to Chemical Warfare Agents Cannot Be Adequately Supported

DOD’s conclusion about the extent of U.S. troops’ exposure to chemical warfare agents during and immediately after the Gulf War, based upon DOD and CIA plume model estimates, cannot be adequately supported. This is because of uncertainty associated with the source term data and meteorological data. Further, the models themselves are neither sufficiently certain nor precise to draw reasonable conclusions about the size or path (that is, the direction) of the plumes.

In particular, we found five reasons to question DOD’s conclusion. First, the models DOD and the CIA selected were in house models not fully developed for analyzing long-range dispersion of chemical warfare agents as environmental hazards. DOD and CIA officials selected several in-house models to run plume simulations. For Khamisiyah and the other Iraqi sites selected for examination, DOD selected the COAMPS and OMEGA meteorological models and the HPAC/SCIPUFF and VLSTRACK dispersion models. However, these models were not at the time fully developed for modeling long-range environmental hazards.

Second, the assumptions about the source term data used in the models are inaccurate. The source term data DOD used in the modeling for sites at Khamisiyah, as well as Al Muthanna and Muhammadiyat, contain significant unreliable assumptions. DOD and the CIA based assumptions on field testing, intelligence information, imagery, UNSCOM inspections, and Iraqi declarations to UNSCOM. However, these assumptions were based on limited, nonvalidated, and unconfirmed data concerning (1) the nature of the Khamisiyah pit demolition, (2) meteorology, (3) agent purity, (4) amount of agent released, and (5) other chemical warfare agent data. In addition, DOD and the CIA excluded from their modeling efforts many other sites and potential hazards associated with the destruction of binary chemical weapons, vast stores of chemical warfare agent precursor materials, and the potential release of toxic byproducts and chemical warfare agents from other sites.

Third, in most of the modeling performed, the plume heights were significantly underestimated. Actual plume height would have been significantly higher than the height DOD estimated in its modeling of demolition operations and bombings. The plume height estimates that the CIA provided for demolition operations at the Khamisiyah pit were 0 to 100 meters. However, neither DOD nor the CIA conducted testing to support estimated plume height associated with the bombings of Al Muthanna, Muhammadiyat, or Ukhaydir. According to DOD modelers, neither plume height nor any other heat or blast effects associated with these bombings were calculated from the models; instead, these data were taken from DOD’s Office of the Special Assistant for Gulf War Illnesses. In addition, according to a principal Defense Threat Reduction Agency modeler, DOD’s data on plume height were inconsistent with other test data for the types of facilities bombed.

Fourth, postwar field testing at the U.S. Army Dugway Proving Ground, in Utah, to estimate the source term data did not realistically simulate the actual conditions of the demolition operations at Khamisiyah or the effects of the bombings at any of the other sites in Iraq. For field testing to be effective, conditions have to be as close to the actual event as possible, but these tests did not provide more definitive data for DOD and CIA’s models. The tests did not realistically simulate the conditions of the demolition of 122 mm chemical-filled rockets in Khamisiyah. The simulations took place under conditions that were not comparable with those at Khamisiyah. There were differences in meteorological and soil conditions; the construction material of munitions crates; rocket construction (including the use of concrete-filled pipes as rocket replacements to provide inert filler to simulate larger stacks); and the number of rockets, with far fewer rockets and, therefore, less explosive materials. In addition, in the tests, the agent stimulant used had physical properties different from those of the actual agent.

Finally, there are wide divergences—with regard to the size and path of the plume and the extent to which troops were exposed—among the individual models DOD selected. The models DOD used to predict the fallout from Khamisiyah and the other sites showed great divergence, even with the same source term data. While the models’ divergences included plume size and paths, DOD made no effort to reconcile them. The IDA expert panel observed that the results were so divergent that it would not be possible to choose the most exposed areas or which U.S. troops might potentially have been exposed. IDA therefore recommended a composite model, which DOD adopted. However, this approach only masked differences in individual model projections with respect to divergences in plume size and path. In addition, DOD chose not to include in the composite model the results of the LLNL simulation, performed at the IDA expert panel’s request. The LLNL simulation estimated a larger plume size and different path from DOD’s models. The IDA panel regarded the LLNL model as less capable than other models because it modeled atmospheric phenomena with less fidelity. A modeling simulation done by the Air Force Technical Applications Center (AFTAC) also showed significant divergences from DOD’s composite model.

MOD Relied on U.S. Plume Modeling to Determine Their Troops’ Exposure to Chemical Warfare Agents

According to British officials, the MOD did not collect any source term or meteorological data during the 1991 Persian Gulf War. It also did not independently model the plume from Khamisiyah, relying instead on the 1997 DOD and CIA modeling of Khamisiyah. However, according to British MOD officials, they were reassessing the extent of British troops’ exposure, based on DOD’s revised 2000 remodeling of Khamisiyah. We requested from the British MOD, but did not receive, information on the findings from this reassessment.

The MOD also determined that a number of British troops were within the boundary of the plume in the DOD and CIA composite model. The MOD estimated that the total number of British troops potentially exposed was about 9,000 and the total number of troops as “definitely” within the path of the plume, however, was about 3,800. In addition, of 53,500 British troops deployed, at least 44,000 were estimated as “definitely not” within the path of the plume. However, since the MOD relied exclusively on DOD’s modeling and since we found that DOD could not know who was and who was not exposed, the MOD cannot know the extent of British troops’ exposure.

Total U.S. Plume- Modeling Costs

The DOD and CIA were the primary agencies involved in the modeling and analysis of U.S. troops’ exposure from the demolition at Khamisiyah and bombing of chemical facilities at Al Muthanna, Muhammadiyat, and Ukhaydir, but several other agencies and contractors also participated. Funding to support the modeling efforts was provided to various DOD agencies and organizations, the military services, and non-DOD agencies and contractors. We collected data on the direct costs these agencies incurred or funds they spent. As shown in table 1, direct costs to the United States for modeling the Gulf War were about $13.7 million.

DOD’s and VA’s Epidemiology-Based Conclusions on Chemical Warfare Exposure and Rates for Hospitalization and Mortality Cannot Be Adequately Supported

DOD and VA each funded an epidemiological study on chemical warfare agent exposure—DOD’s on hospitalization rates and VA’s on mortality rates. From the hospitalization study, conducted by DOD researchers, and the mortality study, conducted by VA researchers, on exposed and nonexposed troops, DOD concluded that there was no significant difference in the rates of hospitalization and VA concluded no significant difference in the rates of mortality. These conclusions, however, cannot be supported by the available evidence. These studies contained two inherent weaknesses: (1) flawed criteria for classifying exposure, resulting in classification bias, and (2) an insensitive outcome measure, resulting in outcome bias. In addition, in several other published studies of 1991 Persian Gulf War veterans, suggest an associations between chemical warfare exposure and illnesses and symptoms have been established.

DOD and VA Used Flawed Criteria for Determining Troops’ Exposure

In the two epidemiological studies, DOD and VA researchers used DOD’s 1997 plume model for determining which troops were under the path of the plume—who were estimated to be exposed—and which troops were not—those who were estimated to be nonexposed. However, this classification is flawed, given the inappropriate criteria for inclusion and exclusion.

In the hospitalization study, the DOD researchers included 349,291 Army troops “coded” as being in the Army on February 21, 1991. However, the researchers did not report cutoff dates for inclusion in the study—that is, they did not indicate whether these troops were in the Persian Gulf between January 17, 1991, and March 13, 1991, the period during which the bombings and the Khamisiyah demolition took place. Although we requested this information, DOD researchers failed to provide it. Finally, the total number of 349,291 troops is misleading because many troops left the service soon after returning from the Persian Gulf and therefore would not have been hospitalized after the war in a military hospital—another criterion for inclusion in the study. Moreover, the researchers did not conduct any analyses to determine what number or percentage of those who left active duty were in the exposed or nonexposed group (including uncertain low-dose exposure or estimated subclinical exposure). Given all the methodological problems in this study, it is not possible to accurately estimate the total size or makeup of the exposed and nonexposed population that may have sought or may have been eligible for care leading to military hospitalization.

In the mortality study, the VA researchers included 621,902 Gulf War veterans who arrived in the Persian Gulf before March 1, 1991. Troops who left before January 17, 1991—the beginning of the bombing of Iraqi research, production, and storage facilities for chemical warfare agents— were included in the study. This group was not likely to have been exposed. Therefore, including them resulted in VA’s overestimation of the nonexposed group.

Troops who came after March 1, 1991—the period during which Khamisiyah demolition took place—were excluded from the VA study. The Defense Manpower Data Center (DMDC) identified 696,000 troops deployed to the Persian Gulf, but the mortality study included only the 621,902 troops deployed there before March 1, 1991. This decision excluded more than 74,000 troops, approximately 11 percent of the total deployed. In addition, 693 troops who were in the exposed group were excluded because identifying data, such as Social Security numbers, did not match the DMDC database. VA researchers did not conduct follow-up analysis to determine whether those who were excluded differed from those who were included in ways that would affect the classification.

DOD and VA Used an Insensitive Outcome Measure for Determining Hospitalization Rates

Hospitalization rates—the outcome measure used in the hospitalization study—were insensitive because they failed to capture the chronic illnesses that 1991 Persian Gulf War veterans commonly report, but that typically do not lead to hospitalization. Studies that rely on this type of outcome as an end point are predetermined to overlook any association between exposure and illness.

Based on DOD’s 1997 plume model, DOD’s hospitalization study compared the rates for 1991 Persian Gulf War veterans who were exposed with the rates for those who were nonexposed. This study included 349,291 active duty Army troops who were deployed to the Persian Gulf. However, DOD researchers did not determine the resulting bias in their analyses, because they did not account for those who left the service.

The Institute of Medicine noted that the hospitalization study was limited to Army troops remaining on active duty and to events occurring in military hospitals. Conceivably, those who suffered from Gulf War-related symptoms might leave active duty voluntarily or might take a medical discharge. Hospitalization for this group would be reflected in VA or private sector databases, but not in DOD databases. The health or other characteristics of active duty troops could differ from those of troops who left active duty and were treated in nonmilitary hospitals. Moreover, economic and other factors not related to health are likely to affect the use of nonmilitary hospitals and health care services.

This limiting of the study to troops remaining on active duty produced a type of selection bias known as the healthy warrior effect. It strongly biased the study toward finding no excess hospitalization among the active duty Army troops compared with those who left the service after the war.

Some Studies Suggest an Association between Chemical Warfare Exposure and Gulf War Illnesses

We found some studies that suggest an association between chemical warfare agent exposure and Gulf War illnesses. Each of these studies has both strengths and limitations. In one privately funded study of Gulf War veterans, Haley and colleagues reported an association between a syndromic case definition of Gulf War illnesses, based upon the ill veterans’ symptomatic complaints, with exposure to chemical warfare agents. Factor analysis of the data on symptoms was used to derive a case definition identifying six syndrome factors. Three syndrome factor variants found to be the most significant were (1) impaired cognition, (2) confusion-ataxia, and (3) arthro-myo-neuropathy.

Conclusions

In evaluating the plume models used, the results from the DOD and CIA modeling can never be definitive. Plume models can allow only estimates of what happens when chemical warfare agents are released in the environment. Such estimates are based on mathematical equations, which are used to predict an actual event—in this case, the direction and extent of the plume. However, in order to predict precisely what happens, one needs to have accurate data on relative to both source term and meteorological conditions. DOD had neither of these.

Given the unreliability of the input data, the lack of individual troop location information, and the widely divergent results of the simulations conducted based on varying models, DOD’s analyses cannot adequately estimate the extent of U.S. troops’ exposure to chemical warfare agents and other related releases. In particular, the models selected were not fully developed for projecting long-range environmental fallout, and the assumptions used to provide the source term data were inaccurate or flawed. Even when models with the same source term data were used, the results diverged. In addition, the models did not include many potential exposure events and exposures to some key materials—for example, binary chemical weapons, mustard agent combustion by-products, and chemical warfare agent precursor materials. It is likely that if models were more fully developed and more credible data for source term and meteorological conditions were included in them, particularly with respect to plume height as well as level and duration of exposure, the hazard area would be much larger and most likely would cover most of the areas where U.S. troops and Coalition forces were deployed. However, given the lack of verifiable data for analyses, it is unlikely that any further modeling efforts would be more accurate or helpful.

The results of DOD’s modeling efforts were, nonetheless, used in epidemiological studies to determine the troops’ chemical warfare agent exposure classification—i.e., exposed versus nonexposed. As we noted in 1997, to ascertain the causes of veterans’ illnesses, it is imperative that investigators have valid and reliable data on exposure, especially for low- level or intermittent exposures to chemical warfare agents. To the extent that veterans are misclassified as to exposure, relationships will be obscured and conclusions misleading. In addition, DOD combined the results of individual models that showed smaller plume size and ignored the results of the LLNL which showed much larger plume size and divergent plume path. Given the uncertainties in source term data and divergences in model results, DOD cannot determine or estimate—with any degree of certainty—the size and path of the plumes or who was or who was not exposed.

Recommendations for Executive Action

In our report, we are recommending that the Secretary of Defense and the Secretary of Veterans Affairs not use the plume-modeling data for future epidemiological studies of the 1991 Gulf War, since VA and DOD cannot know from the flawed plume modeling who was and who was not exposed.

We are also recommending that the Secretary of Defense require no further plume-modeling of Khamisiyah and the other sites bombed during the 1991 Persian Gulf War in order to determine troops’ exposure. Given the uncertainties in the source term and meteorological data, additional modeling of the various sites bombed would most likely result in additional cost, while still not providing DOD with any definitive data on estimating who was or was not exposed.

We obtained comments on a draft of this report from VA, DOD, and CIA. VA concurred with the recommendation that VA and DOD not use the plume-modeling data for future epidemiological studies, since VA and DOD cannot know from the flawed plume modeling who was and who was not exposed. DOD did not concur with the recommendation, indicating that to them it called for a blanket prohibition of plume modeling in the future, where the limitations of the 1991 Gulf War may not apply. The intent of our recommendation is only directed at epidemiological studies involving the DOD and CIA plume modeling data from the 1991 Gulf War and not a blanket prohibition of plume modeling in the future. We have clarified the recommendation along these lines. DOD concurred with our second recommendation, indicating that despite enhancements in the models, uncertainties will remain. CIA did not concur with our report, indicating that it could not complete its review in the time allotted.

If you or your staff have any questions about this testimony or would like additional information, please contact me at (202) 512-6412 or Sushil Sharma, Ph.D., Dr.PH., at (202) 512-3460. We can also be reached by e-mail at rhodesk@gao.gov and sharmas@gao.gov. Individuals who made key contributions to this testimony were Venkareddy Chennareddy, Susan Conlon, Neil Doherty, Jason Fong, Penny Pickett, Laurel Rabin, and Katherine Raheb. James J. Tuite III, a GAO consultant, provided technical expertise.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.