A debate over Iraqi death estimates

John Bohannon's article “Iraqi death estimates called too high; methods faulted” (News of the Week, 20 Oct., p. [396][1]) contains several errors that require comment. ![Figure][2] An Iraqi woman collapses after learning of the death of a relative in a bomb attack on a police car. CREDIT: SLAHALDEEN RASHEED/REUTERS Bohannon fails to appreciate that cluster sampling is a random sampling method. Sampling for our study was designed to give all households an equal chance of being included. In this multistage cluster sampling, random selections were made at several levels ending with the “start” house being randomly chosen. From there, the house with the nearest front door was sampled until 39 consecutive houses were selected. This usually involved a chain of houses extending into two or three adjacent streets. Using two teams of two persons each, 40 houses could be surveyed in one day. Of our 47 clusters, 13 or 28% were rural, approximating the UN estimates for the rural population of Iraq. Bohannon states that Gilbert Burnham did not know exactly how the Iraqi team conducted its survey. The text sent to Bohannon, which he fails to cite, said, “As far as selection of the start houses, in areas where there were residential streets that did not cross the main avenues in the area selected, these were included in the random street selection process, in an effort to reduce the selection bias that more busy streets would have.” In no place does our Lancet paper say that the survey team avoided small back alleys. The methods section of the paper was modified with the suggestions of peer reviewers and the editorial staff. At no time did Burnham describe it to Bohannon as “oversimplified.” Those who work in conflict situations know that checkpoints often scrutinize written materials carried by those stopped, and their purpose may be questioned. Unique identifiers, such as neighborhoods, streets, and houses, would pose a risk not only to those in survey locations, but also to the survey teams. Protection of human subjects is always paramount in field research. Not including unique identifiers was specified in the approval the study received from the Johns Hopkins Bloomberg School of Public Health Committee on Human Research. At no time did the teams “destroy” details, as Bohannon contends. Not recording unique identifiers does not compromise the validity of our results. Concerning mortality estimates, Michael Spagat may be content, as Bohannon claims, with mortality data collected barely 1 year into an escalating 3.5-year war. Others might not find these so helpful. # Response {#article-title-2} I do appreciate that cluster sampling relies on random samples. It is indeed the very bone of contention. “Sampling for our study was designed to give all households an equal chance of being included,” Burnham and Roberts write. But according to their methods as published in The Lancet , that is not the case. My article reports the concerns of Sean Gourley and Neil Johnson, who point out that the starting house was always on a street “randomly selected from a list of residential streets crossing the main street.” This excludes all the smaller streets—including back alleys—that do not cross a main street. Maps of Iraqi cities, freely available at , show that many residential areas would be excluded by this survey protocol. People living in those underrepresented households, Gourley and Johnson argue, are less likely to be exposed to the violence—car bombs, drive-by shootings, airstrikes—that accounts for most of the reported deaths. When I asked Burnham by e-mail about this possible source of bias, he replied that “in areas where there were residential streets that did not cross the main avenues in the area selected, these were included in the random street selection process, in an effort to reduce the selection bias that more busy streets would have.” When I asked him why the published methods leave out this wiggle room, he replied that “in trying to shorten the paper from its original very large size, this bit got chopped, unfortunately.” I used the term “oversimplified” to describe this discrepancy. I stated that “the details about neighborhoods surveyed were destroyed.” The details in question are the “scraps” of paper on which streets and addresses were written to “randomly” choose households, and as Burnham and Roberts explained to me, that record has indeed been destroyed. I appreciate the difficulty of conducting a study in a combat zone and also the researchers' desire to protect the survey team and respondents. At the same time, scientists concerned about the true number of Iraqi casualties want to know which method was used to select households and whether sample bias can explain the high number of violent deaths reported by Burnham et al . But without a clear and explicit methodology or raw data to independently examine, it is impossible to know. [1]: /lookup/doi/10.1126/science.314.5798.396 [2]: pending:yes