Microbicides, Meta-Analysis, and the N-9 Question (1998)

From the Editor The editorial by Roddy, Schulz, and Cates interjects some much needed caution concerning our accepting that topical microbicides reduce sexually transmitted disease (STD) transmission. A strong desire to have an effective female-initiated method of self-protection against STDs could lead to a wishful acceptance of results generated from less than optimal studies. Roddy and colleagues have raised a number of issues concerning study design. I would like to make a microbiological point. It is not clear to me that evaluations of STDs in women who have been using topical microbicides have considered the potential effect of those microbicides on the assays being used to diagnose the infections. It is incumbent on researchers to either show that the presence of residual microbicide does not interfere in an assay or to allow an adequate time period from last use of microbicide to specimen collection so that there is no effect. It would be sad if we concluded that microbicides prevent infection when the microbicides may actually interfere with our ability to detect the infections. Julius Schachter, PhD TOPICAL MICROBICIDES have become our scientific "hope de jour" because they would ideally provide a female-initiated way to protect women against sexually transmitted disease (STD). In this issue, Cook and Rosenberg present a review of the literature and a meta-analysis of the effect of the spermicide compound nonoxynol-9 (N-9) on male-to-female transmission of STD, with an emphasis on gonorrhea and chlamydial infection.1 They have arrived at a summary statistic that they interpret as providing evidence for "an appreciable" and "clearly protective effect" for women from these infections. We are more cautious about endorsing N-9 for STD protection based on our recent data and the current meta-analysis. First, let us examine the process of meta-analysis. Just as with any original scientific investigation, meta-analysis depends on the quality of its data, namely the studies included in the systematic review. Moreover, as with any aggregate collection of studies, the biases included in each individual investigation may be cumulative. Thus, the inferences drawn from a synthesis may not agree with a well done, large, randomized controlled trial (RCT).2 Our emphasis here is on the "well done" because poorly conducted randomized trials, similar to nonrandomized investigations, yield, on average, exaggerated estimates of effect.3 Thus, meta-analysts must adequately address the methodologic quality of the studies. Indeed, a meta-analysis (systematic review) does not necessarily have to produce a synthetic, combined estimate from all the RCTs identified. For one thing, not all RCTs are well done.4,5 As an example relevant to our current topic, the study by Cutler et al.6 used in the N-9 meta-analysis shows a strong, statistically significant effect and set the protective direction for the summary statistic the authors calculated. However, the study's6 supposed random allocation may have been inadequate (given the lack of reporting on the sequence generation and allocation concealment aspects of randomization).5,7 Moreover, in this study, loss to follow-up was extensive and, more important, differential (56% for control subjects and 76% for N-9 users). Also, the time interval for analysis wavered in the original report. Although initially designed for 1 year, the investigators did not think the products were used as instructed after 6 months, so they also evaluated the results at 6 months (lost to follow-up: 42% for control subjects and 73% for N-9 users). They also evaluated the protective effect to first gonorrhea infection (rate ratio [RR] and 95% confidence interval: 0.2, 0.0–1.1) and for all gonorrhea infections (0.1, 0.0–0.8) at 6 months, and to first infection (0.5, 0.2–1.4) and all infections (0.6, 0.3–1.3) at 1 year. Obviously, the choice of which of these four measures was used for the meta-analysis has an impact on the magnitude of the summary statistic. Other RCTs included in the systematic review were of relatively short duration and had no masking. The study by Rosenberg et al.8 was only for 6 weeks, the one by Niruthisard et al.9 only for 9 weeks, and neither study was masked. These time intervals seem inadequate to test products that women will be using for long periods of time. Because studies of lower quality have been found to exaggerate estimates of treatment effect,3,7 we are not surprised that the summary statistic calculated by Cook and Rosenberg showed "an appreciable" protective impact of N-9. How do we weigh the available evidence? Taking into consideration that we have access to a Family Health International unpublished study, the three better-quality RCTs have compared three different N-9 products-a gel,10,11 the sponge,12 and the film.13 Because the formulations and dosages of N-9 differ across the studies, we hesitate to combine them in a summary measure. However, taken together in a common graphic, these studies have shown N-9 reduces the risks of both gonorrhea and chlamydial infection, albeit at relatively low levels of protection (Figure 1). In Alabama,10,11 regular use of N-9 gel by women attending an STD clinic reduced cervical gonorrhea by 24%, cervical chlamydial infection by 22%, trichomoniasis by 17%, and bacterial vaginosis by 14%. In Kenya,12 where a nonstandard measure of effect (number of infections over the number of examinations) was used, the widest 95% confidence interval was obtained-in a protective direction for gonorrhea but in a harmful direction for human immunodeficiency virus (HIV). However, the use of nonstandard measures limits comparisons with the other studies. In Cameroon,13 no protection was found for any of the STD.Fig. 1: Nonoxynol-9 spermicide and sexually transmitted disease among women. Relative risk (log scale) and 95% confidence interval.Second, how do we factor in results from observational studies? The present meta-analysis article, which emphasized RCT, also provided observational studies as supporting evidence. We agree that well done observational studies can be useful, but they can also be misleading. An example of how an observational analysis might mislead is the recently completed N-9 film study in Cameroon.13 When this study is analyzed as an RCT, it shows no protective effect against HIV infection. Also, when we compare N-9 to placebo across different levels of compliance, no effect occurs at any level of use. However, when analyzed as a cohort study (consistent users compared with inconsistent users), N-9 has a protective effect (RR = 0.3). We think this divergence is the result of compliance bias, namely that consistent users of either N-9 or placebo practiced other behaviors that lowered their risks. Thus, we get the same effect with placebo (RR = 0.5) when it is analyzed in a similar cohort manner. Controlling for the many factors that differ between women who use a spermicide consistently and those who use it inconsistently is difficult, if not impossible. This compliance bias is also the reason RCTs should use data analytic methods that maintain the participants in their originally randomized groups, an intention-to-treat analysis. Studies of barrier methods of contraception or STD prophylaxis are particularly susceptible to compliance bias. This type of bias is most likely to occur in epidemiologic studies when the exposure (e.g., N-9) is measured throughout the study period and the amount of exposure (e.g., frequency of use) is under the control of the participants. Coitally dependent prevention methods fall into this category. In other observational studies where the exposure occurs only before or at enrollment (e.g., one-time radiation exposure) or the exposure is out of the participants' control, the problem is less likely to occur. Third, what level of protection will lead women actually to use microbicides? N-9 may provide modest protection from STD infections, but is the amount of protection provided going to convince women who want prophylaxis to use it? For illustrative purposes, we can compare the amount of protection topical microbicides provide against STD to the level of protection spermicidal contraceptives provide against unintended pregnancy. N-9 is about 75% effective under typical use conditions in preventing pregnancy; thus, 25 of 100 women will get pregnant over the course of a year.14 Likewise, based on our synthesis of current RCT data (Figure 1), N-9 is about 15% to 25% effective in preventing STD; thus, of 100 women who would get an STD, 75 to 85 women using N-9 will get infected. Although this degree of protection might have an important population-level impact on an STD epidemic, it seems less appealing at the individual level for women at risk of infection. So, where is the research to help settle the N-9 issue? Why are we limited to attempts at conducting a meta-analysis of a few studies, most of which have serious methodologic problems, of widely different N-9 products? The public needs to know whether they should put their trust in, spend their money for, and possibly risk their lives with the use of N-9. Policy makers need to know whether to recommend N-9 for use and whether it will help stem the STD epidemic. It is time we found better answers to the N-9 question. Does N-9 use provide a reasonable amount of protection from STD, and if so, which formulation works best? Finally, the protection provided by N-9 is likely to be modest, so we should be developing and testing new microbicides (concurrent with N-9 testing) so that advancement can be made as quickly as possible.