1

June 23, 2015

EPA-HSRB-15-02

Thomas A. Burke, Ph.D., MPH

EPA Science Advisor 

Office of the Science Advisor 

1200 Pennsylvania Avenue, NW 

Washington, DC 20460 

Subject: April 22-23, 2015 EPA Human Studies Review Board Meeting Report 

Dear Dr. Burke,

The United States Environmental Protection Agency (EPA or Agency) requested that the Human 

Studies Review Board (HSRB) provide scientific and ethics reviews of two items: The Completed Study 

and Monograph Report for Backpack and Handgun Application of Liquid Spray in Utility Rights 

of Way (Agricultural Handlers Exposure Task Force); and Protocol for Field Testing of Skin 

Applied Mosquito Repellent Products (S.C. Johnson).

The Board’s responses to the charge questions are presented in the enclosed final meeting report.

Signed,

Liza Dawson, PhD

Chair

EPA Human Studies Review Board
2

INTRODUCTION 

On April 22-23, 2015, the United States Environmental Protection Agency’s (EPA or Agency) Human 

Studies Review Board (HSRB or Board) met to address the scientific and ethical charge questions related 

to two items: The Completed Study and Monograph Report for Backpack and Handgun 

Application of Liquid Spray in Utility Rights of Way (Agricultural Handlers Exposure Task Force); 

and Protocol for Field Testing of Skin Applied Mosquito Repellent Products (S.C. Johnson).

REVIEW PROCESS 

The Board conducted a public meeting on April 22-23, 2015.Advance notice of the meeting was 

published in the Federal Register as “Human Studies Review Board; Notification of a Public Meeting” 

(Federal Register / Vol. 80, No. 43 / Thursday, March 5, 2015 / Notices). 

The Board heard presentations from EPA for each agenda item in sequence, consisting of the Agency’s 

review of scientific and ethical aspects of the two studies. This Final Report of the meeting describes the 

HSRB’s discussion, recommendations, rationale and consensus in response to each charge question for 

each of these items. 

For each agenda item, Agency staff first presented their review of the science and the Board asked the 

Agency presenters clarifying questions. The staff then described their review of the ethical aspects and the 

Board asked clarifying questions about those. The HSRB solicited public comments and next asked 

Agency staff to read the Charge Questions for the topic under consideration. The Board discussed the 

science questions first and then the ethics question. The Chair then called for a vote to confirm 

concurrence on a summary statement in response to each charge question.

For their evaluation and discussion, the Board considered materials presented at the meeting, oral 

comments, related materials and documents provided by the study sponsors, the Agency’s science and 

ethics reviews of the studies, oral responses from a study sponsor and protocol team, and public 

comments made at the meeting. A comprehensive list of background documents is available online at 

http://www.epa.gov/hsrb/. 
3

CHARGE TO THE BOARD AND BOARD RESPONSE

HSRB review of the Completed Study and Monograph Report for Backpack and Handgun 

Application of Liquid Spray in Utility Rights of Way (Agricultural Handlers Exposure Task Force)

Overview of the completed studies and monograph report

The completed study, conducted by the Agricultural Handlers Exposure Task Force (AHETF) was a 

monitoring study to assess dermal and inhalation exposure of workers using liquid pesticide sprays in 

utilities’ rights of way and similar settings such as parks and roadsides. The study consisted of two parts, 

one assessing exposure for workers using backpack sprayers and one for handgun sprayers. The workers 

did not mix the spray solutions, so the monitoring encompassed only the application of the sprays on 

actual work days. Workers in the two scenarios handled a range of active ingredient from 0.03 to 45.95

lbs, sprayed between 4.5 and 2900 gallons of solution. The period of work time ranged from 2 to 11.4 

hours, and the number of acres covered ranged from less than 1 acre to approximately 20 acres. 

Dermal exposure was monitored using hand washes, face and neck wipes, and whole body dosimeters 

that consisted of 100% cotton union suits worn under the clothing. Inhalation exposure was monitored 

using personal air sample pumps and OSHA Versatile Samplers mounted on the shirt collar.

Total dermal exposure was calculated by adding the results for the whole body dosimeters, hand washes 

and face and neck wipes. Inhalation exposure was calculated from air sample data, assuming a standard 

breathing rate for all workers to calculate total inhalation during the work period.

The studies were planned as a “7×3” design, meaning that a total of 7 sites were planned with 3 workers 

(“monitoring units” or MUs) assessed at each site, in accordance with the AHETF Governing Document 

and Backpack and Handgun ROW Scenario Construction Plan. The Handgun study included the full 21 

MUs, while the Backpack study included 19 MUs due to difficulties with recruitment and a protracted 

study timeline.

Science

Charge to the Board

1. Was the research reported in the Agricultural Handler Exposure Task Force (AHETF) completed 

monograph reports and associated field study report for AHE400 faithful to the design and objectives of 

the protocol, SOPs, and governing documents? 

2. Has the Agency adequately characterized, from a scientific perspective, the limitations on these data 

that should be considered when using the data in estimating exposure of those who apply liquid pesticide 

sprays to utilities rights-of-way using backpack or handgun spray equipment? 

Board Response, Charge question #1

The Board concluded that the research reported in the Agricultural Handler Exposure Task Force 

(AHETF) completed monograph reports and associated field study report for AHE400 were faithful to the 

design and objectives of the protocol, SOPs, and governing documents.

HSRB Detailed Recommendations and Rationale:

The research was conducted largely as planned, with some deviations due to difficulties in recruitment 

and due to laboratory changes. While there were a number of deviations from Good Laboratory Practice 

Standards (GLPS), the rationale in the AHETF documentation provides a needed level of assurance that 
4

procedures and data were well tracked. The protracted study schedule added unplanned but valuable 

temporal diversity by collecting samples over 3 years, and the steps taken to recruit and monitor backpack 

and handgun applicators across 9 and 10 different U.S. states, respectively, slightly expanded the spatial 

diversity. Randomization was used where possible, despite the constraints of recruiting within an overall 

purposive sampling scheme using strata in the amounts of active ingredient handled. For example, in 

both groups of applicators "When multiple workers in a given area/time were available, the monitored 

workers were selected at random." (p. 8 Crowley AHE1012 and p. 8 Crowley AHE1013). There were no 

repeat measurements on the same applicator. Only two pairs of applicators (one backpack and one 

handgun) worked for the same employer, and these workers were monitored in different calendar years, at 

different job sites, and were part of different application crews. Overall, the brands/types of backpacks 

and handgun sprayers, their configurations, and spray techniques were also varied (p. 9 Crowley 

AHE1012 and p. 8 Crowley AHE1013).

The Board agrees with the Agency's determination "... that the data set has captured routine behavior as 

well as limiting the likelihood of “low-end” exposures via certain scripting aspects (e.g., requiring a 

minimum monitoring time to avoid cases in which there is no detectable exposure); both attributes are 

valuable for regulatory assessment purposes. The potential for significant carryover of residual AI from 

prior days’ exposures in the chaps worn by many applicators adds realism, although it complicates 

explaining the high variability discussed in response to the second charge question. And the 

randomization in the selection of individual applicators likely mitigated selection bias on the part of 

participants or recruiters. Thus, with respect to costs, feasibility, and utility, the resulting dataset is 

considered a reasonable approximation of expected exposure for this population." (p. 22 Crowley 

AHE1012 and p. 21 Crowley AHE1013).

The final exposure data for the handgun applicators met both the primary objective of 3-fold accuracy for 

select statistics [fRA95] and the secondary objective of having sufficient statistical power to distinguish 

whether there is proportionality between total dermal exposure both before and after being adjusted for 

the recovery method efficiency [MEA] and the amount of active ingredient handled [AaiH]. The final 

exposure data for the backpack applicators met the secondary objective but not the primary objective. 

The explanation that not meeting the 3-fold statistical accuracy goal due was primarily due to higher than 

expected variability in exposures (a measured geometric standard deviation [GSD] of 5 to 6 versus the 

GSD of 4 planned within the protocol, see p. 14 Crowley AHE1012) was in fact anticipated by the Board 

(p. 23, HSRB Final Report for October 2010 meeting). 

The Board also agrees with the Agency that the additional recruitment of applicators (10 to 16 additional 

depending on the number of clusters) that might be needed to meet the 3-fold accuracy requirement is 

unlikely to change either the variability in exposures or the estimated exposure statistics themselves. 

Thus, as long as the Agency can accept the existence of the high variability within this exposure scenario, 

the Board also agrees "that it would not be worth the multi-year delay – nor ethically justifiable – to 

monitor these additional workers to meet the accuracy benchmark." (p. 15 Crowley AHE1012) 

Board Response, Charge Question #2

The Board identified three more limitations that should be considered when using this data:(1) A higher 

breathing rate to estimate the inhaled dose of applicators while on foot should be used;(2) the variability 

in the number of hand washes and the resulting time that deposits were allowed to reside on the skin 

before attempting removal adds a small, unrecognized weakness to the data; and (3) the inability to 

explain the high variability in general and the cause of the high-end exposures in particular is also an 

additional limitation of the study. Neither of the first two concerns is thought to have a major impact on 

the study’s conclusions.
5

Breathing Rate

The assumed breathing rate of 1.67 L/min representative of "light activity" (Table 2 p 3 of 22 of Crowley 

AHE400) seems too low to apply to applicators while on foot. All of the backpack applicators were on 

foot carrying a backpack with up to 3-4 gallons of water that itself would weigh in excess of 25-30 

pounds. The majority of handgun applicators were on foot and manipulating heavy hoses at least part of 

the time. However, even if the breathing rate while on foot were doubled, the mean inhalation UE values 

for backpack applicators would be only 0.2% that of the mean dermal UE values (0.07 mg/lb ai inhaled / 

31 mg/lb ai dermal); and for handgun applicators, that ratio would be 0.8% (0.014 / 1.8). Therefore, 

adjusting this breathing rate upward will not have a significant effect on the overall study results.

Variability in Number of Hand Washes

The variability in the number and timing of hand washes presents a weakness in the data that was not 

previously commented upon by the HSRB or others. Information about the number of wipes that were 

made to collect the face/neck sample from each applicator was only descriptive: "Generally, 1-2 face/neck 

wipe samples were collected for each worker then analyzed as a composite sample." (p 10 of 22 of 

Crowley AHE400) However, data on the number of hand washes were provided in spread sheets, as 

summarized below. As can be seen, the number of hand washes is even more variable than the number of 

face/neck wipes.

# of washes 1 2 3 4 5

Backpack applicators 4 11 3 0 1

Handgun applicators 5 11 4 1 0

Unfortunately, the efficiency at which pesticides can be recovered from the skin decreases the longer that 

they remain on the skin before removal is attempted. While the actual times between hand washes can 

probably be obtained from the descriptive comments within the AHETF report (p. 65-174 for backpacks 

and 254-369 for handguns), the average times between washes was calculated by dividing the “Dermal 

Monitoring Time” by the “# Hand Washes” as listed in the spreadsheet and was found to vary from 100 to 

358 minutes for backpack applicators and from 106 to 407 minutes for handgun applicators. 

The variability in skin residence time adds an unknown level of uncertainty to the measured hand 

exposures. The variability in residence times on the hands is more important in terms of overall 

contribution to exposure, than variations in residence times on the face because measured hand exposures 

for backpack applicators were on average 70 times higher than the measured face/neck exposures; those 

for handgun applicators were about nine times higher. However, the importance of these variations on the 

overall outcome of the study is probably not great because before a 2× method efficiency adjustment 

[MEA] was applied, the hands and head together averaged only 27% of the overall dermal dose (p. 15 

Crowley AHE400) and importantly, only ranged from 1.1% to 8.8% for the five backpack applicators 

with the highest overall doses (see Figure 1a herein); the corresponding data for handguns are 30% of the 

overall dermal dose and from 6.4% to 12.5% for the three handgun applicators with the highest overall 

doses (see Figures 2a). 

In the process of obtaining the above data, it was observed that the four highest hand exposure values (16 

to 21 mg) were all among backpack applicators, although only two of the four were among those 

applicators with the four highest overall UE values. A Student’s t-test of the highest four backpack 

applicator hand exposures (A1, A25, A23, and A16) against the other fifteen backpack applicators 

(averaging 2.1 μg with a high of 7 μg) yielded a p of 0.0002; a similar t-test assuming lognormal 

distribution yielded a p of 0.00005, suggesting that there may have been something significantly different 
6

about the glove protection experienced by these four applicators.1

One positive implication of these high 

values is that it is unlikely that there was saturation of the applicators’ hands as passive dosimeters at the 

thirty-six lower values, thus increasing the level of confidence in the majority of the data.

It is unknown whether the high hand exposures observed in four cases were the result of glove penetration 

(meaning through openings in the glove including both the cuff and holes), permeation (basically solidstate diffusion), or breakthrough (saturation of the glove matrix resulting in very high permeation). While 

poor glove handling while doffing and re-donning contaminated gloves can result in hand exposures, it is 

unlikely that much of the 16 to 21 mg of active ingredient was simply the result of mishandling. 

Information provided at the meeting suggests that there was a low probability of glove permeation or 

breakthrough time due to the selection of inappropriate types of gloves. If that is confirmed, then the 

explanation for these high hand exposures remains unknown. 

Analyses to Yield Evidence-based Conclusions

The Board concluded that the existence of high, unexplained variability, in particular the high-end 

exposures, was a limitation of the study. While the Board agreed with the spirit of the investigations into 

the categorical variables examined by the EPA on p. 16-20 of Crowley AHE1012 and p. 15-19 of 

Crowley AHE1013 and is not recommending further formal statistical analyses, the Board believes that 

different analyses might help illuminate at least some of the mechanisms underlying the high levels of 

dermal exposure. The use of data or other information collected in the field to explain both the pattern of 

exposures and their high variability would have several benefits. First, it could help demonstrate the 

robustness of the study as a whole and provide more confidence in the representativeness of the data. 

And second, it could provide the basis for evidence-based applicator training and education. The 100-to1 ratio of the three highest to the three lowest backpack applicator exposures should provide a strong 

incentive to manufacturers, employers, and employees to believe that well-founded changes in work 

practices have the potential to make large reductions in exposures. 

By visual inspection, the Agency demonstrated that neither the expanded "site of application" categories,2

the three "prevalence of overhead spraying" categories, nor the use of chaps3 had a significant effect on 

the overall MEA adjusted normalized total dermal exposure (MEA-nTDE in the spreadsheet) in either 

group of applicators. Similarly, no effect was observable either by categorizing backpack applicators by 

whether or not they conducted frill (or "hack-and-squirt") applications or by categorizing handgun 

applicators by whether or not they sprayed from the vehicle, while walking, or both. However, it is 

interesting (and counter intuitive if one anticipates that the use of chaps may reduce exposure) but not 

statistically significant to note that the average normalized lower leg whole body dosimeters [nWBD LL 

in μg/lb ai in the spreadsheet] of those seven backpack applicators who wore chaps is slightly more than 

six times higher than the average of the twelve backpack applicators who didn't. 

One different approach undertaken by a Board member in an attempt to gain some insight into exposure 

mechanisms was to examine the distribution of the dermal doses measured on these applicators. Data 

provided within the Excel spread sheets was used to construct the following four figures. Each figure 

 1 A similar two t-tests conducted to compare these four high hand exposures to all of the other thirty-six 

applicators (including the handgun applicators who averaged 2.4 with a high of 9 μg) yielded p values of 0.0002 

and 1×10-12, respectively, and seems to confirm the uniqueness of the four highest backpack hand exposures. 2 Student's t-test of the backpack applicator data from the three major intended categories of Distribution, 

Pipeline, and Transmission versus all the other categories (drainage ditch, park, and wildlife refuge) found a pvalue of about 0.1. 

3 Student's t-test of the backpack applicator data for those who used chaps versus those not using them found a pvalue of about 0.2.
7

shows how the measured dose normalized for the AaiH for each applicator was distributed among their 

lower body (comprising upper leg, lower leg, and sock dosimeters), their upper body (comprising upper 

arm, lower arm, front torso, and rear torso dosimeters), their MEA hand washes, and their MEA face and 

neck wipes. The sum of these four components comprises each applicator's unit exposure value [UE]. 

The applicators within each figure are displayed in rank-order by UE (aka MEA-nTDE within the spread 

sheets). Figure 1a displays the data for all backpack applicators; Figure 1b displays the same data but 

with the highest three doses removed and the exposure scale magnified. Figure 2a and 2b displays the 

same full and truncated data for the handgun applicators. 

Figure 1a. Distribution of dermal doses

to all backpack applicators.

Figure 2a. Distribution of dermal doses 

to all handgun applicators.

Figure 1b. Distribution of dermal doses 

to the lowest 16 backpack applicators.

Figure 2b. Distribution of dermal doses 

to the lowest 18 handgun applicators.

It is readily apparent from these figures that the inclusion of data from applicators with the highest doses 

(visible only in the "a" versions of these figures) produces a different pattern than that seen for most

applicators (more visible in the "b" versions). While the hands are a prominent point of exposure for the 

majority of both backpack and handgun applicators and the lower body exposures are often not visible in 

the “b” versions, the body (and often the lower body in particular) becomes the predominant point of 

exposure for those applicators with the largest doses visible in the “a” versions.
8

The divergent exposure patterns suggest that it is unlikely that any single variable will correlate strongly 

with the full array of exposures within either of these two groups of applicators. And while the challenge 

of explaining the cause of the above pattern may still elude resolution from this data, one approach would 

be to look for various multistep mechanisms of exposure. 

For instance, it can be hypothesized that treating high or/and thick foliage might be a prerequisite to high 

exposures, but only in cases in which the applicator frequently walks into or through the foliage that he 

had just sprayed. Another example might be that applicators who wear chaps might correctly anticipate 

encountering thick foliage, which, in conjunction with the worker practices of walking through the 

sprayed foliage, will cause higher lower body exposures than applicators who don’t wear them.4

A similar but as yet unenlightening variable is the spray pressure of handgun applicators. When MEAnTDE is plotted as a function of spray pressure (psi), the correlation is not good (p > 0.08); however, the 

exposures by the two applicators with pressures in the 400 psi range (with MEA-nTDEs of 12,123 and 

6,942 μg/lb) are clearly well above the bulk of handgun applicators (averaging 754 μg/lb). Again, this 

single variable fails as an explanation because the exposure of the one applicator with an 800 psi system 

is not at all high, and the applicator with the next (third) highest exposure was using a 45 psi system, one 

of the lowest pressure systems within the study. Handgun spray pressure may be a factor but is 

insufficient in itself to explain high exposures and additional factors such as occasional work practices 

must play a role.

No categories of foliage or/and work practices were found in the AHETF report, and it is not clear from a 

cursory review of the descriptive comments contained within it (p. 65-174 for backpacks and 254-369 for 

handguns) whether sufficient information has been recorded to create such categories post-hoc. While the 

Board made comments about the desirability of collecting this information in their 2010 review of the 

protocol, these recommendations were not included in the Board’s final report from 2010.

5

 4 The use of chaps added two complications. One (based on a response to a question from the Board) is that these 

chaps could have been used on previous days and if unwashed, would have had significant carryover from those 

previous uses; moreover, it seems likely (but undocumented) that those previous uses were with the same AI 

(possibly even the same tank mix) as that tested. Second, if the years of experience leads an applicator to 

anticipate more accurately that they will encounter high or/and thick foliage, then a third category of 

“experience” could be added to the prediction; however, the small number of samples within this study are even 

less expected to have the power to demonstrate the statistical significance of such higher level associations.

5 In October 2010, the HSRB reviewed the protocol for this research. Unfortunately, the amount of detailed 

comments included within the HSRB Final Report is minimal, viz., p. 14, "However, the Board noted a few 

weaknesses in the proposed study design. In particular, the variability in individual dermal and inhalation 

exposure levels may be extremely high because of the diversity of terrains and locations selected for the study 

and the opportunity for large (but potentially categorical) personal differences in application practices." While 

weaknesses were noted, no specific changes were recommended in the HSRB’s final report. 

The minutes of the meeting were a bit more specific, but they are not the primary record of the Board’s actions. 

For example, on p. 19 of the Minutes of the October 2010 Meeting, “... ensure that the Board has an opportunity 

to examine some of the pieces of information that Ms. Sherman discussed earlier: task duration records, work 

pattern that the participants follow (e.g., do they advance while they are spraying or do they go out to the end of 

the area and retreat; are they walking through treated foliage?); height and density of foliage; and hose length. 

It would be useful to have some kind of categorical analysis because it might turn out that there are differences 

between advancing and retreating application patterns.” And on p. 23 of the Minutes of the October 2010 

Meeting “... a tremendous amount of variability will result from this scenario because the terrain is so variable 

... and therefore good notes and categorizing some of the variables will be valuable.”
9

Statistical analyses

The primary benchmark objective for each scenario in the AHETF exposure studies is to report the 

geometric mean, arithmetic mean, and 95th percentile of the reference distribution of normalized 

exposure. Currently these three parameter estimates are computed from a two-parameter lognormal 

distribution assuming data is coming from a 1) simple random sample 2) correlated nested sample using 

variance component estimates. Based on these methods, the reported back transformed geometric means 

and variance adjusted arithmetic mean estimates are valid. However, the reported 95th percentile value 

estimated from a two-parameter lognormal distribution geometric mean and standard deviation estimates 

generally underestimates the 95th percentile upper limit. Currently advanced statistical methods are 

available in UNIVARIATE procedure in SAS to estimate the 95th percentile upper limit based on threeparameter lognormal distribution. 

For simple random sample exposure data, the Board recommend using the UNIVARIATE procedure 

three-parameter lognormal distribution method, estimate arithmetic mean, arithmetic standard deviation 

and 95th percentile upper limit after confirming the exposure data distribution is lognormal. 

For correlated nested sample data, estimate the log transformed mean and standard deviation from the 

mixed model variance component method and substitute the scale and sigma in the three-parameter 

lognormal distribution with these variance component values, then estimate the theta, threshold value 

from the UNIVARITE procedure. Using these adjustments, estimate and report arithmetic mean, 

arithmetic standard deviation and 95th percentile upper limit for correlated nested sample data using the 

UNIVARIATE procedure. Furthermore, the Board recommended using these revised estimate in 

computing bootstrap confidence intervals and power and sample size calculation related to AHETF 

exposure studies. 

Ethics

Charge to the Board

Does available information support a determination that the studies were conducted in substantial 

compliance with subparts K and L of 40 CFR Part 26? 

Board Response

The information provided on the Agricultural Handlers Exposure Task Force’s completed study 

“Backpack and Handgun Application of Liquid Spray in Utility Rights of Way (AHE400)” supports a 

determination that the studies were conducted in substantial compliance with subparts K and L of 40 CFR 

Part 26. 

HSRB Detailed Recommendations and Rationale

As detailed above, this study monitored the exposure of trained herbicide sprayers as they applied liquid 

herbicide sprays to unwanted plants in electric power line rights of way or similar outdoor areas in 

scenarios that used a backpack sprayer or a handgun sprayer.

The original protocol was reviewed and approved in May 2010 by Independent Investigation Review 

Board (IIRB) of Plantation, Florida; that in October 2012 IIRB became part of Schulman Associates IRB 

(SAIRB), an AAHRP-accredited IRB registered with Office of Human Research Protections (OHRP). 

The protocol and IRB documents were reviewed by the HSRB in October 2010. At that time, the Board 

concluded that, pending modifications recommended by EPA, minor revisions to the informed consent 

document, and translation of the informed consent materials into Spanish, the protocol was likely to meet 
10

the applicable requirements of 40 CFR 26, subparts K and L. OPP confirmed that these modifications 

were made prior to the initiation of the study, and documents provided by AHETF for the HSRB’s current 

review also reflect the requested modifications.

Recruitment began in February 2011. Beginning in May 2011, the study enrolled 40 participants. 

Researchers conducted 19 backpack sprayer monitoring units (MUs) and 21 handgun sprayer MUs in 12 

states. Although the protocol was written to recruit both men and women, no women were enrolled and 

only men participated in the study. Participants ranged from 19 to 68 years of age; no children or minors 

were enrolled. Documentation from AHETF indicates that all participants received approved information 

about the nature of the study, its risks and procedures to minimize risk, and that these disclosures were 

provided in either of English or Spanish, depending on participant’s choice, both orally and in writing. 

All participants signed approved informed consent documents; one Spanish-speaking participant with 

self-reported low literacy skills had the document read aloud to him. Participants received a total 

compensation of $100 for their time and participation in the study. The study closed in September 2013.

Over the period of the study there were eight approved protocol amendments, which added study sites, 

added certain items from participants’ own personal protective equipment to the list acceptable/required 

PPE; expanded the list of prescribed modified analytic methods; and expanded the list of approved 

analytic laboratories and laboratory personnel. These amendments had no discernible effect on the 

study’s ethical characteristics.

Over the period of the study there were also five reported protocol deviations. One was administrative; 

others involved incomplete collection of samples for analysis; use of alternative analytic methods not 

previously listed in the protocol; monitoring periods that were shorter than times prescribed in the 

protocol, including episodes in which stopping rules were invoked due to the heat index; failure of 

monitoring equipment; and individual participants’ actions that went against protocol and that were not 

recognized by study personnel in time to be prevented. Of these protocol deviations, the only event that 

appears to have any potential effect on the study’s ethical protections was the potentially increased risk of 

exposure incurred by a participant who smoked a cigarette during the monitoring period without first 

having his hands cleaned by study personnel.

Except as noted below, the board agreed with the conclusions and observations EPA’s Ethics Review 

(Sherman 2015), and found that the study substantially meets the provisions of subparts K and L of 40 

CFR Part 26. Specifically aspects are discussed below.

Acceptable risk-benefit ratio

• Identified risks were appropriately minimized, particularly the risk of heat-related stress and 

illness. There were no adverse effects reported and no withdrawals from the study. Stopping 

rules for heat stress were invoked on five occasions. Following the stopping rules should not 

have been considered “protocol deviations” because the rules were built into the protocol as for 

the protection of participants. No other incidents were reported during monitoring. 

• The degree of potentially increased risk harm incurred by one participant who smoked a 

cigarette without having his hands cleaned by study personnel is unknown. The potentially 

increased risk of harm that accompanied periods of non-monitoring (from the intentional actions 

of participants and the inadvertent failure of a personal air pump) is unknown. 
11

Voluntary and informed consent of all participants

• All participants were provided information on the study and signed a consent document in their 

choice of English or Spanish. One Spanish-only speaking participant with self-reported low 

literacy skills had the Spanish-language consent form read aloud to him by a bilingual member of 

the study staff. 

• There no were withdrawals from the study.

• Participants received compensation of $100 ($20 for taking part in the recruitment interview and 

$80 for participating in monitored spraying). This amount was not so high as to create an undue 

inducement to participate, and was sufficient to justly compensate participants for their time and 

inconvenience. 

Equitable selection of study participants

• The study was designed to recruit an appropriately diverse population representing the trained 

herbicide/pesticide sprayers in each area and to minimize selection bias. A 3-phase recruitment 

process was used to identify qualifying employment sites and to minimize the risk that employers 

might coerce their workers either to participate or not participate.

• EPA notes that “statistical inference requires assumption that the sample is representative of all 

U.S. backpack and handgun applicator scenarios in applicable areas” (Crowley 2015- Slide 63, 

Conclusions). Although the study protocol was written to include both male and female 

participants, and had appropriate measures in place to exclude pregnant and lactating women, 

ultimately the enrolled participants were exclusively male, presumably due to the composition of 

the applicator workforce being almost exclusively male. The questions of whether and how this 

sample is representative of exposure for female workers remains unanswered and may need to be 

revisited in the future, especially if changing labor trends bring more women into this line of 

work. 

HSRB Review of the Protocol for Field Testing of Skin Applied Mosquito Repellent Products (S.C. 

Johnson)

Overview of the protocol

This protocol outlines a study to be conducted with human volunteers using EPA-registered personal 

mosquito repellent products produced by S.C. Johnson & Son, Inc. (“S.C. Johnson”) in field tests against 

populations of wild mosquitoes. The purpose of the study is to determine complete protection time (CPT) 

of each product for inclusion on EPA’s new Repellency Awareness Graphic, to be used on consumer 

product labels. The study will be conducted at two sites within the U.S., in Wisconsin and Florida, and 

possibly at additional locations outside the U.S. if needed. Up to 18 EPA-registered repellent products 

may be included in the tests.

Human volunteers will have products applied to either arms or legs by study staff, and a standard volume 

of product will be used, whether aerosol, spray, or lotion. For each product, there will be 10 human 

subjects at each site using the product for a total of 20 replicates per product. CPT will be calculated by 

measuring the time from product application to first confirmed landing of mosquitoes, and estimating 

median CPT for each product at each site, using a Kaplan-Meier estimator. The lowest median CPT from 

the test locations will be used for the repellency graphic.
12

Science

Charge to the Board 

Is the protocol “Field Testing of S.C. Johnson Personal Mosquito Repellent Products to Support their Use 

of the EPA Repellency Awareness Graphic” likely to generate scientifically reliable data, useful for 

estimating the complete protection time of various EPA-registered S.C. Johnson skin-applied mosquito 

repellents in the field against wild adult mosquito populations? 

Board Response

The Board concluded that the protocol titled “Field Testing of S.C. Johnson Personal Mosquito Repellent 

Products to Support their Use of the EPA Repellency Awareness Graphic”, if modified according to 

Agency and HSRB recommendations, is likely to generate scientifically reliable data, useful for 

estimating the complete protection time as defined in the protocol, of various EPA-registered S.C. 

Johnson skin-applied mosquito repellents in the field against wild adult mosquito populations.

HSRB Detailed Recommendations and Rationale

The objective of the S.C. Johnson protocol is clearly stated and follows EPA’s Product Performance Test 

Guidelines OPPTS 810.3700: Insect Repellents to be applied to Human Skin. The field tests are designed 

to provide data to support the EPA’s Repellency Awareness Graphic. The Board approved of the protocol 

team’s responses to earlier Board and Agency recommendations. The Board recognizes that there is a 

long history of insect repellant efficacy studies and applauds the efforts to bring this information and 

experience into a metric or label that better protects the population by clearly informing consumers about 

product efficacy. The Board concurs with the Agency recommendations except where noted below. We 

also identify several opportunities to improve the quality of complete protection data in support of the 

Agency graphic or to improve clarity of the protocol.

Product application rate

The protocol proposes using a standardized dose of 1 gram product per 600 cm2 treated skin for all 

products (lotions, aerosols and pump sprays). The main reason given for using a standardized dose is to 

reduce variability in the results that would come from applying different amounts of products. The 

Agency did not agree with this approach and recommended changing the protocol to include product 

specific doses based on previous dosimetry studies of consumer behavior. Public comment supported the 

use of the single standardized dose recommended by S.C.J. 

The Board understands that the data from this protocol will be used to calculate median CPT values 

across all subjects from each of two sites rounded down to the nearest integer and the lowest value will be 

used for the product graphic. (p. 11 of 39, Sweeney and Sherman). Given the stated use of the data, the 

Board agrees with the use of a standardized dose of 1 gram product per 600 cm2 treated skin. However, 

we strongly suggest changing the language from standardized dose to standardized application rate and 

reserve “dose” to describe how much active ingredient is applied. 

The protocol specifies that the actual amount of product applied will be recorded. The percent activity 

ingredient in each product is also available so the Agency can in fact assess the relationship between dose 

(active ingredient) and efficacy (or CPT) as a quality assurance check of the data. An assessment of the 

relationship between CPT and dose (mass active ingredient per treated area) can provide an indication of 

the quality of the data or point out data that might be suspect because it does not fall along the expected 

dose-response trend. The Board recommends that the data analysis include a dose-response comparison 
13

for all products where multiple concentrations are available (e.g., DEET content in the different products 

ranges from 5% to 98.25%) to help assess data quality. In addition, the Agency might consider 

normalizing the CPT results to better represent the expected application rates derived from earlier 

dosimetry studies when calculating the final graphic number.

Product application method

The protocol proposes to use a variety of application methods including using pipettes for transferring 

liquid contents from pump sprays and spatulas for lotions, while aerosol sprays are applied directly. In 

each case, the product is applied and spread on the subject skin by a staff member. The Board was 

concerned that these application methods were not representative of actual application methods but 

ultimately concluded that the need for consistency outweighs the need for the protocol to be 

representative of consumer behavior. However, the Board stresses the importance of accurately reporting 

the application rate (mass of product per area of skin) for each subject. 

A particular concern with the aerosol application method was that the iterative procedure leaves open the 

possibility to repeatedly apply more than the target 1 gram. Some means should be integrated within the 

protocol to limit or preclude the potential to bias the average dose upward. One (and perhaps not the best) 

way to preclude such a bias would be to place some upper bound on the highest level above 1 gram that 

would be allowed to proceed to the field testing phase. Such an upper bound (or similar restriction) could 

be applied to all application methods, but over-exposures seem most likely to occur with the aerosol 

application method. 

Use of sites outside the U.S.

The protocol specifies the use of two established and ecologically distinct field sites in the United States

for testing, and the Board agrees that this will provide sufficient representation for determining CPT, but 

the Board shares the concerns expressed by the Agency about using sites located outside of the U.S. The 

study sponsor has described that in addition to the two established sites in the U.S., there is at least one 

established site in Australia that could be used. The protocol needs to provide more information on what 

constitutes an “established site” either in the U.S. or another country (i.e., climate, mosquito species 

present, other hazards such as other mosquito borne diseases, presence of cell phone service, 

representativeness of local demographics) and more importantly should describe how the data collected at 

the alternate sites outside the U.S. will be related to the U.S. consumer demographics and the expected 

mosquito populations in the U.S.

Potential for cross contamination

There are a number of places in the protocol that provide opportunities for insecticide to be inadvertently 

either lost or gained from/to the treated area6 on subjects. Simple precautions can be taken to alleviate this 

issue, but the protocol should specify steps that will be taken to insure that the treated area on subjects is 

not impacted by activities that take place before or during the experiment (i.e., rubbing sleeve or pant leg 

across the treated area). 

Potential for “carryover”

No justification was provided for the adequacy of separating multiple participations by any test subject by 

a minimum of one day (Science Response #6 in S.C. Johnson letter of 17 April 2015). It is important to 

 6 Section 10.5.1 indicates that subjects might be transported by vehicle after being treated providing opportunity for 

treatment to be wiped off or transferred to another subject; Section 2.8.2.6 describes a screened enclosure that may 

or may not be treated; Section 12.1.1 suggests that subjects might be used on multiple occasions where carryover of 

insecticide might be an issue.
14

verify that no carryover effect is present on subjects used on multiple days. The protocol suggests that a 

day between treatments will be sufficient when the same subject is used a second time, but justification or 

references are needed to support this. If a subject is treated with 98.25% DEET, is there any residual 

effect after 24 hours that might affect a low dose treatment (application of 5.6% DEET wipe)? 

Landing pressure

The protocol includes untreated control subjects with each test to insure that there is sufficient landing 

pressure to provide valid results. However, the landing pressure is not measured in quantitative terms, 

only whether it is sufficient or not (five landings in five minutes). Discussions during the meeting seemed 

to imply that landing pressure will influence the measured CPT.7 If in fact the landing pressure can 

influence the resulting median CPT and products tested on different days are subjected to different 

landing pressures, then it would be important to collect quantitative information on landing pressure that 

could be used to correct, normalize, or at least interpret the resulting CPT values. The Board recommends 

that the Agency and S.C. Johnson consider how a quantitative estimate of landing pressure can be 

determined without increasing the likelihood of bites if landing pressure is excessive (e.g., recording the 

time of each landing, the time to reach 5 landings, or the total landings in 5 minutes) and how that 

information can be used to normalize or interpret CPTs measured under different landing pressure 

conditions. 

Delayed start

The Board recognizes the advantages of delaying the exposure to mosquitoes for subjects treated with 

products that are known from previous experience to last for a long time. However, the protocol needs to 

provide more information about the criteria used to determine how long to wait before starting the test 

cycles (5 minute exposure at 30 minute intervals). Regardless of how long the subject’s exposure is 

delayed, the protocol should require a minimum number of completed cycles to insure valid results. For 

example, following a delayed exposure, the subject should complete at least three exposure cycles before 

getting a confirmed landing.

Representativeness of the sample population

The Agency should resolve the perceived discrepancy between the “representativeness of the sample 

population” goals of the study. The study sponsor states that participants are representative of the 

population in the area where the field testing will be conducted, while the Agency seems to want the 

sample to represent the population of potential repellent users in the United States. (p. 3 and 23, K. 

Sweeny and K. Sherman, Science and Ethics Review, 31 Mar 2015). The Agency made a request of the 

applicant to provide additional details regarding the demographics and ethnicity of the population in the 

area. (p. 4 ibid) The response by S.C. Johnson was to agree to document that the demographics of the 

“pool of potential volunteers ... represents the demographics of the areas that testing will occur” and to 

add a criterion question that requires that each participant is an insect repellent user (Science Response #2 

in S.C. Johnson letter of 17 April 2015). Although it could turn out that the composite distribution of two 

 7

 An unrecognized risk within any protocol based on confirmed events is the impact of falsely confirmed events 

created when two otherwise random events occur within 30 minutes of each other. Each falsely confirmed event 

will produce an artificially short CPT for a given individual and contribute to a shorter median CPT than would have 

occurred without that event, but their major adverse impact is to decrease the reproducibility of median CPT values 

when measured across multiple tests. Statistical analyses indicate that CPT test results with a product that has a low 

protection factor within its CPT will be less reproducible than with a product that has a high protection factor. And 

a repellent’s effectiveness within its CPT can be expected to depend upon the active ingredient’s intrinsic 

effectiveness, the AI’s concentration, and the insect landing pressure measured by the untreated controls.
15

test areas may approximate the population of potential repellent users in the United States, the essential 

criteria are not stated within the protocol and the discrepancy appears unresolved.

Experimental design

The design as presented tests all ten subjects assigned to a product on a single day. The downside of this 

design is that it does not allow results to be easily generalized to a range of environmental conditions that 

may affect the attractiveness of a subject (e.g., sweating due to temperature and humidity levels). An 

alternative would be to test each product on several days; e.g., five subjects on each of two days or three 

subjects on each of two days and the remaining four subjects on a third day. Each day would form a block 

for the analysis of that product’s data. Such a design would allow testing of multiple products on a given 

day. If it were of interest to compare product formulations and/or application methods, the combined data 

could be analyzed as a block design with multiple replications of each product within each block (day). 

These alternative designs that utilize blocking (e.g., by individual test subject) could be considered to 

account for known sources of variation (e.g., individual effects). 

Randomization 

The randomization mechanism should be described in more detail and rationale should be given for any 

given choice of randomization within the protocol. For example, it is not clear whether/how crosssubstance relations are to be evaluated in the data analysis and why randomization among test substances 

is needed. An explanation of this would be helpful. In addition, when the conditions support use of arm

rather than leg for exposure, then it may be more important to consider handedness when selecting what 

arm to treat, rather than randomly assigning to left or right hand, so the subject can have their dominant 

hand to remove landing mosquitoes before they bite.

Sample size determination

A sample size calculation would be useful here to inform the power of testing and the width of confidence 

intervals. Power and sample size calculation can be implemented using existing SAS procedures. 

Information about appropriate sample size calculations is included in the EPA document “Product 

performance Test Guidelines OPPTS810.3700: Insect Repellents to be Applied to Human Skin.” 

Sources of variation

Multiple sources of variation including, for example, site selection, treatment dosage, application rate, 

mosquito type/age/condition, and landing pressure can impact the results. For the most part, however, 

they are not accounted for explicitly within the study, and when the source of variation is not controlled 

(e.g. as it is using a standard application rate) then the contribution to variance should be acknowledged 

or discussed. The protocol does not currently specify the conditions that might cause the CPT data from 

the two sites to differ; however, the researchers should consider collecting information to explain any 

large and potentially significant differences in the CPT values between otherwise matched studies 

conducted at two different sites.

Ethics

Charge to the Board

Is the research described in the protocol “Field Testing of S.C. Johnson Personal Mosquito Repellent 

Products to Support their Use of the EPA Repellency Awareness Graphic” likely to meet the applicable 

requirements of 40 CFR part 26, subparts K and L?
16

HSRB Recommendation

The study, when modified as recommended by the HSRB in this report, is likely to meet the applicable 

requirements of 40 CFR part 26, subparts K and L.8

HSRB Detailed Recommendations and Rationale

This is a protocol for third-party research involving intentional exposure of human subjects to a pesticide. 

As described above, the study involves use of consumer products for mosquito repellency for defined 

periods of time with exposure to wild populations of mosquitoes for the purpose of determining Complete 

Protection Time (CPT) as defined in EPA guidance. These data will be used for product labeling to better 

inform consumers about relative protection times of different products.

The following aspects of the protocol were deemed acceptable by the HSRB:

• The protocol adequately describes the consent process to be used for subjects enrolling in the 

study and a form has been conditionally approved by SAIRB.

• The research proposal describes risks and procedures to minimize risk to human subjects. Risk 

minimization procedures include, training sessions for study participants, field stopping rules to 

avoid excessive heat exposure when temperatures reach 105 degrees, providing hydration/shade

for study subjects, first-aid access, and 24 hour call number in the event of any health-related 

consequences of study participation. The study has been reviewed and conditionally approved by 

SAIRB, which will presumably provide oversight throughout the study. 

• 40 CFR 26 Subpart Q, at §26.1703, states that EPA must not rely on data from any research 

involving intentional exposure of any human subject who is a pregnant woman (and therefore her 

fetus), a nursing woman, or a child. This protocol appropriately requires that subjects be at least 

18 years old and excludes female subjects who are pregnant or lactating. 

Before the research is initiated, the documents should be revised to address the comments from the 

EPA and the following concerns of the Human Studies Review Board:

• The protocol should discuss whether the number of repeat tests per person should be limited. 

There should be some plan to follow-up with subjects after their participation in the study has 

ended to check for any delayed consequences of study participation. This could consist of a 

phone call to check on health status. 

• The protocol team should provide a process to contact subjects in the unlikely event that new 

information is developed or discovered as a result of the study, for example, if mosquitoes were 

discovered with vector-borne disease or if a contaminated product was identified. 

• The protocol team should provide a justification, in the protocol or IRB documents, for excluding 

non-English speakers, especially at the Florida site where Spanish-speakers are a significant 

proportion of the population; or if no justification for exclusion is provided, the protocol should 

be amended to include Spanish speakers. In order to assess the representativeness of the study 

population, the protocol team should provide to EPA general information on population 

demographics of subject pool at each site, and some information about recruitment strategies. 

This is particularly important if alternate sites located outside of the U.S. are used.

 8 One member of the board abstained from the recommendation due to concerns about the methodology used in 

dermal toxicity studies that are used to determine the margins of exposure (MOEs) for the active ingredients in 

repellent products.
17

The Board discussed two additional aspects of the study that could benefit from clarification. The Board 

expressed concern regarding the statement in the consent document that care and compensation for injury 

would be handled similar to a workers compensation claim. It was not clear if this was an appropriate 

mechanism for care and compensation in a research study. The consent form states “This study is 

considered confidential.” The Board questioned whether this is intended to refer to data collected on 

individual subjects or whether the details of the study should not be disclosed or discussed with others. 

Conclusion

This study is intended to be conducted in substantial compliance with the requirements of 40 CFR 26 

subparts K and L. The amended protocol, when approved by SAIRB, should met all applicable ethical 

standards for the protection of human subjects of research, and all requirements for documentation of 

ethical conduct of the research. If this study is determined to be scientifically valid and relevant, there is 

no regulatory barrier to EPA’s reliance on it in actions under FIFRA or §408 of FFDCA.
18

REFERENCES

Crowley, M. (2015) Science Review of Completed AHETF Worker Exposure Monitoring Study: 

Backpack and Handgun Application of Liquid Spray in Utilities Rights-of-Way (AHE400). 

Washington, DC: U.S. Environmental Protection Agency, Office of Pesticide Programs.

Sherman, K. (2015). Ethics Review of Completed AHETF Worker Exposure Monitoring Study: 

Backpack and Handgun Application of Liquid Spray in Utilities Rights-of-Way (AHE400). 

Washington, DC: U.S. Environmental Protection Agency, Office of Pesticide Programs.

Sweeney, K. and K. Sherman (2015) Science and Ethics Review of a Protocol for Field Testing of 

S.C. Johnson Skin-Applied Mosquito Repellent Products. Washington, DC: U.S. Environmental 

Protection Agency, Office of Pesticide Programs.

U.S. Environmental Protection Agency (EPA). (2010, July 1). 40 CFR part 26. Code of Federal 

Regulations. Washington, DC: U.S. Government Printing Office. Retrieved June 16th, 2016 from 

http://www.gpo.gov/fdsys/pkg/CFR-2010-title40-vol1/xml/CFR-2010-title40-vol1-part26.xml

U.S. Environmental Protection Agency (EPA). (2010, July 7). Product Performance Test Guidelines: 

OPPTS 810.3700: Insect Repellents to be Applied to Human Skin [EPA 712-C-10-001] from 

http://www.regulations.gov/#!documentDetail;D=EPA-HQ-OPPT-2009-0150-0011. Retrieved June 16th, 

2015.

U.S. Environmental Protection Agency (EPA) Human Studies Review Board (HSRB) Meeting Report, 

October 27-28, 2010. Retrieved from http://epa.gov/osainter/hsrb/files/meeting-materials/oct-27-28-

2010-meeting/final-oct2010-mtg-report.pdf on June 16th, 2015.

U.S. Environmental Protection Agency (EPA) Human Studies Review Board (HSRB) Meeting Minutes, 

October 27-28, 2010. Retrieved fromhttp://epa.gov/osainter/hsrb/files/meeting-materials/oct-27-28-2010-

meeting/oct2010-final-min.pdf on June 16th, 2015.
19

NOTICE

This report has been written as part of the activities of the EPA Human Studies Review Board, a Federal 

advisory committee providing advice, information and recommendations on issues related to scientific 

and ethical aspects of human subjects research. This report has not been reviewed for approval by the 

Agency and, hence, the contents of this report do not necessarily represent the view and policies of the 

Environmental Protection Agency, nor of other agencies in the Executive Branch of the Federal 

government, nor does the mention of trade names or commercial products constitute a recommendation 

for use. You may obtain further information about the EPA Human Studies Review Board from its 

website at http://www.epa.gov/osa/hsrb. You may also contact the HSRB Designated Federal Officer, via 

e-mail at ord-osa-hsrb@epa.gov 

In preparing this document, the Board carefully considered all information provided and presented by the 

Agency presenters, as well as information presented by public commenters. This document addresses the 

information provided and presented within the structure of the charge by the Agency.
20

U.S. ENVIRONMENTAL PROTECTION AGENCY

HUMAN STUDIES REVIEW BOARD

Chair 

Liza Dawson, Ph.D., Research Ethics Team Leader, Division of AIDS National Institutes of Health (NIH), National 

Institute of Allergy and Infectious Disease (NIAID), Bethesda, MD 

Vice Chair

Edward Gbur, Jr., Ph.D., Professor, Agricultural Statistics Laboratory, University of Arkansas,

Fayetteville, AR 

Members 

Gary L. Chadwick, Pharm.D., M.P.H., C.I.P., Senior Consultant, HRP Consulting Group, Inc. Fairport, NY

George Fernandez, Ph.D., Statistical Training Specialist, SAS Institute, Statistical Training and Technical Services, 

Sparks, NV

Kyle L. Galbraith, Ph.D., Manager, Human Subjects Protection, Carle Foundation Hospital,

Urbana, IL 

Jewell H. Halanych, M.D., Assistant Professor, Internal Medicine Residency Program, Montgomery Regional 

Campus, University of Alabama at Birmingham, Montgomery, AL

Elizabeth Heitman, Ph.D., Associate Professor of Medical Ethics, Center for Biomedical Bioethics and Society, 

Vanderbilt University Medical Center, Nashville, TN

John C. Kissel, Ph.D., Professor, Department of Environmental and Occupational Health, School of Public Health, 

University of Washington, Seattle, WA

Randy Maddalena, Ph.D., Physical Research Scientist, Indoor Environment, Lawrence Berkeley National 

Laboratory, Berkeley, CA

William Popendorf, Ph.D., M.P.H., Professor Emeritus, Department of Biology, Utah State University, Logan, UT 

Kenneth Ramos, M.D., Ph.D., PharmB, Associate Vice President, Precision Health Sciences, Professor of Medicine, 

Arizona Health Sciences Center, Tucson, AZ

Suzanne M. Rivera, Ph.D., M.S.W., Associate Vice President for Research, Case Western University, Cleveland 

Heights, OH

Jun Zhu, Ph.D., Professor of Statistics and of Entomology, Department of Statistics, University of Wisconsin –

Madison, Madison, WI

Human Studies Review Board Staff 

Jim Downing, Executive Director, Human Studies Review Board Staff, Office of the Science Advisor, United States 

Environmental Protection Agency, Washington, DC