1

PART B: SURVEY METHODOLOGY

1. SURVEY OBJECTIVES, KEY VARIABLES, AND OTHER PRELIMINARIES

1(a) Survey Objectives

As part of its consideration for establishing an efficacy mark for insect repellents, EPA 

seeks to understand which efficacy mark approach and design would be most informative 

and helpful to consumers when purchasing insect repellents.

Specific objectives of the survey include:

(1) Identify the types of information that users of insect repellents want on the label of an 

insect repellent product and 

(2) Test four versions of an efficacy mark, to determine the most helpful and 

comprehensible design.

1(b) Key Variables

The key information to be collected includes the following:

Primary reason(s) consumers use or do not use insect repellents

Frequency of using insect repellents

Key information consumers are searching for on insect repellent labels

Reactions to sample efficacy mark designs

1(c) Statistical Approach

A national sample of 3,000 consumers will be included in the research. This sample size 

will yield results that are statistically reliable within +/- 1.8 percentage points at the 95 

percent confidence level.

Support in conducting the survey will be provided by the following contractor:

Shugoll Research

7475 Wisconsin Avenue

Suite 200

Bethesda, MD 20814

Shugoll Research will be assisted by Opinionology (formerly Western Wats) which 

maintains an Internet consumer panel of more than 1,000,000 individuals drawn from 

throughout the country. 

Panelists for the Opinionology panel are recruited using mixed recruitment approaches 

consisting of multiple offline and online resources. This includes random digit dial 

telephone calls to residential households in the U.S., emails sent by Opinionology’s

online partners/affiliates to their audiences, banner ad placement on portals/websites of 
2

partners/affiliates, search engine text ads, direct mail invitations, use of social networking 

sites and referral programs. This ensures a diverse sample that includes hard-to-reach 

segments of the population. Participants recruited through all these means receive 

information about and are directed to online survey panels.

Panelists recruited for the Internet panel provide some demographic and personal 

information to create a profile for each panel member. In this way, samples can be 

selected to reflect certain characteristics so that specific populations of interest are 

efficiently targeted. Panelist profiles are updated on a regular basis.

Panel members are validated through a rigorous process using both proprietary and 

commercially available solutions. This process compares panelist demographics to 

multiple databases and data vendors specializing in consumer information to confirm key 

identity data, including name, address and date of birth. The identity of each new 

panelist is validated against a host of quality checks including government managed 

postal files, web address databases, Opinionology’s own Bloodhound de-duplication 

software, and TrueSample. There are additional validation checks at the project level 

through Imperium's Verity. Additional proprietary identity checks are run when panelists 

redeem rewards. 

Panelists are rewarded through a number of options including cash, online retail gift 

codes, pre-paid debit cards and charitable donations. Opinionology also offers a small 

guaranteed reward for survey participation, regardless of outcome, which decreases 

panelist attrition and increases participation by up to 25%. Opinionology optimizes 

panelist rewards in ways that maintain a consistent mix of panelist tenure and behavioral 

traits needed to provide consistent data. 

Opinionology maintains their panel through close scrutiny of panelists’ participation 

patterns, survey response patterns, reward redemption patterns, and feedback on 

Facebook, Twitter, and through their panelist service team. This allows a strong source of 

data to understand and control the sampling frame. Opinionology maintains a behavioral 

scoring algorithm to measure answer consistency and panelist attention over time. The 

behavioral score is affected by non-differentiation, completion time, unit non-response 

and nonsense-response, inattention, and inconsistent answers from profiles to surveys. 

Panelists who, over time, do not maintain an acceptable behavioral score are culled from 

the panel. 

In addition to panelist behavioral scoring, Opinionology works with sample source 

auditors at Marketing, Inc. who regularly measure important behavioral metrics across 

panels. These metrics include activity rates, panelist tenure, media consumption and 

buyer behavior, and undesirable survey taking behavior. Marketing, Inc.'s service 

provides important feedback to help evaluate the efficacy and quality of recruiting efforts 

and to maintain a consistent distribution of behavioral attributes in the panel over time. 

1(d) Feasibility
3

A consumer who has purchased insect repellent one or more times in the past 12 months 

will be able to complete the questionnaire without any assistance. The survey instrument 

is designed for ease of completion by the respondents, and employs close-ended 

questions and pre-filled data where possible to limit the respondent burden in entering 

information. The use of an online survey, instead of a telephone survey, allows the 

respondent to complete the questionnaire at their convenience. Further, the online survey 

methodology allows for respondents to easily view the efficacy mark designs under 

consideration.

EPA has planned this data collection as part of its investigation of whether and what type 

of efficacy mark might be appropriate for insect repellent products. EPA has planned for 

and allocated resources for the efficient and effective management of the information to 

be collected, and has a contract mechanism in place to provide logistical support.

2. SURVEY DESIGN

2(a) Target Population and Coverage

The universe for this study encompasses members of the contractors’ Internet panel. The 

sample size of consumers for the survey (n=3,000) was determined based on several 

considerations, including funds available for this study, the number of efficacy design 

marks to be evaluated, and the number of potential sub-segments to analyze, e.g., 

demographic characteristics (age, gender, and education) and behavioral factors 

(frequency of using insect repellent and amount of time spent outdoors).

2(b) Sample Design

This study plans to use a national convenience sample drawn from members of the 

Opinionology Internet panel. Opinionology will pull a proportional sample from the 

panel based on age, gender, ethnicity, income, and region of the country based on the 

most recent U.S. Census figures. The sample is not a probability-based sample and 

cannot be considered representative of all U.S. households because not all U.S. 

households have access to the internet. In addition, the recruitment methods used by 

online panels, while broad and carefully managed, are not all probability-based. The 

sample is likely to under-represent minorities, lower income households and individuals 

who are 65 years of age or older since those populations are less likely to have access to 

the Internet.

Opinionology has studied the relationship between samples from its Internet panel and 

samples collected using more traditional probability sampling techniques, such as 

telephone surveys. Although samples drawn from Internet panels are not probability 

samples, based on its studies, Opinionology has developed proprietary techniques to 

ensure that differences between the results from Internet panel samples and results from 

probability-based samples of the nation are minimized. In particular, Opinionology uses 

a variety of methods, including demographic weighting, propensity score weighting, and 
4

quota sampling to obtain national survey results from its Internet panel that are as 

accurate as possible.

EPA believes that the sample drawn from the Opinionology Internet panel will offer 

valuable insights for purposes of developing efficiency mark designs for insect repellent

products. The survey will be carefully designed and screening criteria will be used to 

define the target population. With this study, the EPA is not attempting to obtain results 

that are statistically representative of the entire U.S. population to accurately estimate 

population values. Therefore, a strict probability-based sample is not required. EPA

recognizes the limitations of a convenience sample drawn from the Internet panel, and,

given the objectives of this study, this type of sample is an acceptable alternative to 

traditional probability-based sampling for its purposes. 

2(c) Precision Requirements

2(c) (i) Precision Targets

Given the objectives of the survey, the study is designed for completion of 3,000 surveys 

from the consumer Internet panel that have purchased insect repellent products in the past 

year. Although the Internet panel is not a nationally representative probability sample, it 

is still desirable that the sample be as representative as possible of individuals at least 18 

years of age in the United States. After considering the costs and benefits of various data 

collection methods, EPA staff concluded that the most efficient way to collect data 

needed to meet the research objectives within a feasible budget is to employ an Internet 

panel with nationwide coverage. Thus, the EPA has contracted with Shugoll Research to 

design an Internet survey that, while not technically representative of the nation as a 

whole, nonetheless reflects the views of a broad population.

2(c) (ii) Non-Sampling Error

One potential source of non-sampling error for this study is non-response bias – i.e., that 

the non-respondents may differ from respondents. There are several ways to minimize 

non-response bias including:

1. Designing the survey so that it will minimize the incidence of non-response. 

Shugoll Research will work very closely with EPA staff to ensure the 

questionnaire instrument is easy to understand and complete in a timely manner. 

EPA has targeted a total response time of 15 minutes per respondent. 

2. Reaching out to non-responders to encourage their participation. Consumers who 

have not responded will receive two separate reminder emails encouraging them 

to complete the survey. 

3. Providing contact information so respondents may obtain help in the event of 

technical difficulties when attempting to complete the survey.
5

Another potential problem with Internet panel research is selection bias. Different subsegments of the population respond in different proportions. One method employed to 

address this occurrence is click balancing or differential sampling. Essentially,

Opinionology factors in estimated response rates by sub-segments of the population (e.g., 

18-24 year old males) and sends more invitations to individuals in these categories so that 

as the data is being collected, the percentage of each sub-segment that responds is 

accurately represented in the collected sample. 

2(d) Survey Instrument Design

The EPA questionnaire will include questions to understand how consumers shop for 

insect repellents, specifically what information they are seeking on product labels as well 

as their reactions to alternative efficacy mark designs.

3. PRE-TESTS AND PILOT TESTS

The questionnaire will be pre-tested before survey implementation using a “slow start” 

methodology which sends a small number of questionnaires (100) out initially in order to 

evaluate the questionnaire timing and completion. It also provides an indication if 

consumers are experiencing any difficulty completing any of the questions. Revisions 

will be made to the questionnaire as needed based on the results of the “slow start” pretest.

4. COLLECTION METHODS AND FOLLOW-UP

4(a) Collection Methods

Once the questionnaire is finalized, the contractor will develop an online program that 

will allow respondents to view the questions and materials based on their responses to 

earlier questions. In this way, an Internet methodology can be an improvement over mail 

questionnaires, where respondents can preview questions and materials that are best 

asked in a controlled sequence.

Data collection will be conducted by Shugoll Research assisted by Opinionology. A total 

of 3,000 individuals 18 years of age or older who have purchased insect repellents one or 

more times in the past 12 months will complete the study; that is, a total of 3,000 

completed surveys will be collected.

In order to obtain a total of 3,000 completed surveys, it is expected that a national 

convenience sample of 30,000 Internet panel members will be sent an e-mail message 

inviting them to participate in a survey about consumer product consumption. The 

invitations will be sent in stages. The first stage outgoing sample will be proportional 

based on age, gender, ethnicity, income and region of the country based on U.S. Census 

figures. The second state sample mailing will be more targeted based on how 

interviewing is proceeding to this point specifically in relation to the proportions of key 

demographics (age, gender, ethnicity, income, region).
6

The e-mail invitation sent to the outgoing sample will contain a link that takes the 

respondents to a website where the online questionnaire resides when they click the link. 

Based on typical response rates achieved from Internet panels, we expect a response rate 

of about 10 percent from the 30,000 panel members contacted, or a total of 3,000 

completed surveys. 

Respondents who visit the site will be screened at the beginning of the survey to ensure 

they qualify before they are allowed to continue (e.g., must have primary responsibility 

for making decisions on purchasing insect repellants or sharing equally in making the 

decisions, have purchased insect repellent one or more times in the past 12 months, and 

do not work for a company that manufactures, distributes or sells insect repellent 

products). It will also be important to screen respondents on age (18 years of age and 

older) and gender (half male and half female) at the beginning of the survey. 

Including these five questions at the beginning of the survey will ensure that respondents 

are qualified to complete the rest of the questionnaire, given that data on insect repellent

purchase/usage behavior and employment for companies involved in making or selling 

insect repellents are not kept on file for panel members. If these qualification questions 

were placed at the end of the questionnaire and not at the beginning, the cost of 

administering the survey would increase because it would not be possible to determine if 

respondents qualify for the survey until they have taken the time to answer all survey 

questions. It is much more efficient and effective to qualify respondents before they are 

allowed to answer any of the survey questions. 

Each respondent will be assigned a numeric ID that prevents them from visiting the web 

site and completing the questionnaires more than once. Each respondent also will have 

an individualized password that prevents anyone other than the panel member from 

completing the survey.

Shugoll Research will provide data related to the total number of invitations sent to 

recruit Internet panel members, the number of those who responded to the e-mails, the 

number of those who qualified for the survey, the number of qualified panel members 

who completed all questions and the number of within quota completes. The total 

response rate will also be calculated. This information will be provided by Shugoll 

Research and Opinionology as the survey progresses to enable monitoring of the 

efficiency and effectiveness of the online panel and methodology. Shugoll Research will 

make the final data on completes and response rate available to EPA immediately after 

the close of the survey as well as in the final report of the study results.

4(b) Survey Response and Follow-Up

To encourage as high a response as possible, two reminder e-mails will be sent to those 

who have not completed a survey. In addition, contact information will be provided 

where respondents may obtain help in the event of technical difficulties when attempting 

to complete the survey.
7

5. ANALYZING AND REPORTING SURVEY RESULTS

5(a) Data Preparation

A project spreadsheet used to collect, store and organize the data will be reviewed, edited 

and validated. All data will be double key-entered to ensure accuracy, and several data 

processing steps will create files suitable for analysis and tabulations.

5(b) Analysis

The data will be analyzed using a statistical software package. Data analysis will include 

descriptive statistics (e.g., frequencies of survey variables) and relationship analysis (e.g., 

cross-tabulations of demographic (age, gender, education) and behavioral (light versus 

heavy insect repellent users) variables.

The contractor, Shugoll Research, will provide EPA with raw data, as well as tabulated 

data with simple significant test results.

5(c) Reporting Results

The contractor will prepare a detailed report of the study which summarizes and 

interprets the key findings, including conclusions and actionable recommendations. 

Professional-quality graphs will be developed to illustrate study findings. The report will 

include an executive summary as well as sections on the project objectives, research 

methodology and procedures, detailed findings and conclusions and recommendations 

that provide direction to EPA regarding insect repellent labeling. 

EPA will use the information collected through the survey for analysis and decisionmaking regarding the need, and potential benefits to consumers, for an efficacy mark.