1

PART B : S U R V E Y M E T H O D O L O G Y

1. SURVEY OBJECTIVES, KEY VARIABLES, AND OTHER PRELIMINARIES

1(a) Survey Objectives

As part of its consideration for establishing an efficacy mark for insect repellents, EPA

seeks to understand which efficacy mark approach and design would be most informative and helpful to consumers when purchasing insect repellents.

Specific objectives of the survey include: (1) I dentify the types of information that users of insect repellents want on the label of an insect repellent product and (2) T est four versions of an efficacy mark , to determine the most helpful and comprehensible design

.

1(b) Key Variables

The key information to be collected includes the following:

Primary reason(s) consume rs use or do not use insect repellents Frequency of using insect repellents Key information consumers are searching for on insect repellent labels Reactions to sample efficacy mark designs

1(c) Statistical Approach

A national sample of 3

, 000 consumers will be included in the research. This sample size will yield results that are statistically reliable within +/

- 1.8 percentage points at the 95 percent confidence level.

Support in conducting the survey will be provided by the following contractor:

S hugoll Research

7475 Wisconsin Avenue

Suite 200 Bethesda, MD 20814

Shugoll Research will be assisted by Opinionology (formerly Western Wats ) which maintains an Internet consumer panel of more than 1,000,000 individuals drawn from throughout the countr y.

Panelists for the Opinionology panel are recruited using mixed recruitment approaches consisting of multiple offline and online resources. This includes random digit dial telephone calls to residential households in the U.S., emails sent by Opinionology ’ s online partners/affiliates to their audiences, banner ad placement on portals/websites of2

partners/affi liates, search engine text ads, direct mail invitations , use of social networking

sites and referral programs. This ensures a diverse sam ple that includes hard

- to

- reach segments of the population. Participants recruited through all these means receive information about and are directed to online survey panels

.

Panelists recruited for the Internet panel provide some demographic and persona l information to create a profile for each panel member. In this way, samples can be selected to reflect certain characteristics so that specific populations of interest are efficiently targeted. Panelist profiles are updated on a regular basis.

P anel m embers are validated through a rigorous process using both proprietary and commercially available solutions. This process compares panelist demographics to multiple databases and data vendors specializing in consumer information to confirm key identity dat a, including name, address and date of birth. The identity of each new panelist is validated against a host of quality checks including government managed postal files, web address databases, Opinionology ’s own Bloodhound de

- duplication software, and TrueSample. There are additional validation checks at the project level through Imperium's Verity. Additional proprietary identity checks are run when panelists

redeem rewards.

Panelists are rewarded throu gh a number of options including cash, online retail gift codes, pre

- paid debit cards and charitable donations. Opinionology also offers a small guaranteed reward for survey participation, regardless of outcome, which decreases panelist attrition and incre ases participation by up to 25%. Opinionology optimizes panelist rewards in ways that maintain a consistent mix of panelist tenure and behavioral traits needed to provide consistent data.

Opinionology maintains their panel through close scrutiny of pane lists’ participation patterns, survey response patterns, reward redemption patterns, and feedback on Facebook, Twitter, and through their panelist service team. This allows a strong source of

data to understand and control the sampling frame. Opinionology maintain s a behavioral scoring algorithm to measure answer consistency and panelist attention over time. The behavioral score is affected by non

- differentiation, completion time, unit non

- response and nonsense

- response, inattention, and inconsistent answe rs from profiles to surveys. Panelists who, over time, do not maintain an acceptable behavioral score are culled from the panel.

In addition to panelist behavioral scoring, Opinionology work s with sample source

auditors at M arketing, Inc. who regularly measure important behavioral metrics across panels. These metrics include activity rates, panelist tenure, media consumption and buyer behavior, and undesirable survey taking behavior. Marketing, Inc.'s service provides important feedback to help evaluate the efficacy and quality of recruiting efforts

and to maintain a consistent distrib ution of behavioral attributes i n the panel over time.

1(d) Feasibility3

A consumer who has purchased insect repellent one or more times in the past 12 months

will be able to complete the questionnaire without any assistance. The survey instrument is designed for ease of completion by the respondents, and employs close

- ended questions and pre

- filled data where possible to limit the respondent burden in entering information. The use of an online survey, instead of a telephone survey, allows the respondent to complete the questionnaire at their convenience. Further, the online survey methodology allows for respondents to easily view the efficacy mark designs unde r consideration.

EPA has planned this data collection as part of its investigation of whether and what type

of eff icacy mark might be appropriate for insect repellent products. EPA has planned for

and allocated resources for the efficient and effective m anagement of the information to be collected, and has a contract mechanism in place to provide logistical support.

2. SURVEY DESIGN

2(a) Target Population and Coverage

The universe for this study encompasses members of the contractors’ Internet panel. The sample size of consumers for the survey (n=3

, 000) was determined based on several considerations, including funds available for this study, the number of efficacy design marks to be evaluated, and the number of potential sub

- segments to analyze, e.g.,

demo graphic characteristics (age, gender, and education) and behavioral factors (frequency of using insect repellent and amount of time spent outdoors).

2(b) Sample Design

This study plans to use a national convenience sample drawn from members of the Opinion ology Internet panel

. Opinionology will pull a proportional sample from the panel based on age, gender, ethnicity, income, and region of the country based on the most recent U.S. Census figures. The sample is not a probability

- based sample and cannot be considered representative of all U.S. households because not all U.S.

households have access to the internet

. In addition, the recruitment methods used by online panels, while broad and carefully managed, are not all probability

- based. The sample is likely to under

- represent minorities, lower income households and individuals who are 65 years of age or older since those populations are less likely to have access to

the Internet.

Opinionology has studied the relationship between samples from its Internet panel and samples collected using more traditional probability sampling techniques, such as telephone surveys. Although samples drawn from Internet panels are not probability samples, based on its studies, Opinionology has developed proprietary techniques to ensure that differences between the results from Internet panel samples and results from probability

- based samples of the nation are minimized. In particular, Opinionology uses a variety of met hods, including demographic weighting, propensity score weighting, and4

quota sampling to obtain national survey results from its Internet panel that are as accurate as possible.

EPA believes that the sample drawn from the Opinionology Internet panel will offer valuable insights for purposes of developing efficiency mark designs for insect repellent products. The survey will be carefully designed and screening criteria will be used to define the target population. With this study, t he EPA is not attemptin g to obtain results that are statistically representative of the entire U.S. population to accurately estimate population values

. T herefore, a strict probability

- based sample is not required

. EPA recognize s the limitations of a convenience sample drawn fr om the Internet panel

, and

, given the objectives of this study, this type of sample is an acceptable alternative to traditional probability

- based sampling for its purposes

.

2(c) Precision Requirements

2(c) (i) Precision Targets

Given the objectives of the survey, t he study is designed for completion of 3

, 000 surveys from the consumer Internet panel that have purchased insect repellent products in the past year. Although the Internet panel is not a nationally representative probability samp le, it is still desirable that the sample be as representative as possible of individuals at least 18 years of age in the United States. After considering the costs and benefits of various data collection methods, EPA staff concluded that the most efficie nt way to collect data

need ed to meet the research objectives within a feasible budget is to employ an Internet panel with nationwide coverage. Thus, the EPA has contracted with Shugoll Research to design an Internet survey that, while not technically representative of the nation as a whole, nonetheless reflects the views of a broad population.

2(c) (ii) Non

- S ampling Error

One potential source of non

- sampling error for this study is non

- response bias

– i.e., that

the non

- respondents may differ from respondents. There are several way s to minimize

non

- response bias including:

1. Designing the survey so that it will minimize the incidence of non

- response. Shugoll Research will work very closely with EPA staff to ensure the questionnaire instrument is easy to understand and complete in a t imely manner. EPA has targeted a total response time of 15 minutes per respondent.

2. Reaching out to non

- responders to encourage their participation. C onsumers who have not responded will receive two separate reminder emails encouraging them to complete t he survey.

3. Providing conta ct information so respondents may obtain help in the event of technical difficulties when attempting to complete the survey.5

Another potential problem with Internet panel research is selection bias. Different sub

segments of the population respond in different proportions. One method employed to address this occurrence is click balancing or differential sampling. Essentially, Opinionology factors in estimated response rates by sub

- segments of the population (e.g.,

18

- 24 year old males) and sends more invitations to individuals in th ese categor ies so that as the data is being collected, the percentage of each sub

- segment that r esponds is accurately represented in the collected sample.

2(d) Survey Instrument Design

The EPA questionnaire will include questions to understand how consumers shop for insect repellents, specifically what information they are seeking on product label s as well as their reactions to alternative efficacy mark designs.

3. PRE

- TESTS AND PILOT TESTS

The questionnaire will be pre

- tested before survey implementation using a “slow start” methodology which sends a small number of questionnaires (100) out initially in order to evaluate the questionnaire timing and completion. It also provides an indication if consumers are experiencing any difficulty completing any of the questions. Revisions will be made to the questionnaire as needed based on the result s of the “slow start” pre

test.

4. COLLECTION METHODS AND FOLLOW

- UP

4(a) Collection Methods

Once the questionnaire is finalized, the contractor will develop an online program that will allow respondents to view the questions and materials based on their responses to earlier questions. In this way, an Internet methodology can be an improvement over m ail questionnaires, where respondents can preview questions and materials that are best asked in a controlled sequence.

Data collection will be conducted by Shugoll Research assisted by Opinio no logy

. A total

of 3

, 000 individuals 18 years of age or older who have purchased insect repellent s one or more times in the past 12 months will complete the study ; t hat is, a total of 3,000 completed surveys will be collected.

In order to obtain a total of 3,000 completed surveys, it is expected that a national

conv enience sample of 30,000 Internet panel members will be sent an e

- mail message inviting them to participate in a survey about consumer product consumption. The invitations will be sent in stages. The first stage outgoing sample will be proportional

based on age, gender, ethnicity, income and region of the country based on U.S. Census figures. The second state sample mailing will be more targeted based on how interviewing is proceeding to this point specifically in relation to the proportions of key demog raphics (age, gender, ethnicity, income, region).6

Th e e

- mail invitation sent to the outgoing sample will contain a link that takes the respondents to a website where the online questionnaire resides when they click the link. Based on typical response rates achieved from Internet panels, w e expect a response rate of about 10 percent from the 30,000 panel members contacted, or a total of 3,000 completed surveys.

Respondents who visit the site will be screened at the beginning of the survey to ensure th ey qualify before they are allowed to continue (e.g., must have primary responsibility for making decisions on purchasing insect repellants or sharing equally in making the decisions, have purchased insect repellent one or more times in the past 12 months, and do not work for a company that manufactures, distributes or sells insect repellent products) . It will also be important to screen respondents on age (18 years of age and older) and gender (half male and half female) at the beginning of the survey.

Including these five questions at the beginning of the survey will ensure that respondents are qualified to complete the rest of the questionnaire , given that data on insect repellent purchase/usage behavior and employment for companies involved in making or selling insect repellents are not kept on file for panel members. If these qualification questions were placed at the end of the questionnaire and not at the beginning, the cost o f administering the survey would increase because it would not be possible to determine if respondents qualify for the survey until they have taken the time t o answer all survey questions. It is much more efficient and effective to qualify respondents bef ore they are allowed to answer any of the survey questions.

Each respondent will be assigned a numeric ID that prevents them from visiting the web

site and completing the questionnaires more than once. Each respondent also will have

an individualized password that prevents anyone other than the panel member from completing the survey.

Shugoll Research will provide data related to the total number of invitations sent to recruit Internet panel members, the number of those who responded to the e

- mails, t he number of those who qualified for the survey

, the number of qualified panel members who completed all questions and the number of within quota completes . The total response rate will also be calculated. This information will be provided by Shugoll

Res earch and Opinionology as the survey progresses to enable monitoring of the efficiency and effectiveness of the online panel and methodology. Shugoll Research will make the final data on completes and response rate available to EPA immediately after

the c lose of the survey as well as in the final report of the study results.

4(b) Survey Response and Follow

- U p

To encourage as high a response as possible, two reminder e

- mails will be sent to those who have not completed a survey . In addition, contact information will be provided where respondents may obtain help in the event of technical difficulties when attempting to complete the survey.7

5. ANALYZING AND REPORTING SURVEY RESULTS

5(a) Data Preparation

A project spreadsheet used to collect, store a n d organize the data will be reviewed, edited

and validated. All data wil l be double key

- entered to ensure accuracy, and several data processing steps will create files suitable for analysis and tabulations.

5(b) Analysis

The data will be analyzed using a statistical software package. Data analysis will include descriptive statistics (e.g., frequencies of survey variables) and relationship analysis (e.g.,

cross

- tabulations of demographic (age, gender, education) and behavioral (light versus heavy insect re pellent users) variables.

The contractor , Shugoll Research, will provide EPA with raw data, as well as tabulated data with simple significant test results.

5(c) Reporting Results

The contractor will prepare a detailed report of the study which summarizes and interprets the key findings, including conclusions and actionable recommendations.

Professional

- quality graphs will be developed to illustrate study findings. The report will include an executive summary as well as sections on the project objectives , research methodology and procedures, detailed findings and conclusions and recommendations that provide direction to EPA regarding insect repellent labeling.

EPA will use the information collected through the survey for analysis and decision

making reg arding the need, and potential benefits to consumers , for an efficacy mark.