Modifying Young Adult Skin Cancer Risk and Protective Behaviors (UV4.me2): A Hybrid Type 2 Dissemination/Effectiveness Trial
Phase: NA
Conditions: Melanoma and Other Malignant Neoplasms of Skin
Sponsor: Rutgers, The State University of New Jersey

Protocol Title 
Young Adult Skin Cancer Risk and Protective Behaviors (UV4.me2) 
Principal Investigator 
Carolyn Heckman, PhD 
Co-Investigator(s): 
Melissa Auerbach, PhD, Fox Chase Cancer Center 
Elliot Coups, PhD, Rutgers Cancer Institute of New Jersey 
Karen Glanz, PhD, MPH, University of Pennsylvania 
Sharon Manne, PhD, Rutgers Cancer Institute of New Jersey 
Lee Ritterband, PhD, University of Virginia/BeHealth Solutions 
 
Collaborating Institutions: 
University of Virginia/BeHealth Solutions 
 
Statistician: 
Elizabeth Handorf, Ph.D 
 
Funding Source: 
NCI 
Initial Version Date: 07/20/2016 
 
Amendment # 
N/A 
 
Version date: 
07/20/2016 
 
 
 
 
 
  

Table of Contents 
Section  
1.0 
Introduction 
3 
2.0 
Objectives 
4 
2.1 
Primary Aims 4 
2.2 
Secondary Aims 
4 
2.3 
Hypotheses 
4 
3.0 
Background/Rationale 5 
4.0 
Study Design 
6 
4.1 
Recruitment 
6 
4.2 
Inclusion and Exclusion Criteria 6 
4.3 
Procedures 
7 
4.3.1 
Overview of Intervention Conditions 
7 
4.4 
Measures 
9 
4.4.1 
Measures for Primary Aims 
9 
4.4.2 
Measures for Secondary Aims 10 
4.5 
Compensation 10 
5.0 
Risks to Participants 
11 
6.0 
Potential Benefits to Participants 
11 
7.0 
Provisions to Maintain the Confidentiality of Data 
11 
8.0 
Costs to Participants 
11 
9.0 
Consent Process 
11 
10.0 
Off-Study Criteria 
12 
11.0 
Drugs and Devices 
12 
12.0 
Multi-Site Research Study 
12 
13.0 
Statisical Analysis 
12 
13.1 
Statistical Analyses for Primary Aims 
12 
13.2 
Statistical Analyses for Secondary Aims 13 
14.0 
Data Safety Monitoring Plan 
14 
15.0 
Adverse Events 14 
16.0 
Quality Assurance Procedures and Participant Confidentiality 
14 
17.0 
Participant Informed Consent 
15 
18.0 
References 
15 
 
 
 
 
 
 
 
 
1.0 
Introduction 
 

Skin cancer is the most common cancer and can be deadly, debilitating, damaging, and disfiguring, yet is 
highly preventable. In 2014, the US Surgeon General made a call to action about the “major public health 
problem” of skin cancer, noting potential contributions of behavioral science and education, and a need 
for investments in such efforts 1. Almost five million Americans are treated for skin cancer annually, and 
incidence is rising 1-5. If current trends continue, melanoma will be the only Healthy People 2020 cancer 
objective to not meet death reduction goals1. Additionally, non-melanoma skin cancer (NMSC) can be a 
chronic disease for some, requiring ongoing costly treatments and decreases of quality of life similar to 
some other cancers6. Risk factors for melanoma and NMSC include personal or family history of 
melanoma or NMSC, certain phenotypic (e.g., fair skin) and other physical characteristics (e.g., numerous 
moles) 7-19, as well as excessive ultraviolet (UV) radiation exposure 18,20-25. 
 
Most skin cancers are preventable with skin protection such as minimizing UV exposure and wearing 
protective clothing and sunscreen 26,27. US adolescents have the lowest skin protection rates of all age 
groups26 and also increase exposure to natural and artificial UV radiation as they progress into 
adulthood27. Though childhood is a particularly high-risk period for UV exposure and skin damage, 
research suggests that only 25% of lifetime UV is accumulated by age 18 28. Our work shows that skin 
cancer risk behaviors, including sunburns, indoor tanning, and lack of protection peak at age 25 29,30. 
Thus, young adulthood is an important window for skin cancer risk reduction interventions. However, 
young adults tend to be resistant to public health recommendations because, as a group, they perceive 
themselves as having more immediate priorities than disease prevention, that the consequences of their 
current health behaviors are in the distant future, and they also tend to be experimenters and risk-takers 
highly influenced by peers 31-33. 
 
RE-AIM Framework. One of the major problems with health behavior research is that many interventions 
that demonstrate initial effects are never tested in effectiveness or dissemination trials, nor are they 
distributed to populations who most need them, thus, having little impact on population health 34. To 
address these concerns, the RE-AIM Framework was developed by Glasgow and colleagues 35. “RE-
AIM” refers to Reach, Effectiveness, Adoption, Implementation, and Maintenance. Reach refers to the 
number, proportion, and representativeness of individuals willing to participate in a given initiative, 
indicating potential generalizabiity 36. Effectiveness refers to the impact of an intervention on important 
outcomes, including potential negative effects, quality of life, and economic outcomes. Adoption refers to 
the absolute number, proportion, and representativeness of settings and intervention agents who are 
willing to initiate a program (not a main focus of the proposed project given the self-administered and 
automated intervention). Implementation refers to the individual’s use of the intervention. Maintenance 
refers to the long-term effects of a program on outcomes. Although widely used since 1999, most reviews 
have found that the components of the RE-AIM framework are not assessed and/or reported 
comprehensively or well 37-41. Additionally, most dissemination and implementation (D&I) studies 
focus on dissemination of guidelines, policies, practices, or interventions to healthcare agencies or 
practitioners rather than D&I of online self-administered interventions directly to consumers. We selected 
the RE-AIM framework over many others because we believe it is the most appropriate to investigate the 
dissemination, implementation, and effectiveness of an innovative self-administered and automated online 
intervention. We could identify no relevant studies that addressed D&I issues related to self-administered 
materials online.  
 

The PI developed a web-based intervention (UV4.me) that was found to significantly decrease UV 
exposure and increase skin protection behaviors among young adults in a randomized controlled trial 
(RCT) of nearly 1000 participants recruited from a consumer research panel. The intervention (UV4.me) 
is individually-tailored, interactive, and multimedia in nature, and based on the Integrative Model of 
Behavioral Prediction. Similar to other online trials, 73% of eligible individuals completed the baseline 
questionnaire, 70% who were randomized to the intervention accessed it, and 68% accessed it and 
completed at least one module. However, we have an opportunity to increase the engagement, 
implementation, and ultimately, the impact of UV4.me. We will do this by adding several key interactive 
features/strategies suggested by participants, our data, and supported by the literature (i.e., by creating a 
mobile version, adding incentives embedded in the intervention, a behavior tracking and feedback feature, 
peer interaction component, and ongoing news updates). This Hybrid Type 2 dissemination-effectiveness 
project’s purpose is to implement the enhanced intervention (UV4.me2) with adults aged 18-25 years at 
moderate to high risk of developing skin cancer and evaluate the intervention’s effectiveness in a sample 
recruited online through national dissemination to the general population. This project will use the RE-
AIM framework to determine the reach, effectiveness, implementation, maintenance, and cost of 
UV4.me2. 
 
2.0 
Objectives 
 
The objective of this project is to investigate the reach, effectiveness, implementation, maintenance, and 
cost of the enhanced UV4.me2 intervention in a large national randomized controlled trial. The ultimate 
goal is to improve the skin cancer protection behaviors (and potentially decrease skin cancer incidence) 
among a national sample of young adults at moderate to high risk of developing skin cancer. 
 
2.1 
Primary Aims 
 
Primary Aim 1. To enhance and determine intervention reach (i.e., enrollment, representativeness).  
 
Primary Aim 2. To determine the effectiveness of the enhanced intervention. 
 
2.2 
Secondary Aims 
 
Secondary Aim 1. To determine maintenance of the UV4.m4 and UV4.me2 interventions through 
evaluation at 6 and 12-month follow-up. 
 
Secondary Aim 2. To determine intervention implementation by young adults.  
 

Secondary Aim 3. To determine the costs of the UV4.me and UV4.me2 interventions. 
 
2.3 
Hypotheses 
 
Hypothesis 1. We hypothesize that the study enrollment rate will be greater than in the original UV4.me 
efficacy study and that our sample will be representative of national demographics of young adults other 
than race/ethnicity (associated with skin cancer risk), indicating potential generalizability. 
 
Hypothesis 2. We hypothesize that the UV4.me2 will be the most effective intervention, followed by 
UV4.me. 
 
Hypothesis 3. We hypothesize that maintenance outcomes will be best in the UV4.me2 group, next best 
for UV4.me, and least for the e-pamphlet group. 
 
Hypothesis 4. We hypothesize that greater intervention utilization (e.g., logins, module completion) and 
satisfaction will be associated with better behavioral outcomes 
 
Hypothesis 5. We hypothesize that UV4.me2 and UV4.me will have higher incremental costs than the e-
pamphlet. However, because UV4.me2 and UV4.me are also expected to result in better behavioral 
outcomes, we will assess cost-effectiveness to address whether these interventions are worth the 
additional costs. 
 
3.0 
Background/Rationale 
 
Large skin cancer prevention campaigns have the potential to reduce incidence, mortality, and morbidity 
as well as be cost-effective. For example, a long-term comprehensive multi-modality skin cancer 
prevention campaign in Australia returned US$3.27 per dollar invested in terms of life-years saved and 
disability-adjusted life-years 42. A recent study reported that a comprehensive US skin cancer prevention 
program similar to Australia’s would be estimated to prevent 20% of US melanomas, or 21,000 
melanoma cases annually, with an average annual reduction of $250 million spent on new melanomas 43. 
The US spends several hundred thousand dollars more per year than Australia on skin cancer 44, and the 
annual cost of treating new melanomas in the US is projected to increase threefold from 2011 to 2030 43. 
US and Australian (one randomized controlled trial) studies have found that NMSC can also be reduced 
by 14-40% with regular sunscreen use 45-47. A few prior interventions targeted to US young adults have 
shown increased skin protection and/or decreased UV exposure in research settings; however, their reach 
and sustainability have been limited because most have been conducted in person and had brief follow-
ups 48-54. Innovative, age-appropriate interventions are important to reduce skin cancer risk among 
young adults.  

 
Ninety-three percent of US young adults use the web 55, and web interventions can be disseminated 
widely and be cost-effective to maintain 56,57. Often websites are designed to provide information rather 
than as interventions to improve health behaviors and/or disease risk. Despite this purpose, a review of 
existing melanoma websites found that the majority failed to include complete information on important 
topics such as risk factors, diagnosis, and prevention, with a significant minority containing inaccuracies 
58. However, web interventions designed to improve health behaviors (e.g., exercise and weight loss) 
have been found to produce medium effect sizes and consistently outperform similar non-web 
interventions 57.  
 
We developed a web intervention that included many of the components found in successful internet 
interventions. We found significantly improved skin cancer risk behaviors among young adults in an RCT 
that enrolled almost 1000 participants from a consumer research panel 59. To our knowledge, this is the 
only empirically-tested internet intervention focused on skin cancer risk behaviors targeting young adults. 
The intervention (UV4.me) is individually-tailored, interactive, and multimedia based on Ritterband and 
colleagues’ 60 model for online behavior change interventions and the Integrative Model of Behavioral 
Prediction 61, which includes background variables such as demographics; cognitive variables such as 
beliefs, attitudes, norms, and self-efficacy; intentions; and behavior. We also emphasized appearance 
concerns, which is a major factor associated with young adult tanning 62-64.  
 
In addition to successful outcomes, approximately two thirds of eligible participants enrolled in the study 
and completed the baseline survey, two thirds randomized to UV4.me accessed the intervention, and two 
thirds completed the 3-month follow-up. Yet, we still have an opportunity to increase UV4.me 
implementation and impact. We propose to do this by enhancing recruitment and enrollment strategies as 
well as incorporating additional interactive features into the intervention (i.e., creating a mobile version, 
embedding incentives into the intervention, adding a behavior tracking and feedback feature, peer 
interaction component, and ongoing news updates). Features/strategies were chosen based on participant 
feedback, our data (e.g., attempted use of mobile devices), and reviews/models of effective e-Health 
interventions including for tobacco cessation, physical activity, weight loss, and nutrition 60,65-68 and 
online implementation strategies 69-72. Several enhancements are expected to improve reach, 
implementation, and behavioral outcomes simultaneously. 
 
To our knowledge, only 15 papers on dissemination and/or implementation of skin cancer-related 
interventions exist, and most were authored by Dr. Glanz about her Pool Cool Program, which improved 
skin cancer risk and protection policies and behaviors among aquatic staff and children at Hawaii and 
Massachusetts pools 73,74. The D&I trial assessed effects of an enhanced vs. basic D&I strategy on 
adoption, implementation, and maintenance of protection promotion policies and practices by aquatic 
staff at >400 pools across the country based on Diffusion of Innovation theory 75-80. Program adoption 
was high (86.6%), the enhanced intervention produced greater adoption, implementation was similar 
across conditions, and core skin protection elements were maintained for two years. Dr. Glanz has 
published several papers describing the correlates of successful adoption and implementation 81-86. She 
has also published other D&I papers related to other health behaviors such as dietary choices and intake. 

 
4.0 
Study Design 
 
4.1 
Recruitment  
 
 
A small number of human subjects will participate in acceptability testing (20-25 participants) and 
usability testing (12-15 participants). After acceptability and usability testing, a national sample of 1500 
human subjects will be recruited to participate in the Randomized Control Trial (RCT). Therefore, we 
have a lower accrual goal of 1,532 and an upper accrual goal of 1,540. 
 
Five types of online sources will be used to reach young adults: 1) national non-profit and for-profit 
skincare organizations (e.g., the Melanoma Research Foundation and Blue Lizard Sunscreen), 2) Google 
Adwords, 3) paid Facebook ads, 4) a consumer research panel, and 5) word of mouth (e.g., unpaid study 
Facebook page, earned social and traditional media [free publicity from a third party]). Google Adwords 
and Facebook are common online recruitment sources for young adults that can produce cost-effective, 
representative, and quality samples/data 87-95. Facebook is the largest social networking website and the 
second most popular website in the US after Google 96. Because some participants may know one 
another (e.g., from Facebook), we will inquire about/control for contamination between intervention 
conditions. Consumer research panels possess the qualities of Google Adwords and Facebook and can 
also produce guaranteed enrollment/data collection within a specified time-frame and budget 97-103, as 
occurred in the original UV4.me study. The team will refine web banner ads and create a Facebook study 
webpage to recruit individuals from the five online sources. Enrolled participants will be informed about 
the Facebook page, and non-enrolled individuals will be able to find the publicly available page 
organically by being referred by friends or searching Facebook or the web for related topics. To facilitate 
recruitment via Google and Facebook (also Bing and Yahoo), we will work with young adults and 
www.PatientRecruitmentOnline.com to help develop our keywords, ads, and recruitment plans. 
PatientRecruitmentOnline.com is made up of experts in online marketing specifically for health-related 
study recruitment. They will assist us in refining our keywords, ads, recruitment campaign, homepage, 
and enrollment process. 
 
4.2 
Inclusion and Exclusion Criteria 
 
 
We will recruit adults aged 18-25 years. Men and women and individuals from all races/ethnicities will be 
eligible for participation. However, only participants at moderate to high risk of developing skin cancer 
will be enrolled. Participants will be screened with the Brief Skin Cancer Risk Assessment Tool 104. Risk 
items include sun sensitivity, sunburn history, number of moles/freckles, and latitude of childhood 
residence. Items are weighted, resulting in a 0-78 score. A cut-off of >27 denotes moderate to high skin 

cancer risk.  Individuals with a history of skin cancer will be excluded. Based on our original UV4.me 
data, we estimate that approximately 50% of individuals who complete a screening form will be eligible 
by these criteria. 
 
Adult participants who are not eligible but are willing to provide informed consent will still be given 
access to UV4.me2 in order to promote goodwill with organizations and individuals and to gather 
preliminary data regarding additional dissemination and effects of UV4.me2 for older individuals and 
those at population-level risk for skin cancer. 
 
4.3 
 Procedures 
 
Acceptability Testing. Once the enhanced intervention is fully developed, acceptability testing modeled 
after an NCI online design project will be conducted 105 as was done for the original UV4.me 106. We 
will focus on the new features added to UV4.me for UV4.me2 (e.g., the mobile version), a prototype 
study Facebook page, and the recruitment ads and strategies. Testing will involve consulting users about 
attractiveness, comprehension, population acceptability, and persuasion of materials. A series of questions 
and structured guides will be prepared in advance relating to the elements to be evaluated is in the original 
UV4.me. Acceptability testing will be conducted using focused interviews with approximately 20 
participants eligible for and recruited using methods similar to the RCT. The first half of sessions will be 
conducted in person, and the second half will be conducted remotely via the web and telephone. 
Participants will use a device (computer, tablet, or smartphone) with access to the online intervention. The 
iterative process will involve discussion of an intervention element, followed by interaction with that part 
of the intervention, followed by discussion reacting the part, etc. The sessions will be video- or audio-
recorded and transcribed, and the moderator will take notes. 
 
Usability Testing. In usability testing, users do typical tasks with a product, or explore it freely, while 
their behaviors are observed and recorded to identify design flaws that cause user errors/difficulties 107. 
As in the original UV4.me 106, we will follow the NCI usability guidelines 108 on planning, analyzing, 
developing, testing, and refining web interventions. We will conduct usability testing with 12-15 
participants as recommended by Bastien 107. The evaluator notes the frequency/duration of behaviors 
that can indicate user problems/difficulties as well as measures such as time to finish a task, time 
recovering from errors, number of wrong choices, observations of frustrations, confusion, satisfaction, 
etc. 107. Morae software video-records the session and provides descriptive statistics on behaviors 
observed (frequencies, duration, etc.) and behavior patterns 107. We will conduct half of the usability 
sessions via the web using similar software without video. 
 
Randomized Controlled Trial. RCT participants will complete a baseline survey focusing on exposure and 
protection behaviors that takes a maximum of approximately 15 minutes on average. The study 
statistician will create a randomization scheme to randomize enrolled participants on a 2:2:1 basis to 
either the UV4.me2, UV4.me, or the e-pamphlet condition, respectively. Participants will not be informed 
of the differences in the intervention conditions beyond that they will all be online skin cancer risk 

reduction interventions. Participants will then be asked to complete follow-up assessments at 3, 6, and 12 
months post-baseline. 
 
4.3.1 
Overview of Intervention Conditions 
 
E-Pamphlet Condition: A free non-interactive e-pamphlet (“Skin Cancer Prevention and Early Detection” 
from the American Cancer Society) will be accessible via our website. We chose to include a pamphlet 
condition for the following reasons: 1) such pamphlets are widely available to the public and used by 
dermatologists and primary care providers, 2) to compare the effects of our intervention to a minimal 
intervention, 3) to compare our effects to prior studies that used pamphlet or minimal interventions, 4) to 
include a standardized rather than a variable intervention, 5) we also considered using an intervention for 
another health issue (e.g., nutrition, physical activity, sleep) but anticipated that skin protection 
organizations would be less likely to participate in that case, and 6) a no-treatment condition could affect 
adoption and reach and could be considered unethical for those with moderate to high skin cancer risk. 
 
Original UV4.me: UV4.me is targeted to young adults, personally tailored, and includes interactive, 
multimedia, and goal-setting components. It includes 12 modules with content related to a specific topic 
important in terms of risk or protective behaviors: Why do people tan? To tan or not to tan? Indoor 
tanning, UV & looks, UV & health, Skin cancer, Skin damage, Sunscreen, Shade, Clothes, Skin exams, 
and Sunless tanning. Several more general sections (e.g., avatar, MyStuff – a printable summary of 
tailored goals/recommendations) are also included. Tailoring algorithms were created to direct 
participants to focus on certain modules based on their responses to a few initial questions (e.g., the 
indoor tanning module was recommended for indoor tanners). Throughout the program, participants are 
asked questions and provided with tailored feedback (e.g., “Do you know people who tan? If so, how 
likely are they to affect your choice to tan or not?”). A number of interactive elements (e.g., videos, 
games) were created to encourage implementation. For example, at the end of each module, participants 
could choose to set a goal for the next two weeks or not (e.g., “For the next two weeks, I will not use a 
tanning bed.”). 
 
Enhanced UV4.me2: New features/strategies were chosen based on participant feedback from the original 
UV4.me grant (e.g., suggested strategies to make UV4.me more interactive), our data (e.g., number of 
mobile users who tried to access UV4.me), and reviews/models of effective e-Health interventions 65,66 
and implementation strategies 69,70,72. Several enhancements are expected to improve reach, 
implementation, and behavioral outcomes simultaneously: 
 
1. Mobile Site. A mobile version will likely improve reach directly because more people will have access 
and may improve effectiveness indirectly by facilitating ongoing implementation. The mobile platform 
will be developed for both Android and Apple devices.  
 

2. Incentives. The enhanced UV4.me2 site will also offer incentives in the form of clickable coupons and 
links to free samples for sun protection products (e.g., sunscreen). Appropriately selected incentives 
generally reinforce behavioral implementation, retention, and health behavior change, including in online 
trials, especially among those initially least motivated and/or for behaviors that are not intrinsically 
enjoyable (e.g., applying sunscreen) 71,109-115. These incentives will be used to reinforce behavioral 
implementation, retention, and health behavior change. 
 
3. Behavioral Tracking & Feedback. Users will be allowed to set goals and enter relevant behavioral 
events (e.g., sunbathing, indoor tanning, sunscreen use) over time and see them summarized graphically 
on the homepage and receive motivating feedback (e.g., “Great job, your tanning is going down over 
time!”). Behavior tracking and feedback are well-established empirically-supported behavior change 
techniques including for internet interventions 116,117. As with the original UV4.me site, participants 
will be able to set “alerts” to notify them (e.g., email, SMS texting) to remind them to work on their 
behavioral goals. 
 
4. Peer Interaction.  We will include an open-text component that will encourage participants to think, 
write, and communicate about relevant topics. Peer, normative, and social factors have a powerful 
influence on individual behavior, especially among adolescents and young adults, and social context was 
a factor identified in the review/model of effective eHealth interventions by Morrison and colleagues 66, 
has been shown to increase intervention implementation 118,119, and is consistent with other reviews of 
online interventions 65,69,70 and Bandura’s Social Cognitive Theory 120. There will be several spaces 
within the UV4.me2 website in which participants will be able to enter open-text comments and upload 
images (e.g., “Tell us about someone you know who has had skin cancer.”). These open-text areas within 
the website will also all be linked to and organized in one central “bulletin board” on the homepage so 
that users can find them easily without searching through the entire website. Participants will be able to 
choose to submit material privately (for themselves only) or publicly (for staff and all UV4.me2 users). 
We will be able to monitor and control this material such that what is submitted for online “publication” 
will initially be reviewed by study staff before public posting to ensure that material is appropriate (e.g., 
not profane, abusive, or irrelevant) and not un-helpful (e.g., does not recommend tanning). If most 
submissions are appropriate for a period of time, we will then move to an immediate posting procedure 
without staff pre-review with users being permitted to “flag” inappropriate content to alert staff to remove 
it. 
 
5. Ongoing News Updates. We plan to add new material (e.g., news/media stories, new research, events 
such as runs for melanoma charities) to UV4.me2 at least weekly with biweekly notifications to interested 
users (e.g., email, SMS texting). 
 
4.4 
Measures 
 
4.4.1 
Measures for Primary Aims 
 

Screener. Participants will be screened with Dr. Glanz’s 9-item Brief Skin Cancer Risk Assessment Tool 
(see Inclusion Criteria above) 104.  Internal and test-retest reliability compare favorably to those reported 
in the literature for similar items/scales 104. Participants will be required to understand English and have 
access to the internet via any modality (e.g., computer, smartphone) at least weekly.  
 
 
Demographics. Standard demographic items used by the census will assess: sex, age/birthdate, 
race/ethnicity, region of residence, education, income, and employment. We will include items on web 
use from the Health Information National Trends Survey 121, and will inquire about family history of 
skin cancer.     
 
Social Desirability. Social Desirability will be assessed using the 4-item Brief Social Desirability Scale 
that has demonstrated adequate internal reliability and validity among young adults 122.  
 
Reach. We will assess eligibility and enrollment overall and by recruitment source (i.e., skin protection 
organizations, Google Adwords, Facebook ads, consumer research panel, word of mouth) over 12-18 
months in order to assess reach/potential generalizability. Sources will be tracked by placing a unique 
identifying pixel in each of our authorized web ads so that we know when an individual referred from a 
pre-specified source accesses the study website. Additionally, we will query participants as to how they 
found out about the study to identify informal word of mouth diffusion (e.g., unpaid study Facebook 
page, earned media). The number of individuals who click on the web ads, access the study homepage, 
complete a screening form, are eligible/ineligible, and use the website without enrolling will be assessed. 
Organizational sources will be monitored and queried as to strategies (i.e., posting the study URL on their 
website, additional promotion) used to encourage individual participation. As we have previously, we will 
use Google Analytics to observe user characteristics and behavior in aggregate in terms of internet source 
including diffusion via word of mouth, geographic location, technology used (i.e., mobile or not), 
eligibility/enrollment rates, and so on. We will ask organizations to set up or allow us to set up a Google 
Analytics “goal” to assess such user characteristics and behavior in terms of individuals accessing the 
study ad from within the organization’s website. We will also query organizations about their number of 
users/members and website views if they are monitoring these. 
 
Efficacy. Skin cancer-related behavioral outcomes will initially be assessed at baseline and 3 months 
later. They will also be assessed within UV4.me2 via behavioral tracking. We will assess sun protection 
(e.g., sunscreen use, clothing, shade) and UV exposure (e.g., sunburns, intentional/incidental sun 
exposure, indoor tanning) using items adapted from Glanz and colleagues and Ingledew and colleagues 
that our team has cognitively tested and assessed psychometrically with young adults 123-125. Sunburns, 
indoor tanning, and sunless tanning will be secondary outcomes since engagement in these activities is 
less frequent than sun exposure and protection in general. Negative effects will be defined as an increase 
in sunburns. Several studies have demonstrated the reliability and validity of self-report questionnaires of 
UV exposure and protection compared to observation and objective measures with no systematic bias 
identified among various populations 126-130. As indicators of quality of life, we will inquire as to the 
amount of perceived benefits of and barriers to UV exposure and protection 62,131-134.  

 
4.4.2 
Measures for Secondary Aims 
 
Maintenance. Skin cancer-related behavioral outcomes (described above) will be assessed at 6 and 12 
months. We will also assess maintenance of reach, intervention implementation, and cost over time. 
 
Implementation. Using BeHealth’s data management system, we will record whether, how frequently, and 
for how long participants logged into the interventions, how many sections are completed, use of the 
enhanced features, and whether these variables are associated with behavioral outcomes. Whether 
individuals click into the pages with discounts/free samples will be noted. Whether and when a user 
accesses the e-pamphlet will be noted as well. We will ask participants to give Likert ratings of their 
satisfaction with and perceived helpfulness of the study/intervention and selected components (e.g., the 
enhanced features) both from within the intervention and a one-month follow-up inquiry. The 4-item App 
Subjective Quality scale from the Mobile App Rating Scale will be used to assess satisfaction (a = .93) 
135. We have experience using Google Analytics “goals” to track how frequently pages within modules 
are accessed. These implementation variables will also be linked to participant source (i.e., skin protection 
organizations, Google Adwords, Facebook ads, consumer research panel) when possible so that sources 
can be compared by participant intervention implementation. 
 
Cost. We will assess total and incremental costs and evaluate the cost-effectiveness of the interventions. 
We will first estimate costs from a payer perspective, capturing the explicit resources required to deliver 
the program after all start-up costs (e.g., development/programming costs) have been incurred. We will 
also estimate costs from a societal perspective, which includes all costs, regardless of who bears them. 
These will include all explicit costs as well as the implicit program costs related to participant time (i.e., 
opportunity costs 136. We will also denote whether costs are related to research or intervention delivery. 
We will collect cost data using a modified version of prior cost surveys developed by RTI 137,138 and 
adapted by Dr. Honeycutt for many interventions 139,140. The surveys capture all relevant labor- and 
non-labor-related inputs necessary to quantify costs. We will quantify participant time spent on the 
intervention using data from BeHealth’s system and valuing participant time using age- and sex-specific 
wage rates, netting out research incentive payments to participants. We will quantify labor costs as 
program staff time (via questionnaires) valued using actual or estimated wages. Non-labor costs will be 
collected using program billing records and include materials/supplies used to support program activities 
and costs for facilities and contracted services. To evaluate the cost-effectiveness of UV4.me2 versus 
UV4.me and the e-pamphlet, we will combine cost estimates with effectiveness outcomes to estimate the 
cost per incident of UV exposure and sunburn averted. 
 
 
4.5 
Compensation 
 

Acceptability/Usability Testing. Participants who complete testing will be eligible to receive $25 in gift-
cards or cash equivalent.  
 
RCT. In addition to discounts and free samples, enrolled participants in each intervention condition will 
be eligible to earn the following incentives in gift-cards or cash equivalent for completing study 
assessments: 1 month = $10, 3 months = $15, 6 months = $20, 12 months = $25, plus a $25 bonus for 
completing all time-points, and $5 for logging into the interventions for the first time, for a possible total 
of $100 throughout the year, plus a chance in a raffle for $100 at each time-point. We chose this level of 
incentives to balance two issues: 1) we wanted to encourage young adults to complete the longitudinal 
study assessments, but 2) when the research phase is complete, individuals will not be paid for using the 
program (other than the discounts and free products). 
 
5.0 
Risks to Participants 
 
The main potential risks are breach of confidentiality and mild emotional distress.  Some participants may 
become mildly distressed when considering their risk for skin cancer.   
 
6.0 
Potential Benefits to Participants 
 
We believe that the UV4.me and UV4.me2 internet interventions may reduce skin cancer risk behaviors 
among participants and thus ultimately reduce their skin cancer risk. 
 
7.0 
Provisions to Maintain the Confidentiality of Data 
 
The data that will be collected will not be sensitive in nature.  Therefore, any potential breach of 
confidentiality would likely not be serious. Research staff will be carefully trained and supervised to 
maintain confidentiality of data (e.g., by password protecting electronic data).  Data collected through the 
Internet will be obtained through secured means and stored on password protected servers as described 
above.  All data on servers are password-protected and limited to authorized research personnel. This 
high-level architecture is the same system that has been IRB-approved for use in multiple randomized 
controlled trials of Internet interventions with which BeHealth has been involved. 
 
Self-report psychosocial data will be collected directly from participants on-line using encrypted and 
password protected data collection software designed for specifically for that purpose.  Password-
protected data files will be backed-up onto tape on a daily basis.  Data will be maintained in such a way as 
to facilitate confidential data sharing.  Data will be encrypted for transmission.  Only members of the 
research team, all of whom will have been trained in research with human subjects and the importance of 

maintaining confidentiality, will have access to the data. The data will be labeled with participant 
numbers only.  
 
8.0 
Costs to Participants 
 
We anticipate no costs to participants during involvement in this research study. 
 
9.0 
Consent Process 
 
For the acceptability and usability testing, participants will have an opportunity to read the informed 
consent and HIPAA documents and review them with a staff member (or others) and ask questions before 
signing.  
 
For the RCT, participants will be screened, provide consent, and if eligible, complete a survey (< 15 min.) 
all online. Participants will have an opportunity to read the informed consent and HIPAA documents and 
review them with a staff member (or others) and ask questions. 
 
10.0 
Off-Study Criteria 
 
Based on our data, experience, the enhanced study procedures, and the consumer research panel’s 
experiences with young adults, we anticipate attrition of approximately 20% at each time-point. For drop-
outs, the reason for attrition will be analyzed if available to search for evidence of informative 
missingness.  
 
11.0 
Drugs and Devices 
 
Not applicable. 
 
12.0 
Multi-Site Research Study 
 
Not applicable. 
 
13.0 
Statistical Analysis 

 
13.1 
Statistical Analyses for Primary Aims 
 
Reach. We will use a one-sample test of a binomial proportion (two-sided, α=0.05) to determine whether 
the fraction of eligible subjects who enroll and complete the baseline is higher in this project than the 
prior study. Assuming that 70% of eligible subjects enroll and complete the baseline, we will need access 
to 2,143 (i.e., 1500/0.70) eligible subjects. Given this sample size, we will be able to detect an increased 
enrollment and baseline completion proportion with at least 98% power if the true proportion is 70%. 
Demographic data from enrolled participants will be compared to census data using chi-square goodness 
of fit tests (two-sided, α=0.05). We will also use the same methods to compare enrollment rates and 
representativeness separately by recruitment source (i.e., skin protection organizations, Google Adwords, 
paid Facebook ads, consumer research panel, word of mouth [e.g., unpaid Facebook page, earned 
media]). 
 
Efficacy. We will compare demographics across randomization arms using chi-squares and ANOVAs for 
categorical and continuous data, respectively. The primary outcomes are baseline to 3-month changes in 
UV exposure and protection. We will use multivariable linear regression to compare the effectiveness of 
UV4.me2 with UV4.me, and UV4.me2 with the e-pamphlet. We will not compare UV4.me to the e-
pamphlet because UV4.me compared to a control condition was established in the prior study. Covariates 
in these models will include study arm, recruitment month/season, US region, and demographic factors 
identified as significantly imbalanced across arms (confounders). Analyses will be performed separately 
for the two primary outcomes. Variance stabilizing transformations may be applied as needed. Hypothesis 
tests will be two-sided with a 1.25% Type I error to account for multiple testing (2 intervention 
comparisons x 2 outcomes). Based on our data, experience, the enhanced study procedures, and the 
consumer research panel’s experiences with young adults, we anticipate attrition of approximately 20% at 
each time-point. With 1500 young adults at baseline, we anticipate collecting 3-month follow-up data 
from 1200 participants, with 240 from the e-pamphlet arm and 480 participants from each of the other 
arms. The means (SDs) of the baseline to 3-month changes in UV exposure and protection indexes from 
the prior study control arm were 0.34 (0.74) and -0.37 (0.90), respectively. Given these parameters, we 
will have 80% power to detect small standardized effect sizes of 0.22 between the UV4.me2 and UV4.me 
arms, and 0.26 between the UV4.me2 and the e-pamphlet. These correspond to small detectable 
differences between intervention arms of 0.16 and 0.19 in baseline to 3-month changes in UV exposure 
and protection indexes, respectively. We will be able to detect slightly larger differences when the 
UV4.me2 and e-pamphlet arms are compared (detectable differences - UV exposure: 0.20; protection: 
0.24). Analyses will assume that missingness depends on observed data (i.e., missing at random; MAR) 
141. For drop-outs, the reason for attrition will be analyzed if available to search for evidence of 
informative missingness. Baseline variables will be compared by whether participants drop out or not. 
Significant variables will be added to the model to strengthen our MAR assumption. As a sensitivity 
analysis, we will use a shared parameter model with the assumption of missing not at random 142. If large 
changes in parameter estimation are found, we will use the results from the shared parameter model. 
 
13.2 
Statistical Analyses for Secondary Aims 

 
Maintenance.  We will compare baseline to 6- and 12-month changes in UV exposure and protection 
across study arms. Tests will be two-sided with a 5% type I error for these secondary analyses. We 
anticipate having follow-up data from approximately 960 and 768 participants at 6- and 12-months, 
respectively. This will allow for 154 participants in the e-pamphlet arm and 307 in each of the other two 
arms at 12 months. Detectable differences with 80% power are presented in Table 3. 
 
 
Implementation. Using BeHealth’s Wasabi data management system, we will create a summary index 
representing how frequently and for how long participants are logged into the interventions and how 
many sections are completed and determine whether this index is associated with behavioral outcomes. 
Whether and when a user accesses the e-pamphlet will be noted. We have experience using Google 
Analytics “goals” to track how frequently individual pages within modules are accessed. We will ask 
participants about their satisfaction with the study/intervention and selected components (e.g., the 
enhanced features) both from within the interventions and a one-month follow-up inquiry. These 
implementation variables will also be linked to participant source (i.e., skin protection organizations, 
Google Adwords, Facebook ads, consumer research panel) when possible so that sources can be 
compared in terms of participant intervention implementation. We will use the methods described for 
Aim 2 to compare these utilization and satisfaction measures across study arms, and Spearman’s 
correlation to measure the level of association between implementation metrics and longitudinal changes 
in UV exposure and protection indexes within and across study arms. Tests will be two-sided with a 5% 
type I error for these secondary analyses. We anticipate 1350 participants at 1-month follow-up (e-
pamphlet = 270, others = 540). Whether individuals click into the pages with discounts and free samples 
from skincare companies will be noted. We will assess whether accessing more incentives within the 
interventions is associated with greater intervention utilization and better behavioral outcomes. 
 
Cost. We will estimate mean and median costs for each intervention. We will distinguish between 
development costs (one-time capital investment or “sunk” costs) and implementation costs, or the costs to 
maintain the interventions. We will also denote whether costs are related to research or intervention 
delivery. Analyses will distinguish between fixed costs (costs that do not vary with enrollment, e.g., 
server maintenance, data storage), and variable costs, which increase with additional participants. Because 
most costs are expected to be fixed, the mean cost per participant will be driven largely by the number 
enrolled. To explore costs of scale-up, we will conduct sensitivity analyses around additional 
dissemination efforts and higher take-up rates. We will also estimate the cost-effectiveness of UV4.me2 
relative to the other conditions, calculating cost-effectiveness ratios as the difference in estimated costs 
divided by the difference in estimated effectiveness to determine the incremental cost per sunburn (or 
incident of UV exposure) averted for each intervention pair 141,142. We will also assess and estimate 
medical cost (e.g., OTC medication use, healthcare visits) offsets resulting from averted sunburns. To 
estimate longer-term cost-effectiveness, we will develop an Excel-based model to estimate quality-
adjusted life year (QALY) gains associated with each sunburn (which increases melanoma risk) averted. 
For the cost-effectiveness modeling, we will use data from the literature on the probability of developing 
various forms of skin cancer as a result of sunburns and on QALY losses associated with various cancer 
outcomes to estimate the cost per QALY gained. Expressing cost-effectiveness as cost per QALY gained 
will facilitate comparison of the UV4.me2 intervention with other preventive health interventions. For all 

cost-effectiveness analyses, we will perform one-way (and n-way) sensitivity analyses to examine the 
impact of varying input values over a plausible range. 
 
14.0 
Data Safety Monitoring Plan 
 
Computerized data collection allows for greater accuracy and completeness than other types of data 
collection. Computerized skip logic and data validation procedures will be utilized (e.g., “Please enter a 
number rather than a word.”).  Participants will be encouraged to complete a majority of the surveys.   
 
Raw data will be downloaded into password protected Excel and SPSS spreadsheets for cleaning, scoring, 
and initial descriptive analyses.  Data cleaning will involve checks for random and inconsistent 
responding.  Analyses will be conducted by the biostatistician (see Analyses section). 
 
15.0 
Adverse Events 
 
Adverse events will be reported to the PI immediately, and the PI will report adverse events and planned 
interventions to the FCCC IRB within the required time-frame.  Any subsequent AE will be reported 
along with prior ones for monitoring purposes. If 3 or more adverse events are reported, the study team 
will assess potential causes of the adverse events and, if events are clearly linked to study participation, 
consider discontinuing the study.  
 
16.0 
Quality Assurance Procedures and Participant Confidentiality 
 
Information about study subjects will be kept confidential and managed according to the requirements of 
the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Those regulations required a 
signed subject authorization informing the subject of the following: The protected health information 
(PHI) that will be collected from patient; who will have access to that information and why; who will use 
or disclose that information; the rights of a research subject to revoke their authorization or use their PHI. 
In the event that a participant revokes authorization to collect or use PHI, the investigator, by regulation, 
retains the ability to use all information prior to the revocation of subject authorization.  Data collected 
through the Internet will be obtained through secured means and stored on password protected servers as 
described above.  These data will be labeled with participant numbers only. All data on servers are 
password-protected and limited to authorized research personnel. 
 
We will employ several strategies to minimize repeat enrollment and enhance data quality, as 
recommended by the literature 98,143-151. These will include 1) verification of enrollment of 
participants with non-unique Personal Health Information (e.g., IP [Internet Protocol] address, email 
address, telephone number, name), and 2) separate analysis of data that indicates patterned (e.g., 

responding “always” to several items in a row), careless (e.g., too quick, illogical, gibberish, profane), or 
potentially fraudulent responding. We will create a summary index of such variables and conduct a latent 
class analysis to make a final decision about data inclusion/exclusion 146,152 as we did for the original 
UV4.me. The consumer research panel also employs such strategies to ensure data accuracy from its 
panel members such as providing researchers with panel entry data such as demographics, geolocation, 
socioeconomics, and health/behaviors to confirm eligibility and response consistency. By employing 
these procedures, we believe that we will be able to keep poor quality questionnaires relatively low at 5% 
of initially eligible participants, which is consistent with typical inattention 144,147,149. Thus, we will 
plan to over-recruit by as much as 5% until we have the required number of quality baseline surveys from 
unique eligible participants.  
 
17.0 
Participant Informed Consent 
 
Participants in the usability and acceptability phase will indicate their consent either written or electronic 
form, depending on the circumstances of their participation. Participants in the RCT will give electronic 
consent before accessing the baseline survey.  
 
18.0 
References 
 
1. 
USDHHS. The Surgeon General’s Call to Action to Prevent Skin Cancer. Washington, DC2014. 
2. 
American Cancer Society. How many people get melanoma skin cancer? 2009; 
http://www.cancer.org/docroot/CRI/content/CRI_2_2_1X_How_many_people_get_melanoma_skin_canc
er_50.asp. Accessed February, 12, 2010. 
3. 
Donaldson MR, Coldiron BM. No end in sight: the skin cancer epidemic continues. Semin. 
Cutan. Med. Surg. 2011;30(1):3-5. 
4. 
Linos E, Swetter SM, Cockburn MG, Colditz GA, Clarke CA. Increasing burden of melanoma in 
the United States. J. Invest. Dermatol. 2009;129(7):1666-1674. 
5. 
Rogers HW, Weinstock MA, Harris AR, et al. Incidence estimate of nonmelanoma skin cancer in 
the United States, 2006. Arch. Dermatol. 2010;146(3):283-287. 
6. 
Sutton A, Crew A, Wysong A. Redefinition of Skin Cancer as a Chronic Disease. JAMA 
Dermatol. 2015:1-2. 
7. 
American Cancer Society. Skin Cancer:Basal and Squamous Cell. Learn About Cancer 2012; 
http://www.cancer.org/Cancer/SkinCancer-BasalandSquamousCell/DetailedGuide/skin-cancer-basal-and-
squamous-cell-what-is-basal-and-squamous-cell. Accessed August 1, 2012. 
8. 
Baxter AJ, Hughes MC, Kvaskoff M, et al. The Queensland Study of Melanoma: environmental 
and genetic associations (Q-MEGA); study design, baseline characteristics, and repeatability of 
phenotype and sun exposure measures. Twin Research and Human Genetics 2008;11(2):183-196. 

9. 
Centers for Disease Control and Prevention. Skin Cancer: Risk Factors. 2011; 
http://www.cdc.gov/cancer/skin/basic_info/risk_factors.htm. Accessed July 30, 2012. 
10. 
Chuang TY, Brashear R. Risk factors of non-melanoma skin cancer in United States veterans 
patients: a pilot study and review of literature. J. Eur. Acad. Dermatol. Venereol. 1999;12(2):126-132. 
11. 
Goldberg MS, Doucette JT, Lim HW, Spencer J, Carucci JA, Rigel DS. Risk factors for 
presumptive melanoma in skin cancer screening: American Academy of Dermatology National 
Melanoma/Skin Cancer Screening Program experience 2001-2005. J. Am. Acad. Dermatol. 
2007;57(1):60-66. 
12. 
Grulich AE, Bataille V, Swerdlow AJ, et al. Naevi and pigmentary characteristics as risk factors 
for melanoma in a high-risk population: a case-control study in New South Wales, Australia. Int. J. 
Cancer. 1996;67(4):485-491. 
13. 
Markovic SN, Erickson LA, Rao RD, et al. Malignant melanoma in the 21st century, part 1: 
epidemiology, risk factors, screening, prevention, and diagnosis. Mayo Clin. Proc. 2007;82(3):364-380. 
14. 
Oberyszyn TM. Non-melanoma skin cancer: importance of gender, immunosuppressive status 
and vitamin D. Cancer Lett. 2008;261(2):127-136. 
15. 
Perkins JL, Liu Y, Mitby PA, et al. Nonmelanoma skin cancer in survivors of childhood and 
adolescent cancer: a report from the childhood cancer survivor study. J. Clin. Oncol. 2005;23(16):3733-
3741. 
16. 
Psaty EL, Scope A, Halpern AC, Marghoob AA. Defining the patient at high risk for melanoma. 
Int. J. Dermatol. 2010;49(4):362-376. 
17. 
Qureshi AA, Zhang M, Han J. Heterogeneity in host risk factors for incident melanoma and non-
melanoma skin cancer in a cohort of US women. J. Epidemiol. 2011;21(3):197-203. 
18. 
Siskind V, Aitken J, Green A, Martin N. Sun exposure and interaction with family history in risk 
of melanoma, Queensland, Australia. Int. J. Cancer. 2002;97(1):90-95. 
19. 
Etzkorn JR, Parikh RP, Marzban SS, et al. Identifying risk factors using a skin cancer screening 
program. Cancer control : journal of the Moffitt Cancer Center. 2013;20(4):248-254. 
20. 
Armstrong BK, Kricker A. How much melanoma is caused by sun exposure? Melanoma Res. 
1993;3(6):395-401. 
21. 
Dusza SW, Halpern AC, Satagopan JM, et al. Prospective study of sunburn and sun behavior 
patterns during adolescence. Pediatrics. 2012;129(2):309-317. 
22. 
Lazovich D, Forster J. Indoor tanning by adolescents: prevalence, practices and policies. Eur. J. 
Cancer. 2005;41(1):20-27. 
23. 
Skin Cancer Foundation. Preventing Skin Cancer. 2012; 
http://www.skincancer.org/prevention/sun-protection/prevention-guidelines/preventing-skin-cancer. 
Accessed July 31, 2012. 
24. 
Swerdlow AJ, Weinstock MA. Do tanning lamps cause melanoma? An epidemiologic 
assessment. J. Am. Acad. Dermatol. 1998;38(1):89-98. 

25. 
Vishvakarman D, Wong JC. Description of the use of a risk estimation model to assess the 
increased risk of non-melanoma skin cancer among outdoor workers in Central Queensland, Australia. 
Photodermatol. Photoimmunol. Photomed. 2003;19(2):81-88. 
26. 
Gies PH, Roy CR, Toomey S, McLennan A. Protection against solar ultraviolet radiation. 
Mutation Research/Fundamental and Molecular Mechanisms of Mutagenesis. 1998;422(1):15-22. 
27. 
Moloney FJ, Collins S, Murphy GM. Sunscreens: safety, efficacy and appropriate use. American 
journal of clinical dermatology. 2002;3(3):185-191. 
28. 
Lim HW, Cyr WH, DeFabo E, et al. Scientific and regulatory issues related to indoor tanning. J. 
Am. Acad. Dermatol. 2004;51(5):781-784. 
29. 
Coups EJ, Manne SL, Heckman CJ. Multiple skin cancer risk behaviors in the U.S. population. 
Am. J. Prev. Med. 2008;34(2):87-93. 
30. 
Heckman CJ, Coups EJ, Manne SL. Prevalence and correlates of indoor tanning among US 
adults. J. Am. Acad. Dermatol. 2008;58(5):769-780. 
31. 
Freedman KS, Nelson NM, Feldman LL. Smoking initiation among young adults in the United 
States and Canada, 1998-2010: a systematic review. Prev Chronic Dis. 2012;9(E5). 
32. 
Andrews JA, Tildesley E, Hops H, Li F. The influence of peers on young adult substance use. 
Health Psychol. 2002;21(4):349-357. 
33. 
Galliher RV, Kerpelman JL. The intersection of identity development and peer relationship 
processes in adolescence and young adulthood: contributions of the special issue. J. Adolesc. 
2012;35(6):1409-1415. 
34. 
Brownson RC, Colditz G, Proctor E, eds. Dissemination and Implementation Research in Health. 
Oxford University Press; 2012. 
35. 
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion 
interventions: the RE-AIM framework. Am. J. Public Health. 1999;89(9):1322-1327. 
36. 
RE-AIM.  http://www.re-aim.hnfe.vt.edu/ Accessed Jan 5, 2015. 
37. 
Blackman KC, Zoellner J, Berrey LM, et al. Assessing the internal and external validity of mobile 
health physical activity promotion interventions: a systematic literature review using the RE-AIM 
framework. J Med Internet Res. 2013;15(10):e224. 
38. 
Carnevale TD. Universal adolescent depression prevention programs: a review. J. Sch. Nurs. 
2013;29(3):181-195. 
39. 
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact 
of health promotion programs: using the RE-AIM framework to form summary measures for decision 
making involving complex issues. Health Educ. Res. 2006;21(5):688-694. 
40. 
Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to 
"employ" the RE-AIM model? Eval. Health Prof. 2013;36(1):44-66. 
41. 
Klesges LM, Dzewaltowski DA, Glasgow RE. Review of external validity reporting in childhood 
obesity prevention research. Am. J. Prev. Med. 2008;34(3):216-223. 

42. 
Shih ST, Carter R, Sinclair C, Mihalopoulos C, Vos T. Economic evaluation of skin cancer 
prevention in Australia. Prev. Med. 2009;49(5):449-453. 
43. 
Guy GP, Jr., Thomas CC, Thompson T, et al. Vital signs: melanoma incidence and mortality 
trends and projections - United States, 1982-2030. MMWR. Morb. Mortal. Wkly. Rep. 2015;64(21):591-
596. 
44. 
Gordon LG, Rowell D. Health system costs of skin cancer and cost-effectiveness of skin cancer 
prevention and screening: a systematic review. Eur. J. Cancer Prev. 2014. 
45. 
Do sunscreens have a role in preventing skin cancer? Drug Ther Bull. . 2011;49(6):69-72. 
46. 
Gordon LG, Scuffham PA, van der Pols JC, McBride P, Williams GM, Green AC. Regular 
sunscreen use is a cost-effective approach to skin cancer prevention in subtropical settings. J. Invest. 
Dermatol. 2009;129(12):2766-2771. 
47. 
Stern R, Weinstein M, SG B. Risk reduction for nonmelanoma skin cancer with childhood 
sunscreen use. Arch. Dermatol. 1986;122(5):537-545. 
48. 
Gibbons FX, Gerrard M, Lane DJ, Mahler HI, Kulik JA. Using UV photography to reduce use of 
tanning booths: a test of cognitive mediation. Health Psychol. 2005;24(4):358-363. 
49. 
Greene K, Brinn LS. Messages influencing college women's tanning bed use: statistical versus 
narrative evidence format and a self-assessment to increase perceived susceptibility. Journal of Health 
Communication. 2003;8(5):443-461. 
50. 
Jackson KM, Aiken LS. Evaluation of a multicomponent appearance-based sun-protective 
intervention for young women: uncovering the mechanisms of program efficacy. Health Psychol. 
2006;25(1):34-46. 
51. 
Roberts DC, Black D. Comparison of interventions to reduce sun exposure. Behav. Med. 
2009;35(2):67-76. 
52. 
Mahler HI, Kulik JA, Gerrard M, Gibbons FX. Effects of photoaging information and UV photo 
on sun protection intentions and behaviours: a cross-regional comparison. Psychol Health. 
2013;28(9):1009-1031. 
53. 
Turrisi R, Mastroleo NR, Stapleton J, Mallett K. A comparison of 2 brief intervention approaches 
to reduce indoor tanning behavior in young women who indoor tan very frequently. Arch. Dermatol. 
2008;144(11):1521-1524. 
54. 
Aarestrup C, Bonnesen CT, Thygesen LC, et al. The effect of a school-based intervention on 
sunbed use in Danish pupils at continuation schools: a cluster-randomized controlled trial. J. Adolesc. 
Health. 2014;54(2):214-220. 
55. 
Lenhart A, Purcell K, Smith A, Zickuhr K. Social media and mobile internet use among teens and 
young adults. Washington, DC: Pew Research Center;2010. 
56. 
Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions 
delivered over the internet? A systematic review of the published literature. Journal of medical Internet 
research. 2006;8(2):e10. 

57. 
Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: 
review and recommendations.  nn. Behav. Med. 2009. 
58. 
Bichakjian CK, Schwartz JL, Wang TS, Hall JM, Johnson TM, Biermann JS. Melanoma 
information on the Internet: often incomplete--a public health opportunity? J. Clin. Oncol. 
2002;20(1):134-141. 
59. 
Heckman CJ, Darlow S, Ritterband L, Handorf E, Manne S. Efficacy of an Intervention to Alter 
Skin Cancer Risk Behaviors in Young Adults. Am. J. Prev. Med. 2016. 
60. 
Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change 
model for internet interventions.  nn. Behav. Med. 2009. 
61. 
Fishbein M, Hennessy M, Yzer M, Douglas J. Can we explain why some people do and some 
people do not act on their intentions? Psychology, health & medicine. 2003;8(1):3-18. 
62. 
Cafri G, Thompson JK, Roehrig M, van den Berg P, Jacobsen PB, Stark S. An investigation of 
appearance motives for tanning: The development and evaluation of the Physical Appearance Reasons 
For Tanning Scale (PARTS) and its relation to sunbathing and indoor tanning intentions. Body Image. 
2006;3(3):199-209. 
63. 
Danoff-Burg S, Mosher CE. Predictors of tanning salon use: behavioral alternatives for enhancing 
appearance, relaxing and socializing. Journal of health psychology. 2006;11(3):511-518. 
64. 
Hillhouse J, Turrisi R, Stapleton J, Robinson J. A randomized controlled trial of an appearance-
focused intervention to prevent skin cancer. Cancer. 2008;113(11):3257-3266. 
65. 
Litvin EB, Abrantes AM, Brown RA. Computer and mobile technology-based interventions for 
substance use disorders: an organizing framework. Addict. Behav. 2013;38(3):1747-1756. 
66. 
Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health 
interventions? A review using techniques from Critical Interpretive Synthesis. Telemedicine journal and 
e-health : the official journal of the American Telemedicine Association. 2012;18(2):137-144. 
67. 
Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a 
systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, 
and mode of delivery on efficacy. J Med Internet Res. 2010;12(1):e4. 
68. 
Williams AL, Grogan S, Clark-Carter D, Buckley E. Appearance-based interventions to reduce 
ultraviolet exposure and/or increase sun protection intentions and behaviours: a systematic review and 
meta-analyses. British journal of health psychology. 2013;18(1):182-217. 
69. 
Brouwer W, Oenema A, Crutzen R, de Nooijer J, de Vries NK, Brug J. An exploration of factors 
related to dissemination of and exposure to internet-delivered behavior change interventions aimed at 
adults: a Delphi study approach. J Med Internet Res. 2008;10(2):e10. 
70. 
Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered 
interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ. Res. 
2008;23(3):427-439. 
71. 
Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Strategies to facilitate 
exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: 
a systematic review. Health Educ. Behav. 2011;38(1):49-62. 

72. 
Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical 
innovations in health and mental health. Med. Care Res. Rev. 2012;69(2):123-157. 
73. 
Geller AC, Glanz K, Shigaki D, Isnec MR, Sun T, Maddock J. Impact of skin cancer prevention 
on outdoor aquatics staff: the Pool Cool program in Hawaii and Massachusetts. Prev. Med. 
2001;33(3):155-161. 
74. 
Glanz K, Geller AC, Shigaki D, Maddock JE, Isnec MR. A randomized trial of skin cancer 
prevention in aquatics settings: The pool cool program. Health Psychol. 2002;21(6):579-587. 
75. 
Glanz K, Escoffery C, Elliott T, Nehl EJ. Randomized Trial of Two Dissemination Strategies for 
a Skin Cancer Prevention Program in Aquatic Settings. Am. J. Public Health. 2014:e1-e9. 
76. 
Glanz K, Steffen A, Elliott T, O'Riordan D. Diffusion of an effective skin cancer prevention 
program: design, theoretical foundations, and first-year implementation. Health Psychol. 2005;24(5):477-
487. 
77. 
Hall D, Kline M, Glanz K. Analysis of participatory photojournalism in a widely disseminated 
skin cancer prevention program. Health Promot Pract. 2011;12(5):666-672. 
78. 
Hall DM, Escoffery C, Nehl E, Glanz K. Spontaneous diffusion of an effective skin cancer 
prevention program through Web-based access to program materials. Prev Chronic Dis. 2010;7(6):A125. 
79. 
Hall D, Elliott T, Nehl E, Glanz K. Effectiveness of a targeted, peer-driven skin cancer prevention 
program for lifeguards. International Journal of Aquatic Research and Education. 2008;2:287-297. 
80. 
Hall DM, McCarty F, Elliott T, Glanz K. Lifeguards' Sun Protection Habits and Sunburns: 
Association With Sun-Safe Environments and Skin Cancer Prevention Program Participation. Arch. 
Dermatol. 2009;145(2):139-144. 
81. 
Escoffery C, Glanz K, Elliott T. Process evaluation of the Pool Cool Diffusion Trial for skin 
cancer prevention across 2 years. Health Educ. Res. 2008;23(4):732-743. 
82. 
Escoffery C, Glanz K, Hall D, Elliott T. A multi-method process evaluation for a skin cancer 
prevention diffusion trial. Eval. Health Prof. 2009;32(2):184-203. 
83. 
Rabin BA, Nehl E, Elliott T, Deshpande AD, Brownson RC, Glanz K. Individual and setting 
level predictors of the implementation of a skin cancer prevention program: a multilevel analysis. 
Implement Sci. 2010;5:40. 
84. 
Elliott T, Glanz K, Nehl EJ. Characteristics of swimming pools with high rates of objectively 
measured sunscreen use. J. Am. Acad. Dermatol. 2009;60(4):684-688. 
85. 
Glanz K, Isnec MR, Geller A, Spangler K. Process evaluation of implementation and 
dissemination of a sun safety program at swimming pools. In: Steckler A, Linnan L, eds. Process 
Evaluation in Public Health Interventions. San Francisco, CA: Jossey-Bass; 2002:58-82. 
86. 
Hall D, Dubruiel N, Elliott T, Glanz K. Linking agents' activities and communication patterns in a 
study of the dissemination of an effective skin cancer prevention program. J. Public Health Manag. Pract. 
2009;15(5):409-415. 
87. 
Amon KL, Campbell AJ, Hawke C, Steinbeck K. Facebook as a recruitment tool for adolescent 
health research: a systematic review. Academic pediatrics. 2014;14(5):439-447 e434. 

88. 
Batterham PJ. Recruitment of mental health survey participants using Internet advertising: 
content, characteristics and cost effectiveness. International journal of methods in psychiatric research. 
2014;23(2):184-191. 
89. 
Chu JL, Snider CE. Use of a social networking web site for recruiting Canadian youth for medical 
research. J. Adolesc. Health. 2013;52(6):792-794. 
90. 
Fenner Y, Garland SM, Moore EE, et al. Web-based recruiting for health research using a social 
networking site: an exploratory study. J Med Internet Res. 2012;14(1):e20. 
91. 
Lohse B. Facebook is an effective strategy to recruit low-income women to online nutrition 
education. Journal of nutrition education and behavior. 2013;45(1):69-76. 
92. 
Morgan AJ, Jorm AF, Mackinnon AJ. Internet-based recruitment to a depression prevention 
intervention: lessons from the Mood Memos study. J Med Internet Res. 2013;15(2):e31. 
93. 
Park BK, Calamaro C. A systematic review of social networking sites: innovative platforms for 
health research targeting adolescents and young adults. Journal of nursing scholarship : an official 
publication of Sigma Theta Tau International Honor Society of Nursing / Sigma Theta Tau. 
2013;45(3):256-264. 
94. 
Ramo DE, Prochaska JJ. Broad reach and targeted recruitment using facebook for an online 
survey of young adult substance use. J Med Internet Res. 2012;14(1):e28. 
95. 
Lane TS, Armin J, Gordon JS. Online Recruitment Methods for Web-Based and Mobile Health 
Studies: A Review of the Literature. J Med Internet Res. 2015;17(7):e183. 
96. 
Alexa_Internet_Inc. The top 500 sites on the web. 2012; http://www.alexa.com/topsites. 
Accessed April 21, 2015. 
97. 
Chang LL, Krosnick JA. National surveys via RDDTelephone Interviewing Versus the Internet: 
Comparing Sample Representativeness and Response Quality. Public Opin. Q. 2009;73(4):641-678. 
98. 
Craig BM, Reeve BB. Methods Report on the PROMIS Valuation Study: Year 1. 2013. 
99. 
Liu H, Cella D, Gershon R, et al. Representativeness of the Patient-Reported Outcomes 
Measurement Information System Internet panel. J. Clin. Epidemiol. 2010;63(11):1169-1178. 
100. 
National_Cancer_Institute. Food Attitudes and Behaviors (FAB) Survey. 2013; 
http://cancercontrol.cancer.gov/brp/fab/index.html. Accessed August 28, 2014. 
101. 
National_Cancer_Institute. FLASHE: Family Life, Activity, Sun, Health, and Eating Study. 2014; 
http://cancercontrol.cancer.gov/brp/hbrb/flashe.html. Accessed August 28, 2014. 
102. 
Stein K, Dyer M, Crabb T, et al. A pilot Internet "value of health" panel: recruitment, 
participation and compliance. Health and quality of life outcomes. 2006;4:90. 
103. 
West R, Gilsenan A, Coste F, et al. The ATTEMPT cohort: a multi-national longitudinal study of 
predictors, patterns and consequences of smoking cessation; introduction and evaluation of internet 
recruitment and data collection methods. Addiction. 2006;101(9):1352-1361. 
104. 
Glanz K, Schoenfeld E, Weinstock MA, Layi G, Kidd J, Shigaki DM. Development and 
reliability of a brief skin cancer risk assessment tool. Cancer Detect. Prev. 2003;27(4):311-315. 

105. 
Atkinson NL, Massett HA, Mylks C, Hanna B, Deering MJ, Hesse BW. User-centered research 
on breast cancer patient needs and preferences of an Internet-based clinical trial matching system. J Med 
Internet Res. 2007;9(2):e13. 
106. 
Heckman C, Darlow S, Munshi T, et al. Development of an Internet Intervention to Address 
Behaviors Associated with Skin Cancer Risk among Young Adults. Internet Interv. 2015;2(3):340-350. 
107. 
Bastien JM. Usability testing: a review of some methodological and technical aspects of the 
method. Int J Med Inform. 2009. 
108. 
Services USDoHaH. The Research-Based Web Design & Usability Guidelines, 
Enlarged/Expanded edition. . Washington, DC: Government Printing Office; 2006. 
109. 
Giles EL, Robalino S, McColl E, Sniehotta FF, Adams J. The effectiveness of financial incentives 
for health behaviour change: systematic review and meta-analysis. PloS one. 2014;9(3):e90347. 
110. 
Mantzari E, Vogt F, Marteau TM. Financial incentives for increasing uptake of HPV 
vaccinations: a randomized controlled trial. Health Psychol. 2015;34(2):160-171. 
111. 
Mantzari E, Vogt F, Shemilt I, Wei Y, Higgins JP, Marteau TM. Personal financial incentives for 
changing habitual health-related behaviors: A systematic review and meta-analysis. Prev. Med. 
2015;75:75-85. 
112. 
Alexander GL, Divine GW, Couper MP, et al. Effect of incentives and mailing features on online 
health program enrollment. Am. J. Prev. Med. 2008;34(5):382-388. 
113. 
Khadjesari Z, Murray E, Kalaitzaki E, et al. Impact and costs of incentives to reduce attrition in 
online trials: two randomized controlled trials. J Med Internet Res. 2011;13(1):e26. 
114. 
Schierenberg A, van Amsterdam J, van den Brink W, Goudriaan AE. Efficacy of contingency 
management for cocaine dependence treatment: a review of the evidence. Curr Drug Abuse Rev. 
2012;5(4):320-331. 
115. 
Young JM, O'Halloran A, McAuley C, Pirotta M, Forsdike K, Currow D. Unconditional and 
conditional incentives differentially improved general practitioners' participation in an online survey: 
randomized controlled trial. J. Clin. Epidemiol. 2014. 
116. 
Manzoni GM, Pagnini F, Corti S, Molinari E, Castelnuovo G. Internet-based behavioral 
interventions for obesity: an updated systematic review. Clin Pract Epidemiol Ment Health. 2011;7:19-28. 
117. 
Prestwich A, Kellar I, Parker R, et al. How can self-efficacy be increased? Meta-analysis of 
dietary interventions. Health psychology review. 2014;8(3):270-285. 
118. 
Brouwer W, Kroeze W, Crutzen R, et al. Which intervention characteristics are related to more 
exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med 
Internet Res. 2011;13(1):e2. 
119. 
Franklin VL, Greene A, Waller A, Greene SA, Pagliari C. Patients' engagement with "Sweet 
Talk" - a text messaging support system for young people with diabetes. J Med Internet Res. 
2008;10(2):e20. 
120. 
Bandura A. Social cognitive theory of mass communication. In: Bryant J, Oliver MB, eds. Media 
Effects: Advances in Theory and Research. New York, NY: Routledge; 2002:94-124. 

121. 
National_Cancer_Institute. Health Information National Trends Survey.  http://hints.cancer.gov/. 
Accessed August 28, 2014. 
122. 
Haghighat R. The Development of the Brief Social Desirability Scale (BSDS). Europe's Journal 
of Psychology. 2007;3(4). 
123. 
Glanz K, Yaroch AL, Dancel M, et al. Measures of sun exposure and sun protection practices for 
behavioral and epidemiologic research. Arch. Dermatol. 2008;144(2):217-222. 
124. 
Lazovich D, Stryker JE, Mayer JA, et al. Measuring nonsolar tanning behavior: indoor and 
sunless tanning. Arch. Dermatol. 2008;144(2):225-230. 
125. 
Ingledew DK, Ferguson E, Markland D. Motives and sun-related behaviour. Journal of health 
psychology. 2010;15(1):8-20. 
126. 
Glanz K, Mayer JA. Reducing ultraviolet radiation exposure to prevent skin cancer methodology 
and measurement. Am. J. Prev. Med. 2005;29(2):131-142. 
127. 
Glanz K, McCarty F, Nehl EJ, et al. Validity of self-reported sunscreen use by parents, children, 
and lifeguards. Am. J. Prev. Med. 2009;36(1):63-69. 
128. 
Glanz K, Gies P, O'Riordan DL, et al. Validity of self-reported solar UVR exposure compared 
with objectively measured UVR exposure. Cancer Epidemiol. Biomarkers Prev. 2010;19(12):3005-3012. 
129. 
O'Riordan DL, Glanz K, Gies P, Elliott T. A pilot study of the validity of self-reported ultraviolet 
radiation exposure and sun protection practices among lifeguards, parents and children. Photochem. 
Photobiol. 2008;84(3):774-778. 
130. 
O'Riordan DL, Nehl E, Gies P, et al. Validity of covering-up sun-protection habits: Association 
of observations and self-report. J. Am. Acad. Dermatol. 2009;60(5):739-744. 
131. 
Buller DB, Reynolds KD, Yaroch A, et al. Effects of the Sunny Days, Healthy Ways curriculum 
on students in grades 6 to 8. Am. J. Prev. Med. 2006;30(1):13-22. 
132. 
Buller DB, Taylor AM, Buller MK, Powers PJ, Maloy JA, Beach BH. Evaluation of the Sunny 
Days, Healthy Ways Sun Safety Curriculum for Children in Kindergarten through Fifth Grade. Pediatr. 
Dermatol. 2006;23(4):321-329. 
133. 
Cafri G, Thompson JK, Jacobsen PB. Appearance reasons for tanning mediate the relationship 
between media influence and UV exposure and sun protection. Arch. Dermatol. 2006;142(8):1067-1069. 
134. 
Robinson NG, White KM, Young RM, et al. Young people and sun safety: the role of attitudes, 
norms and control factors. Health Promot J Austr. 2008;19(1):45-51. 
135. 
Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating 
scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth. 
2015;3(1):e27. 
136. 
Foster EM, Dodge KA, Jones D. Issues in the Economic Evaluation of Prevention Programs. 
Appl Dev Sci. 2003;7(2):76-86. 
137. 
Honeycutt AA, Khavjou OA, Jones DJ, Cuellar J, Forehand RL. Helping the Noncompliant 
Child: An Assessment of Program Costs and Cost-Effectiveness. J Child Fam Stud. 2015;24(2):499-504. 

138. 
Khavjou OA, Honeycutt AA, Hoerger TJ, Trogdon JG, Cash AJ. Collecting costs of community 
prevention programs: communities putting prevention to work initiative. Am. J. Prev. Med. 
2014;47(2):160-165. 
139. 
French MT, Dunlap LJ, Zarkin GA, McGeary KA, McLellan AT. A structured instrument for 
estimating the economic cost of drug abuse treatment. The Drug Abuse Treatment Cost Analysis Program 
(DATCAP). J. Subst. Abuse Treat. 1997;14(5):445-455. 
140. 
Zarkin G, Dunlap L, Homsi G. The substance abuse services cost analysis program (SASCAP): A 
new method for estimating drug treatment services costs. Eval. Program Plann. 2004;27(1):35-43. 
141. 
Little RJA, Rubin DB. Statistical Analysis with Missing Data. 2nd edition ed. New York: John 
Wiley; 2002. 
142. 
Ten Have TR, Pulkstenis E, Kunselman A, Landis JR. Mixed effects logistic regression models 
for longitudinal binary response data with informative dropout. Biometrics. 1998;54:367-383. 
143. 
Bauermeister J, Pingel E, Zimmerman M, Couper M, Carballo-Dieguez A, Strecher VJ. Data 
Quality in web-based HIV/AIDS research: Handling Invalid and Suspicious Data. Field methods. 
2012;24(3):272-291. 
144. 
Bowen AM, Daniel CM, Williams ML, Baird GL. Identifying multiple submissions in Internet 
research: preserving data integrity. AIDS Behav. 2008;12(6):964-973. 
145. 
Bull SS, Vallejos D, Levine D, Ortiz C. Improving recruitment and retention for an online 
randomized controlled trial: experience from the Youthnet study. AIDS Care. 2008;20(8):887-893. 
146. 
Huang JL, Curran PG, Keeney J, Poposki EM, DeShon RP. Detecting and Deterring Insufficient 
Effort Responding to Surveys. Journal of Business Psychology. 2012;27:99-114. 
147. 
Konstan JA, Rosser BRS, Ross MW, Stanton J, Edwards WM. The Story of Subject Naught: A 
Cautionary but Optimistic Tale of Internet Survey Research. Journal of Computer Mediated 
Communication. 2005;10(2). 
148. 
Koo M, Skinner H. Challenges of internet recruitment: a case study with disappointing results. J 
Med Internet Res. 2005;7(1):e6. 
149. 
Meade AW, Craig SB. Identifying careless responses in survey data. Psychol Methods. 
2012;17(3):437-455. 
150. 
Reips UD. Standards for Internet-based experimenting. Exp Psychol. 2002;49(4):243-256. 
151. 
Quach S, Pereira JA, Russell ML, et al. The good, bad, and ugly of online recruitment of parents 
for health-related focus groups: lessons learned. J Med Internet Res. 2013;15(11):e250. 
152. 
Meade AW, Craig SB. Identifying careless responses in survey data. Psychological Methods. 
2012;17(437-55).