Mobile App Promoting Sexual Health for Young Black Men Who Have Sex With Men
Phase: NA
Conditions: HIV/AIDS and Infections, Sexual Behavior, Health Knowledge, Attitudes, Practice
Sponsor: ETR Associates

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
1 
 
3T: TUNE IN! TURN ON! TURN UP! 
PROTOCOL WITH IMPACT ANALYSIS PLAN 
  
 
Public Health Narrative (Excerpt from Phase II Proposal) 
Black men who have sex with men (BMSM) have been disproportionately affected by HIV/AIDS 
epidemic. Nearly half of the 1.2 million individuals living with HIV in the United States are African 
American, even though African Americans represent only 12% of the U.S. population. In 2009, 
Black MSM represented 73% of new HIV infections among Black men, and more new HIV 
infections occurred among young Black MSM (aged 13-24) than any other age and racial group of 
MSM, with an increase of 87% from 2005-2014. Young Black MSM (between the ages of 13 and 24) 
are more than twice as likely to be infected with HIV than young MSM of any other ethnic group. 
Yet there is a lack of school-based sexual education for this group of young people and few HIV 
interventions designed specifically to meet the needs of young BMSM. This project will develop a 
mobile app based multimedia interactive HIV/STI and sexual health intervention developed for use 
by YBMSM ages 14- 19 that delivers the intervention as a series of interactive activities tailored to 
the user and employs a theory based approach to address essential knowledge, perceptions of risk, 
peer norms, attitudes and skill with two primary goals: (1) To reduce HIV/STIs, it emphasizes 
partner reduction, avoidance of concurrent partners, condom use and HIV/STI tests; and (2) To 
improve sexual health, it helps participants to become clearer about what they do/don’t want to do 
sexually, to communicate their choices, and to learn ways to enhance sexual experience without 
increasing HIV/STI risk.    

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
2 
 
  
1) Research Questions That Address Program Effectiveness on Behavioral Outcomes 
 
These are consistent with outcomes listed on the www.ClinicalTrials.gov website. 
a. Primary research question(s): 
i. 
What is the impact of the 3T app relative to the 3T control website on the 
number of times engaged in condomless receptive or insertive anal intercourse 
or condomless vaginal intercourse in the last 3 months at follow-up, 
approximately 3 months post baseline.  
b. Secondary research question(s): 
Assessed at the 3-month follow-up (note: had to drop 6-month follow-up due to 
impact of COVID on study progress and timeline) 
i. What is the impact of the 3T app relative to the 3T control website on the 
number of times participants reported having been tested for STI, 
including HIV in the last 3  months?  
ii. What is the impact of the 3T app relative to the 3T control website on 
communication with a partner about sexual desires/behavior? (Dyadic 
Communication Scale) 
iii. What is the impact of the 3T app relative to the 3T control website on condom 
attitudes? (Condom Attitude Scale for Adolescents) 
iv. What is the impact of the 3T app relative to the 3T control website on condom 
use self-efficacy? (Sexual Risk Behavior Beliefs and Self-Efficacy Scale) 
v. What is the impact of the 3T app relative to the 3T control website on attitudes 
toward lube?  
c. Dropped outcomes:  
The following secondary outcome was dropped before the study launch and was not asked 
on the survey in a way that allowed this calculation:  
i. What is the impact of the 3T app relative to the 3T control website on the 
number of partners with whom had anal or vaginal sex without using 
condoms in the last 3 months?    
 
2) Description of the Intended Intervention and the Attention Control Conditions 
Intervention condition: The intervention is a mobile app delivered sexual health 
promotion program designed specifically for young black men who have sex with men or 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
3 
 
who are attracted to men. The mobile app includes 36 interactive activities including 
resource maps, pre-exposure prophylaxis (PrEP) and post-exposure prophylaxis (PEP) 
content, and an asynchronous communication forum. The app helps participants to 
become clearer about what they do/don’t want to do sexually, to communicate their 
choices. It also focuses on ways to increase healthy relationships, enhance sexual experience 
if having sex while reducing HIV/STI risk. 
Attention control condition: The control intervention featured a website containing seven 
health promotion materials in online pamphlet or post formats focusing on mental health 
(2 pamphlets), nutrition (2 pamphlets), relationships (2 pamphlets) and COVID (1 poster). 
Young people were asked to select 4 materials to review as part of their participation.  
• Mental Health 
o 5 Ways to Stop Stress 
o Suicide Talk, What To Do If You Hear It 
• Nutrition 
o Eating Well With No Time & No Money 
o What’s a Serving? Choose healthy serving sizes to stay at a healthy weight. 
• Relationships 
o Yes Means Yes, What is Affirmative Consent? 
o Relationship Check, Healthy or Un? 
• COVID 
o Actions to Protect Yourself and Others 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
4 
 
 
 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
5 
 
 
Study Design  
a. Sample   
The study was open to participants who self-identify as male, Black, African American, 
or biracial Black/African American; report being ages 14-19 at baseline; self-identify as 
cisgender male; report sexual attraction to men; and reside in any state in the United 
States. The age range was expanded to include young people ages 18-19 in May 2022 
after facing significant COVID-related challenges in recruiting young people in the 14–
17-year age range.  
b. Recruitment approaches 
The study went through four distinct recruitment phases to accrue the sample (Table 
2). Recruitment was impacted severely by the COVID pandemic, requiring multiple 
pivots in recruitment plans over a two-year period spanning August 2020 to July 2022. 
After facing COVID-related and other challenges in our proposed recruitment 
approaches, the study sample was secured through paid advertising on social 
networking sites.   
Table 2. 3T Recruitment Phases 
Phase 
Dates 
Recruitment Focus 
Results 
Phase 1: CBO 
recruitment 
partners 
August-
November 
2020 
We engaged selected CBOs to do 
direct recruitment for the study 
through their networks and social 
media, referring interested youth 
to the 3T recruitment website 
directly.  We had signed MOUs 
with the following CBOs: AIDS 
Alabama; the LGBTQ Center, Long 
Beach; Michigan Organization on 
Adolescent Health; Nashville 
Cares; and Triangle 
Empowerment Center.  
We established MOUs with 
five agencies; agencies were 
enthusiastic about study but 
ultimately not successful in 
engaging youth due to 
challenges of COVID and its 
impact on youth engagement 
with agency services. No 
youth were recruited during 
this phase, and we pivoted to 
general social media 
strategies.  
Phase 2: Broad 
social media and 
micro influencers 
May 2021 
We worked with Commando, a 
company specializing in social 
media study recruitment and ad 
placement for LGBTQ populations 
to use social media (Instagram) to 
promote the study nationally. At 
the same time, we identified a 
small number of micro influencers 
(2) who were paid to promote the 
study using IRB approved 
recruitment messages through 
their social media channels. We 
also tried TikTok and Snapchat but 
those platforms would not allow 
us to promote the study due to 
their restrictions on promoting 
studies (Snapchat) and promoting 
Over 800 individuals 
completed the screener over 
the launch weekend in May 
2021. Nearly all appeared to 
be out of country and age 
range but were claiming to be 
in age range and within the 
US.  We closed recruitment 
and rejected all surveys 
coming in during this period 
because the sample was not 
our desired population. We 
also stopped general social 
media recruitment and 
moved toward a new phase 
of more targeted recruitment 
to avoid international and 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
6 
 
sexually related content (Tik Tok). 
We also planned to use a 
snowball sampling approach 
during this phase, which allowed 
young people to refer up to three 
others from their networks and 
receive a stipend if the individual 
met eligibility criteria. 
national scam attacks. 
Further, we added an online 
video screening requirement 
for age and country 
verification for those who did 
screen as eligible. Further, we 
dropped use of social media 
influencers as their 
promotion also contributed 
to scam attacks. Finally, we 
also dropped snowball 
sampling to reduce scam 
attacks and because it was 
not yielding expected results 
in a different study.  After this 
phase, we considered and 
explored other avenues (e.g., 
schools and school clubs such 
as GSAs) but COVID 
continued to impact schools’ 
and agencies’ ability to reach 
youth and engage with us.  
Phase 3: Change in 
study design to 
conduct study 
directly at CBOs 
November 
2021-March 
2022 
We shifted our study design to 
work with direct service CBOs 
who serve the study population. 
The design involved training staff 
to recruit for the study and then 
invite young people to the site 
where study staff would have 
participants engage with their 
assigned app or website and 
complete the intervention while 
at the agencies. We aimed to 
recruit 1-2 sites near each of two 
project advocate staff members 
(1-2 in Atlanta or MS and 1-2 in 
TX) to minimize travel expenses 
and maximize the number of 
times we might be able to host 
young people at the sites.  
Up to 4 agencies expressed 
interest and then could not 
move forward because they 
were not seeing/serving 
enough young people who 
met the inclusion criteria at 
their agencies. This alternate 
design was then dropped. We 
discussed and decided to 
return to using paid 
advertising and shifted our 
age of eligibility to include 18- 
and 19-year-olds, seeking IRB 
approval to recruit via Grindr, 
Jack’d and Scruff.  
Phase 4: Paid 
advertising via 
social networking 
sites (Grindr, 
Jack’d/Scruff) 
Grindr inbox 
ad: May 24, 
2022 (12 
states) 
Jack’d/Scruff 
ads: May 2022 
and July 2022 
(National) 
We re-engaged with Commando, 
an agency specializing in paid 
advertising for studies centering 
LBBTQ populations with the new 
study age criteria. They prepared 
ads and placed them during the 
designated time frames.  
Screened 2,047.  
Enrolled 178 eligible 
participants who completed 
baseline (see final flow 
diagram for specifics).  
 
 
 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
7 
 
c. Screening criteria and process 
Participants were invited to complete a screening survey (via Qualtrics) that assessed 
eligibility for the study. Those who did not meet eligibility criteria received an immediate 
message thanking them for this interest in the study and informing them that they were 
not eligible for participation. Those who met eligibility criteria were forwarded to the 
study consent form and asked to read and complete the form to indicate their interest 
in taking part. (Note: As noted below: The IRB approved a waiver of parental consent 
for those ages 14-17).  
d. Consent process   
Participants who screened into the study were directed to an online assent (14-17) or 
consent (18-19) form that explained the study expectations along with risks and 
benefits. The IRB approved a waiver of parental consent for participants ages 14-17). 
Participants were asked to provide their consent if interested in taking part by clicking 
“yes” on the consent form and providing basic contact information (mobile number 
for texting and email as well as a question on which was their preferred method of 
contact). Those who provided affirmative consent went through a further confirmation 
protocol to reduce the risk for bots, scams, or those out of the country to enroll. The 
protocol included review of metrics from Qualtrics’ internal fraud detection system, 
ReCAPTCHA and Scamalytics. All prospective participants whose application was 
flagged as follow were rejected:  
Indicator 
Reject if 
 RelevantIDDuplicate 
 True/1 
 RelevantIDDuplicateScore 
 >75 
 RecaptchaScore 
 <.5 
 RelevantIDFraudScore 
 >30 
 BallotBoxStuffing 
 True/1 
 
Young people also were given a chance to indicate they needed more information 
before making  a study decision. Those who marked that option were asked to provide 
contact information so a study ambassador could follow-up to address any unanswered 
questions. Once participants’ eligibility, consent, and geographic location (US) were 
confirmed, participants received a link to the baseline survey.  
e. Random assignment process  
The 2-arm randomized controlled trial involves randomizing individual participants 
using an equal allocation (allocation ratio 1:1) to receive the 3T app or access to the 
attention control website. Participants were randomized after completing their baseline 
surveys using the randomizer feature of Qualtrics.   
 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
8 
 
f. Baseline and follow-up data collection 
The primary source of data for the outcome analyses is the self-report survey. The 
survey was administered two times: Baseline and 3-months post-baseline. The original 
study design included a 6-month follow-up timepoint that had to be dropped because 
of delays and challenges in getting the study launched during COVID.  
Data collection was ongoing due to rolling enrollment. Surveys were primarily 
distributed by SMS message. Shortly after their screener responses were reviewed and 
approved, prospective participants received a text message containing an explanatory 
message and a link that would be them to the baseline survey on the Qualtrics platform. 
Follow-up surveys were sent out in a similar fashion approximately 1 month and 3 
months post baseline. Participants  who had not yet completed a survey automatically 
received up to 5 reminders for survey completion in addition to their initial invitation. 
In cases where  the Qualtrics system indicated an SMS message bounced, email 
invitations to take the survey were attempted.  
3) Analysis 
Table 3. Behavioral outcomes used for primary impact analyses research questions. 
Outcome name 
Description of the outcome, 
including how it is 
operationalized (e.g. “The 
outcome is a yes/no response 
taken directly from the survey” 
or “the risk outcome is calculated 
as the average of the five risk 
indicator variables”).  
Source of the 
measure (e.g. 
performance 
measure) 
Timing of 
measure 
(e.g., 6 
months 
after 
program 
ends) 
i. Number of times engaged in 
condomless receptive or insertive 
anal intercourse or condomless 
vaginal intercourse in the last 3 
months at follow-up, approximately 3 
monthsa  
Count of condomless receptive 
anal, insertive anal, and/or 
vaginal sex acts with main 
partners and casual partners. 
Note: Participants who report 
they have never had sex or did 
not have sex in past 3 months 
(anal or vaginal) are assigned a 0. 
Constructed from the following 
variables (both T1 and T3 were 
used but only T3 is listed): 
T3_Q69_1 and T3_Q95_1 
(insertive anal, no condom); 
T3_Q69_3 and T3_Q95_3 
(receptive anal, no condom); 
T3_Q69_5 and T3_Q95_7 
(insertive vaginal, no condom); 
T3_Q29 (EVER had vaginal/anal 
sex); T3_Q30 (had vaginal/anal 
sex in last 3 months).  
Outcome 
survey 
3 months 
following 
baseline 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
9 
 
a This outcome was assigned a value of 0 if a respondent reported they have never had that type of sex or 
didn’t have that type of sex in the past 3 months. This allows for the full sample to be included in analytic 
models. This outcome will be dichotomized if the distribution is highly skewed.  
  
 
 
Table 4. Behavioral and attitudinal outcomes used for secondary impact analyses research questions. 
Outcome name 
Description of the outcome, including how it is 
operationalized (e.g. “The outcome is a yes/no 
response taken directly from the survey” or “the 
risk outcome is calculated as the average of the 
five risk indicator variables”). 
Source of the 
measure (e.g. 
performance 
measure) 
Timing of 
measure 
(e.g., 6 
months after 
program 
ends) 
i. Number of times 
participants 
reported having 
been tested for HIV 
in the last 3  
monthsa 
Count of number of times reported being tested 
for HIV in the last 3 months. Constructed from 
T1_Q55/T3_Q55 (how many times tested for HIV 
in last 3 months).   
Outcome survey 
3 months 
post baseline 
ii. Number of times 
participants 
reported having 
been tested for STIs 
(excluding HIV) in 
the last 3  monthsa 
Count of number of times reported being tested 
for STIs other than HIV in the last 3 months. 
Constructed from T3_Q58 (how many times 
tested for an STI/STDs in last 3 months). 
Outcome survey 
3 months 
post baseline 
iii. Sexual 
Communication Self-
Efficacy scale 
Average of 20 items on a 4-point scale (1 to 4), 
with a range of 4 - 80, with a higher score 
indicating better outcomes. Calculate the 
average if at least 50% of items were answered 
(10 of 20). Items embedded in and drawn from 
questions Q46, Q94, Q47, Q48 and Q49.  
 
 
Outcome survey 
3 months 
post baseline 
iv. Condom Attitudes 
Scale for 
Adolescents 
Average of 21 items on a 4-point scale (1 to 4) 
with a range from 21 to 84 with higher scores 
indicating better outcomes. (Two items were 
omitted from the original 23-item scale.) Ten 
questions are reverse coded. Items embedded in 
and drawn from questions Q40 and Q93.  
Outcome measure 
3 months 
post baseline 
v. Condom use self-
efficacy 
Average of 4 items adapted from the Sexual Risk 
Behavior Beliefs and Self-Efficacy Scale scored on 
a 4-point scale (1 to 4) with a range from 4 to 16 
with higher scores indicating better outcomes.  
Items embedded in and drawn from question 
Q42.  
Outcome measure 
3 months 
post baseline 
i. Attitudes towards 
lube 
Average of 3 items on openness to using sexual 
lubricant on a scale from 1 to 4 with higher 
scores indicating more positive views towards 
 
 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
10 
 
lube (better outcome). The third item is reverse 
coded. Items embedded in and drawn from 
question Q41. 
a If an outcome assessing the number of times has a very skewed distribution, it may be collapsed into a 
dichotomous variable (e.g., 0 times, 1 or more times) based on the final distribution.  
 
a. Analytic sample   
 
We will use complete case analysis as the benchmark sample for this study. The analytic 
samples for both primary and secondary outcomes will include all individuals who 
respond to the questions making up the outcomes at baseline and follow-up and have 
complete data on the covariates included in each  model.  
 
b. Data cleaning 
Missing data. We will examine missingness of potential covariates and use data from the 
follow-up survey to fill in missing values from baseline on covariates that are 
hypothesized not to change between the two survey timepoints. The primary and 
secondary outcomes will be analyzed using complete cases.  
 
Inconsistencies. Because of forced skip patterns pre-programmed into the electronic data 
collection devices, we expect the sexual behavior outcome data to be relatively clean 
within time points. Within time consistencies will be checked on questions asking about 
the type of sex had in the last 3 months and the number of times respondents had the 
type of sex across all partners. For respondents with inconsistencies, we will assess 
patterns across all variables and discuss decision rules. Across time inconsistency will be 
checked on questions asking whether respondents have ever had vaginal or anal sex. 
With only two data points, we will analyze the data with and without inconsistent cases.    
 
Extreme values. For questions asking about the number of times a behavior occurred we 
will examine extreme values in the context of other responses and apply a consistent 
decision rule. 
 
c. Assessment of baseline equivalence  
In addition to the baseline measures of the outcomes, we also will assess equivalence 
of the intervention and control groups at baseline on age and having a main partner. 
To assess equivalence between groups on each of these variables, we used regression 
analyses with the variable of interest as the dependent variable and the intervention 
indicator as the independent variable. The groups are considered not equivalent on a 
given variable if the p-value for the intervention indicator’s regression coefficient is < 
0.05 using the Wald test.  
d. Condition crossover and contamination 
An intent to treat model will be used in the analysis so that participants will be analyzed 
as randomized to treatment or control and followed up regardless of whether they 
interact with participants in the other condition.  

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
11 
 
e. Analytic approach for primary research questions 
i. Model specification:  
SPSS will be used to conduct the descriptive and bivariate analyses as well as 
the multivariable analyses. Statistical significance levels will be set at p<.05 
unless otherwise noted.  
Separate data analyses will be performed on each outcome variable named 
above and will progress through the following stages: 
First, descriptive analyses will be used to explore the data on all model variables 
at baseline and follow up for distributional assumptions and unanticipated 
patterns that might affect subsequent analyses. For example, as noted in the 
outcomes table, some outcomes may need to be dichotomized if the 
distribution is too skewed towards 0.   
Second, bivariate analyses will be used (1) to compare the baseline demographic 
and background characteristics of the intervention and control groups and (2) 
to test for relationships between the outcome variable and potential covariates 
(confounders) being measures. A potential covariate will be included in the 
model if: (1) the variable is associated with the condition indicator at p<0.15; 
and (2) the variable is associated with the outcome at p<0.15 in individual 
bivariate analyses. Our plan for covariate screening is derived from those 
suggested in Altman (1991) and Hosmer and Lemeshow (1989). In the latter, 
it is suggested that p<.25 as a screening criterion may be more appropriate than 
p<.05 because the latter often fails to identify variables that may be important 
to control. We have traditionally “split the difference” and selected a p<.15 to 
preserve degrees of freedom of the model. We routinely include the baseline 
value of the outcome regardless of screening criteria (Pocock et al, 2002). 
Below, we clarify which covariates will be included regardless of these 
screening criteria. (4) Bivariate analyses will also be conducted to assess 
baseline equivalence for the final primary analytic samples.  
Third, multivariable analyses will be conducted using regression analyses (linear 
or logistic) to evaluate the research questions as definitively as possible.  
Each model will include an indicator variable denoting (1) intervention 
group, (2) the baseline outcome variable, (3) the number of days between the 
baseline and follow-up survey (if there is a large range), (4) age at baseline, and 
(5) a set of a priori demographics and outcome-related covariates that screened 
positively as candidates for inclusion in the bivariate screening step. 
Covariates  
Covariates to be forced into all models on an a prior basis:  
1. Baseline value of outcome 
2. Actual # of days between baseline and follow-up 

3T Protocol+Impact Analysis Plan 
Version: 2.0 
 
Last updated: 3.8.23 
 
3T Protocol with Impact Analysis Plan  
12 
 
3. Age at baseline 
Race was not included as it was a study eligibility requirement (black 
identifying).  
 
A priori covariates to be screened into the models: 
1. 
With whom have had sex (cis men only vs cis men+others) 
2. 
Effect of COVID pandemic on their ability to see a romantic or 
sexual partner (0=not at all, 1=somewhat, 2=very). 
3. 
Effect of COVID pandemic on ability to get condoms (0=not 
harder, 1=may or may not be harder, 3=harder). 
4. 
Has a main partner.     
 
NOTE: if there is high correlation (r>.5) between some of these indicators we 
will not include them all. 
 
ii. Sample attrition 
Attrition analyses will be conducted to examine the rate of attrition by 
condition. Results from the attrition analyses will help evaluate the need to 
temper interpretations of outcome results.  
iii. Sensitivity analyses  
(1) Sensitivity analyses will be conducted examining the sub-group of 
participants with matching self-reported birthdates at baseline and the 
3-month follow-up and within the allowable age range (14-20).    
f. Analytic approach for secondary research questions 
Analyses for secondary research questions will proceed using the same steps as 
described for the primary research questions.  
4) Additional planned analyses   
Additional exploratory analyses of the impact evaluation data may include the following.  
a. Subgroup analyses based on participants in each condition who used their respective 
interventions (use evident from log in data vs no use evident) may be completed for 
the primary and secondary outcome variables.  
b. Analyses of the following variables not specified as primary or secondary: 
a. Lube Knowledge (which are safe w/latex condoms) 
b. HIV Knowledge (includes PrEP and PEP awareness) 
c. Examination of impact on primary outcome for those reporting casual partners if 
sample size warrants this level of exploration.