JMIR FORMATIVE RESEARCH

Savage et al

Original Paper

The Development of a Digital Patient Navigation Tool to Increase
Colorectal Cancer Screening Among Federally Qualified HealthCenter Patients: Acceptability and Usability Testing
Leah C Savage1, BS; Luz Estefhany Soto-Cossio2, MD; Francesca Minardi1, MA, MPH; Matthew Beyrouty2,3, MPA;
Julie Schoonover2,3, MD; Jay Musella2, LMSW; Michaela Frazier2, LMSW, PhD; Cristina N Villagra1, MPH; Jamilia
R Sly1,3, PhD; Joel Erblich1,4, PhD; Steven H Itzkowitz5, MD; Lina H Jandorf1, MA; Neil S Calman2,3, MD; Ashish
Atreja6, MPH, MD; Sarah J Miller1,3, PsyD
1

Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States

2

The Institute for Family Health, New York, NY, United States

3

Department of Family Medicine and Community Health, Icahn School of Medicine at Mount Sinai, New York, NY, United States

4

Department of Psychology, Hunter College, City University of New York, New York, NY, United States

5

Division of Gastroenterology, Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States

6

Innovation Technology Division, University of California Davis, Sacramento, CA, United States

Corresponding Author:
Sarah J Miller, PsyD
Department of Population Health Science and Policy
Icahn School of Medicine at Mount Sinai
Box 1077
1 Gustave L Levy PlaceNew York, NY, 10029
United States
Phone: 1 2128247783
Email: sarah.miller@mssm.edu

Abstract
Background: Federally Qualified Health Centers (FQHCs) are an essential place for historically underserved patients to accesshealth care, including screening for colorectal cancer (CRC), one of the leading causes of cancer death in the United States. Novelinterventions aimed at increasing CRC screening completion rates at FQHCs are crucial.
Objective: This study conducts user testing of a digital patient navigation tool, called eNav, designed to support FQHC patientsin preparing for, requesting, and completing CRC screening tests.
Methods: We recruited English- and Spanish-speaking patients (N=20) at an FQHC in New York City to user-test the eNav
website (2 user tests; n=10 participants per user test). In each user test, participants engaged in a “think aloud” exercise and aqualitative interview to summarize and review their feedback. They also completed a baseline questionnaire gathering data about
demographics, technology and internet use, medical history, and health literacy, and completed surveys to assess the website’s
acceptability and usability. Based on participant feedback from the first user test, we modified the eNav website for a second
round of testing. Then, feedback from the second user test was used to modify and finalize the eNav website.
Results: Survey results supported the overall usability and acceptability of the website. The average System Usability Scalescore for our first user test was 75.25; for the second, it was 75.28. The average Acceptability E-scale score for our first user test
was 28.3; for the second, it was 29.2. These scores meet suggested benchmarks for usability and acceptability. During qualitative
think-aloud exercises, in both user tests, many participants favorably perceived the website as motivating, interesting, informative,
and user-friendly. Respondents also gave suggestions on how to improve the website’s content, usability, accessibility, and appeal.
We found that some participants did not have the digital devices or internet access needed to interact with the eNav website at
home.
Conclusions: Based on participant feedback on the eNav website and reported limitations to digital access across both user
tests, we made modifications to the content and design of the website. We also designed alternative methods of engagement witheNav to increase the tool’s usability, accessibility, and impact for patients with diverse needs, including those with limited accesshttps://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Savage et al

to devices or the internet at home. Next, we will test the eNav intervention in a randomized controlled trial to evaluate the efficacyof the eNav website for improving CRC screening uptake among patients treated at FQHCs.
(JMIR Form Res 2024;8:e53224) doi: 10.2196/53224
KEYWORDS
digital navigation; digital health; Federally Qualified Health Center; colorectal cancer; cancer screening; mobile phone

Introduction
Colorectal cancer (CRC) is the second most deadly cancer, withmore than 1.9 million new cases and 935,000 CRC-related
global deaths annually [1]. Due to recent increases in early-age
onset CRC, the United States Preventive Services Task Forcelowered the recommended age for average-risk adults to begin
CRC screening to 45 (previously 50) [2]. Several tests can screen
for CRC, including at-home stool-based tests and visual tests(eg, colonoscopy) [3]. Recent data indicate that amongindividuals 45 years of age and older, more than 1 in 3 are not
up to date with CRC screening [4]. Of concern, some patient
populations are less likely to complete CRC screenings,
including patients who receive care at Federally Qualified HealthCenters (FQHCs). In fact, among patients treated at HealthResources and Services Administration–funded community
health centers, including FQHCs, only 41.9% of adults aged
50-74 years have completed a CRC screening within therecommended time frame [5]. When including individuals aged
45-50 years, the CRC screening rates at FQHCs are likely evenlower. In a national effort to reduce CRC morbidity andmortality, the National Colorectal Cancer Roundtable launched
the “80% in Every Community” initiative with the goal ofincreasing the national CRC screening rate to 80% [6]. It isparticularly important that we prioritize CRC screening among
patients who are at greater risk of being unscreened, including
the many patients treated at FQHCs.
Patient navigation is an evidence-based intervention designed
to help individuals better navigate complex health care systems
and overcome barriers to care [7]. More than a decade of clinical
trials, including studies conducted in FQHC and community
health care settings, have demonstrated that patient navigation
can significantly improve CRC screening rates [8-16]. Althoughcost-effective [17,18], patient navigation typically requires
economic resources (ie, hiring and training of staff), which canlimit its ability to be widely integrated and sustained in standard
clinical care, particularly in low-resourced or safety net healthcare settings such as FQHCs [16]. Digital patient navigation or
patient navigation delivered via electronic media, is a novel
form of patient navigation that holds promise for overcoming
barriers to person-led navigation. Specifically, digital navigation
may be a low-cost, accessible solution to improve CRC
screening among patients treated at FQHCs. Importantly,
patients can access digital patient navigation platforms frommultiple locations and at convenient days or times. While digitalpatient navigation will likely not replace person-led patientnavigation, it is possible that this intervention could reduce theworkload and burden of patient navigators and providers throughworkflow changes and the optimization of technology [19].

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

Recent studies support the efficacy of digital patient navigation,
especially SMS text message navigation, to improve CRC
screening uptake [20-22], although results are mixed [23]. More
broadly, other studies have examined the efficacy of digital
health interventions to improve cancer screening uptake,including CRC screening [24]. More research is needed to better
understand the impact of digital navigation interventions,especially those that incorporate novel features (eg, motivational
support), can be accessed at flexible times and locations, andare offered in multiple languages.
In a collaborative effort to improve CRC screening uptake in
the FQHC setting, our study team developed a novel digitalnavigation intervention. The eNav intervention includes a
website and follow-up SMS text messages. The intervention isdrawn from the Health Belief Model (HBM) [25] and aims to
impact HBM-informed constructs (eg, perceived benefits andbarriers, perceived susceptibility, and perceived severity) and,in doing so, improve CRC screening completion. Significantly,
the study team conceptualized, developed, and iteratively
user-tested eNav in close collaboration with community
stakeholders. This ongoing partnership is integral in ensuringthat an equity lens informs the eNav website and that it isappropriate for FQHC patients [26]. Our study teamhypothesized that the eNav intervention will help improve CRC
screening uptake among patients treated at FQHCs. The firststep in this program of research was to conduct user-testing inorder to examine the usability and acceptability of the website
component of the eNav intervention with FQHC patients. This
paper presents initial data from the iterative user-testing of theeNav website. This user-testing study is part of a larger research
project aiming to test the efficacy of the eNav intervention in
an FQHC environment within a randomized controlled trial(RCT).

Methods
Team Overview
The investigative team for this study represents a partnership
between the Icahn School of Medicine at Mount Sinai, an
academic medical center, and the Institute for Family Health,an FQHC network with clinics throughout New York City andthe mid-Hudson Valley. Drawing from existing literature andthe HBM, the investigative team developed the first iterationof the eNav website.

Description of the First Iteration of the eNav Website
Intervention Overview
We designed the eNav website as a digital navigation tool forpatients who are due or overdue for CRC screening and havean upcoming primary care appointment at an FQHC. The eNav
JMIR Form Res 2024 | vol. 8 | e53224 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
website can be accessed on multiple digital devices (ie,computer, smartphone, and tablet) and at a time and place thatis convenient for the patient. To maximize reach and impact,
the website is offered in English and Spanish, can be accessed
before and after the patients’ primary care appointment, delivers
information over time (via text, graphics, and close-captioned
videos), and allows patients to request a screening test. The
eNav website contains the components explained next.

Information
On the primary landing pages, the eNav website offers
information about CRC including (1) CRC and its risk factors,(2) polyps, (3) signs and symptoms of CRC, (4) the role ofscreening, and (5) the importance of screening for all people
older than 45 years. The website also includes a “Frequently
Asked Questions” page and an “Additional Information” pagethat provides more detailed information to help navigate patientsto get screened including instructions on how to complete each
CRC screening test and information about overcoming barriers
to screening (eg, information about insurance coverage).

Decisional Support
Previous research has shown that CRC screening completion
rates may improve when patients are given a choice of whichtest to complete [27]. As such, the eNav website includes a“screening options” page that provides information via text,
images, and animated videos about 3 commonly used CRC
screening tests: fecal immunochemical test (FIT), multitarget
stool DNA test (FIT-DNA), and colonoscopy. The website also
provides a table comparing the requirements for each test (eg,collecting a stool sample at home for FIT and FIT-DNA andgoing to a clinic or hospital for the colonoscopy procedure),acknowledging barriers patients may face for each test.
Crucially, the website makes clear that for some patients, acolonoscopy may be the test recommended by their provider.
Significantly, information about coverage of the costs related
to each procedure is also included.

Motivational Support
An integral component of the eNav website is motivational
support for completing CRC screening, offered through a “MyWhy” video that features patients from diverse backgrounds
sharing their reasons for completing CRC screening. The eNav
website encourages users to identify their own reasons forwanting to get screened for CRC.

Risk Assessment
Not all participants are eligible for stool-based testing. For
example, some patients with symptoms (eg, rectal bleeding) or
certain risk factors (eg, Lynch syndrome) may need acolonoscopy rather than a stool-based test. To facilitate
conversations between patients and providers about CRC
screening options, the eNav website includes a risk assessmentthat helps determine whether patients are eligible for stool-based
testing. Although not a comprehensive risk assessment, the toolassesses whether patients have current physical symptoms that
could necessitate a colonoscopy (eg, blood in the stool). They
are also asked whether they have a personal or family historyof conditions that can increase the risk of developing CRC (eg,Lynch syndrome). After completing the risk assessment, patients
https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

Savage et al
receive an output delineating their CRC screening options. If
patients do not report any of the listed symptoms or potential
risk factors, the eNav website offers patients a choice to request
a stool-based screening (ie, FIT or FIT-DNA) or a visual
screening (ie, screening colonoscopy). Of note, if patients are
flagged as potentially ineligible for stool-based testing, eNav
recommends that these patients talk to their primary careprovider about the most appropriate screening option anddisplays a video of a provider discussing CRC risk factors. The
results from the risk assessment are sent to the treating primarycare provider and can be used to guide shared decision-making
about CRC screening. The questions and responses for the risk
assessment were adapted (with permission) from the Colorectal
Cancer Alliance CRC screening quiz [28]. The content of thequestions was also guided by the C5 Colon Cancer Prevention
Risk Assessment and Screening Form [29]. Our investigative
team and clinical partners, including family physicians andgastroenterologists, reviewed the final risk assessment.

Option to Request a CRC Screening Test
If patients are deemed potentially appropriate for stool-based
testing through the risk assessment, they are prompted to selecta CRC screening test—FIT, FIT-DNA, or colonoscopy. If apatient requests a CRC screening test, the treating primary careprovider will be sent a message notifying them that their patient
is interested in the selected CRC screening test. The treating
primary care provider will then determine whether to place theorder for the CRC screening test. Importantly, all patients,
regardless of risk level, will have the chance to discuss CRC
screening with their provider at their upcoming primary carevisit.

Text Reminders + Instructions
The eNav intervention is composed of the eNav website andfollow-up SMS text messages. In addition to viewing thewebsite, patients will receive a series of SMS text message
reminders to help navigate them to get screened for CRC. For
example, patients who request a FIT test are instructed to pickup the FIT test at their appointment, are reminded to complete
the test after their appointment, and are given information on
how to receive another test, if needed. The SMS text messages
also contain hyperlinks to different sections of the eNav website
to help them effectively prepare for and complete the CRC
screening tests (see Multimedia Appendix 1).

Procedures
As demonstrated in Figure 1, two user tests (n=10 participants
per user test) were conducted to gather feedback on the eNav
website from FQHC patients. The first user test was conductedin December of 2022 and the second was conducted in May of2023. For each user test, study team members distributed flyers
in an FQHC clinic waiting room and lobby to recruit patients
to participate in the study. If patients expressed interest in the
study, they met with a research coordinator in a private room
to complete the informed consent process. After providing
informed consent, patients were asked to view and interact withthe eNav website on a study-provided computer or on theirpersonal device (eg, smartphone). Participants engaged in a“think aloud” exercise, during which they were asked to provide
JMIR Form Res 2024 | vol. 8 | e53224 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
their feedback, in real-time, on the eNav website. During thethink-aloud exercise, study team members produced behavioral
note summaries, including their observations of participants’behaviors and any challenges interacting with eNav. After the
think-aloud exercise, participants engaged in a brief qualitativeinterview to provide additional feedback about the eNav website(eg, suggestions for change). The think-aloud exercises andinterviews were audio-recorded. Although the SMS text message

Savage et al
reminders and instructions component of the intervention was
not comprehensively user-tested, participants also provided
insights on their preferred frequency of SMS text messaging.
Finally, participants in each user test completed a questionnairethat assessed demographic information, technology and internet
use, medical history (eg, history of having CRC screenings),
and health literacy.

Figure 1. Iterative eNav website user-testing procedures.

The study team used the results of the first user test to refine
the content and improve the user experience of the website tocreate eNav version 2.0. Then, this second iteration of the eNav
website was user-tested on a new set of participants, using thesame procedures, to confirm that the new iteration of the app
was both usable and acceptable. The results from the seconduser test were again used to refine and finalize the eNav website.

Participants
Patients were eligible to participate in the user-testing if they
were (1) treated at an FQHC, (2) aged 45-75 years, (3) Englishor Spanish-speaking, and (4) able to provide consent. Patientswere excluded if they were hearing or vision impaired.
Participants who completed the first user test were not eligible
to participate in the second. In total, the sample size for the first
2 user tests was 20 patients.

Measures
Medical History and Demographics
A brief survey was used to assess participant demographics (eg,age, race, ethnicity, and sex), medical history (eg, previous CRC
screening), and technology ownership and use. The survey
questions were adapted from demographic surveys used in our
https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

team’s previous research and from questions from the PEW
Research Center [30,31].

Usability
The System Usability Scale (SUS) [32] evaluated the usabilityof the eNav website. The SUS includes 10 items scored on a
5-point Likert scale (1=strongly disagree and 5=strongly agree).
We adapted the language of the SUS from “this system” to “thiswebsite” so that it was applicable to eNav.

Acceptability
The Acceptability E-scale [33] assessed the acceptability of theeNav website. The E-scale is composed of 6 questions scoredon a 5-point Likert scale. We adapted the language of the E-scalefrom “this computer program (ESRA-C)” to “this website.”
Furthermore, we expanded the question, “How understandable
were the questions?” to be, “How understandable were thequestions about your health on the website (eg, risk
assessment)?” and altered the item, “Was the amount of timeit took to complete this computer program (ESRA-C)
acceptable?” to be, “Was the amount of time it took to navigate
this website acceptable?” We also shortened the question “Howhelpful to you was this computer program (ESRA-C) in

JMIR Form Res 2024 | vol. 8 | e53224 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
describing your symptoms and QOL?” to “How helpful was
this website?”

Savage et al

To evaluate participants’ health literacy, we used 1 question
from the BRIEF scale [34]. The BRIEF scale is composed of 4questions, but previous research studies [35] have demonstrated
that 1 screening question may be sufficient for detecting limited
and marginal health literacy skills in clinic populations “Howconfident are you filling out medical forms by yourself?”

We applied descriptive statistics to analyze the medical anddemographic information. To analyze the SUS score, wesummed a contribution score for each item. Then, we multiplied
the total score by 2.5 to produce an overall score ranging from0 to 100. An SUS score greater than 68, the mean score, isconsidered acceptable [37]. To analyze the website’s
acceptability, the scores on the Acceptability E-scale weresummed to produce an overall score ranging from 6 to 30.
According to the literature, a score of 80% or higher (total scoreof 24 or higher) is considered acceptable [33].

Data Analysis

Ethical Considerations

Thematic analysis [36] was used to analyze the qualitative data.
Study team members transcribed the audio recordings of thethink-aloud exercises and qualitative interviews. Then, the study
team thoroughly reviewed the transcripts and behavioral notes.
The data were first coded into 2 broad categories, based on theinterview
guide—“endorsedfeatures”
and“problems/suggestions for change.” After the first round ofcoding, the study team developed a coding scheme andcodebook based on the “endorsed features” and“problems/suggestions for change” commonly identified byparticipants. Then, transcripts and behavioral notes were coded
a second time, based on the codebook. All transcripts werecoded by at least 2 independent coders. Once the coding wascomplete, themes were identified, reviewed, and defined. Given
that the purpose of the study was to understand the participants’
perceptions of the intervention, our qualitative analyses focusedon identifying semantic-level themes.

The study was performed in line with the principles of the
Declaration of Helsinki. Study procedures were reviewed andapproved by the Icahn School of Medicine at Mount Sinai’s
institutional review board (STUDY-20-01888). Informed
consent was obtained from all individual participants includedin the study. All study data were stored in HIPAA (Health
Insurance Portability and Accountability Act)-compliant secure
servers and were deidentified prior to analysis. Participants werecompensated with a US $50 gift card for their time and effort.

Health Literacy

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

Results
Demographic Results for User Tests 1 and 2
We enrolled 10 participants in user test 1 and 10 participants
in user test 2. See Table 1 for demographic information andmedical history for both user tests.

JMIR Form Res 2024 | vol. 8 | e53224 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Savage et al

Table 1. Demographic characteristics of user test 1 sample (n=10) and user test 2 sample (n=10) for eNav website user-testing study.
Demographic category

User test 1

User test 2

Range

52-69

54-74

Mean (SD)61.3 (5.66)

62.8 (5.98)

English8 (80)

4 (40)

Spanish2 (20)

6 (60)

White

0 (0)

1 (10)

American Indian or Alaska Native

1 (10)0 (0)Black or African American

3 (30)

4 (40)Other

6 (60)

5 (50)

Non-Hispanic or Latino

6 (60)

4 (40)

Hispanic or Latino

4 (40)

6 (60)Male

4 (40)

1 (10)

Female6 (60)

9 (90)

Unemployed

8 (80)

6 (60)

Employed

2 (20)

4 (40)

Colonoscopy

6 (60)

9 (90)

Fecal immunochemical test (FIT)

2 (20)0 (0)Multitarget stool DNA test (FIT-DNA)

2 (20)0 (0)Stool-based test, unsure which one

0 (0)1 (10)Yes

2 (20)

1 (10)

No

8 (80)

9 (90)

Married or domestic partnership

3 (30)

4 (40)

Separated

2 (20)0 (0)Widowed

1 (10)

1 (10)

Single (never married or in a domestic partnership)

4 (40)

5 (50)<High school

2 (20)

3 (30)High school (12th grade)

2 (20)

3 (30)

Some college

4 (40)

3 (30)

College

2 (20)0 (0)Missing

0 (0)1 (10)

No insurance

0 (0)4 (40)

Private health insurance

1 (20)0 (0)Age (years)

Language, n (%)

Race, n (%)

Ethnicity, n (%)

Sex, n (%)

Employment, n (%)

Past completed colorectal cancer screening test, n (%)

Family history of colorectal cancer, n (%)

Marital status, n (%)

Highest level of education completed, n (%)

Health insurance, n (%)

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
Demographic category

Savage et al
User test 1

User test 2

Medicare (including Health First)

1 (10)0 (0)Medicaid

4 (40)

3 (30)

Medicare and Medicaid

2 (20)0 (0)Other (Metroplus)

1 (10)0 (0)Other (United Healthcare)

1 (10)

1 (10)

Other (Undefined)

0 (0)2 (20)

Less than 10,000

1 (10)0 (0)10,000 to 19,999

0 (0)1 (10)

20,000 to 29,999

1 (10)

2 (20)

30,000 to 39,999

0 (0)1 (10)

40,000 to 49,999

0 (0)1 (10)

50,000 or more

1 (10)

2 (20)

Prefer not to answer

7 (70)

3 (30)

Estimated total household income (US $), n (%)

User Test 1 Results
Technology Access and Use
Of the participants who completed the first user test, only 70%(n=7) had a smartphone. Of the respondents who reported nothaving or being unsure if they had a smartphone (n=3), 2
reported using no other devices at home and 1 used a tablet.
Although 9 (90%) of the participants in the first user test
accessed the internet at home, the quality of their internet was
variable. Of the first user test sample, 1 (10%) reported notaccessing the internet at home, 6 (60%) reported having slow
internet speed, and 2 (20%) reported having interrupted internet.
In fact, only 2 (20%) of the participants reported that they had
no problems with their home internet.

Health Literacy

independently; 4 (40%) considered themselves “somewhat
confident,” 1 (10%) said they were “quite a bit confident,” and5 (50%) reported feeling “extremely confident” with the task.

Usability and Acceptability
The average SUS score for our first user test was 75.25, which
is above the suggested benchmarks for usability [37,38]. The
average Acceptability E-scale score for our first user test was
28.3, which is higher than the suggested benchmark for
acceptability, and close to the maximum score of 30 [33].

Feedback From Think-Aloud Exercises, Behavioral Note
Summaries, and Qualitative Interviews
In Table 2, we separated positive and negative feedback on theeNav website from user test 1 into the themes identified in thedata.

Regarding health literacy, all participants reported a certain
degree of confidence in completing medical forms

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Savage et al

Table 2. Positive and negative qualitative feedback on eNav website from user test 1 study sample (n=10).
ThemesExample quotes or behavioral observations

User test 1: positive feedback
Content•••

The content was interesting,•informative, and important
The content was easy to understandThe website had diverse representation••“I like it. You can’t get no plainer than that. You explained everything. You know why you needto do it and how it needs to be done, you know, in all different kinds of ways...it’s very, very interesting, Like I said, there’s a lot of stuff I learned for myself just watching this, that I didn’t
even know. I’m grateful that I took the time to watch it, and I guarantee that anybody thatwatches this, they’re going to learn a lot from it, and change their life.”“The fact that it is such a step by step showing the individual tests that are available and how applicable it is to each, how different it is to each individual...is great...”“This is wonderful. I like the different races in here to show...Beautiful, bravo, that video was
outstanding. Outstanding in so many ways. Hitting the ethnic groups, you know, different cultures,
everybody.”

Usability and acceptability•

The website was user-friendly •and easy to navigate••“It is easy to navigate, and the videos are very attractive and educative.”“I believe that even the person that is not tech savvy will be able to...it’s pretty straightforward.”“It’s ideal for a person that’s busy on the go, so you are stopping for a minute, you’re going througha video, and you can fill out a form. One two three.”••“I like the website, and the color is perfect...the videos are very attractive and educative.”“It feels great...You know, the welcoming faces. They are not frowning or anything. They’re quite
content, which is nice.”“I like the animated [videos]. I think it’s cute. It’s like it’s friendly.”

Design•The style and design of thewebsite was appealing•
Motivation••

The website would motivate
people to get screened for

•
•

CRCa
Individuals wanted to share the•website with others“I’m interested more than ever before to get checked out.”“It’s really good, I like it, it will make you get less scared. And it helps you to be more, you know,
let me go and do this, it is encouraging. Now I’m going home now to talk to my husband, because
you know, he needed to do a colonoscopy...”“...this right here conveys the convenience and the ease of testing so that people won’t be scaredof the process.”

User test 1: negative feedback and suggestions for change
Content•••

The content should include•
more information
The content should use laylanguage and avoid medical•
jargonThe content should address
ambivalence about completing •CRC screening“I think there should be a little more information about what is cancer...That’s what seems to be
missing, you know what I’m saying? Because this right here also assumes the listener and thereader knows what cancer is”“What I want to know is like some people like to get the colonoscopy, right, you have to pay forit, you have to have the medical insurance, some people don’t have medical insurance, so how dothey get it done if they don’t have medical insurance?”“What does [CRC] look like, how does it start? Where does it come from? How does it develop?...And there are food products that you eat, and you don’t know they’re harming you…I want
to see how it is? how does it start?” Translated from Spanish: [“¿Cómo se ve, como empieza [elcáncer colorrectal? ¿De donde proviene? ¿Cómo se desarrolla? Y hay productos alimenticios que
tú comes y no sabes que te están haciendo daño...Yo quiero ver ¿cómo es? ¿ Cómo empieza?”]

Usability and acceptability••

The website should be more•accessible for those with limited technology skills or owner- •ship
Alternate versions of the eNav•
website should be offered

Not all participants requested a CRC screening test (behavioral observation)“Most people, a lot of people, like older people, they don’t have experience with the computer.
To know more, they have to go on the computer, there should be other ways for them to learn
about this...[Doing it on the phone] would be perfect.”“I am tech savvy to navigate through it, however, there are a lot of individuals that are not. And
remember when you’re hitting the low-income areas, that’s a toughie right there. It would be great
to put it in the hospital and have like some volunteers walk them through it.”

Designhttps://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
ThemesSavage et al
Example quotes or behavioral observations•

The style and design of thewebsite should be bolder andmore engaging••“You want to keep it comprehensive and concise, you know, appealing, you know, wonderful,friendly, you know? So this is just a little bland right now. It can be jazzed up a little bit.”“Maybe a little cute card at the end thanking them for that? You know, a very nice little card that
opens, a virtual card, thanking them for caring, for caring about yourselves with something likethat.”

Motivation•a

The website should emphasize •the urgency of screening for
CRC•“Should be a little more dramatic, with more emphasis on the seriousness of symptoms...notenough drama here about the actual condition, the harmful condition like blood, vomiting andpain.”“I also think that you should put somebody that has cancer and somebody that don’t have thecancer, so you could compare how the how it looks...You know, some people need to get scaredbecause a lot of people don’t do the test.”

CRC: colorectal cancer.

Solutions to Identified Problems
Drawing directly from the qualitative and quantitative resultsof the first user test, we made the following changes to the eNav
website.
Include Additional Information
To address patient queries, we added and expanded content (viagraphics, texts, and videos) to provide more information about
CRC and the different CRC screening tests.
Emphasize the “Request Test” Cue to Action
Some participants did not select a test, so we emphasized andhighlighted the button to request a test.
Improve Accessibility of Technology
We removed functions that require an elevated level of tech
literacy (eg, login page). Patients will also have the option tocome into the clinic and receive assistance using aclinic-provided device to complete the risk assessment andrequest a test in advance of their primary care appointments.
Offer Engagement Boosters
Patients who do not complete the eNav risk assessment within
7 days of their primary care appointment will be sent a message
with a link directly to the risk assessment and the “screening
options” video (eNav lite). If patients still do not complete therisk assessment within 2 days of their appointment, they will
be sent a message with the “screening options” video and aphone number that they can call to complete the risk assessment
and request a test (eNav with assistance).
Enhance Style
We added bolder colors and included stock photos.
Improve Accessibility of Content
We made the font bigger and bolder.
Address Ambivalence
We added content to address patients’ ambivalence, particularly
surrounding getting a colonoscopy. For example, we addedreferences to resources about colonoscopy, a video about thecolonoscopy procedure, and a video including a testimonialfrom a patient who had completed a colonoscopy.

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

Emphasize Urgency
We added a “five facts about colorectal cancer” section
highlighting relevant incidence and mortality statistics. We also
included a patient testimonial from a cancer survivor about theirexperience being diagnosed and treated for CRC.

User Test 2 Results
Technology Access and Use
In the second user test, all 10 participants (100%) reported
having a smartphone. Additionally, all 10 participants (100%)accessed the internet at home. Similarly, to the first user test,
however, the quality of their internet at home was variable. In
fact, only 4 (40%) reported having no problems with their homeinternet; 5 (50%) reported experiencing slow internet and 3(30%) reported experiencing interrupted internet.

Health Literacy
In terms of health literacy, 1 (10%) participant consideredthemselves a little bit confident filling out medical forms bythemselves, 1 (10%) said they were quite a bit confident, and
8 (80%) reported feeling extremely confident with the task.

Usability and Acceptability
After excluding 1 participant with a missing value, the average
SUS score for our second user test was 75.28, which is above
the mean SUS score or the suggested benchmark for usability[37,38]. The average Acceptability E-scale score for our seconduser test was 29.2, which is higher than the recommended
benchmark for acceptability and near the maximum score of
30. Furthermore, both scores improved marginally between thefirst and second user tests; the SUS score increased from 75.25
to 75.28 and the Acceptability E-scale score increased from
28.3 to 29.2, confirming that the second iteration of the eNav
website remained highly usable and acceptable.

Feedback From Think-Aloud Exercises, Behavioral Note
Summaries, and Qualitative Interviews
In Table 3, we separated positive and negative feedback on theeNav website from user test 2 into the themes identified fromthe data.

JMIR Form Res 2024 | vol. 8 | e53224 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Savage et al

Table 3. Positive and negative qualitative feedback on eNav website from user test 2 study sample (n=10).
Theme

Example quotes or behavioral observations

User test 2: positive feedback
Content•••

The content was interesting,•informative and important•
The content was easy to understandThe content had diverse representation and perspectives•“Yeah, I feel good. Health is important, and it’s important to know things like this.”“Well, I really liked it. You informed me. and also gave me more information about things, it has
also given me more guidance on the things that I should look for...” Translated from Spanish:[“Bueno en verdad me ha gustado. Por qué me ha informado. y también más información de cosas,
me ha dado más orientación sobre las cosas que yo debo de buscar...”]“Everything, I loved it! I liked the screening stories, and I also liked the information about food,diet, exercise and also the part about what we should do to prevent. Prevention. It is very good,
because we as individuals can avoid that. Not smoking, not using alcohol and you can prevent it
so you don’t get that cancer. It’s very interesting.” Translated from Spanish: [“Todo ¡Me encantó!,
Me gustó la historia de la detección y me gustó también lo de la comida, la dieta, el ejercicio. Lo
que uno debe hacer para prevenir. La prevención. Está muy bien. Porque uno puede evitarlo, no
fumando, no usando alcohol y uno lo puede prevenir para que no le de cáncer. Está muy interesante.”]Usability and acceptability•

The website was user-friendly •and easy to navigate••“Because you see people talking about their story, sometimes we don’t like to read...”“I like the videos because if people don’t know feel comfortable with the computer, but withvideos, they’re just watching them, and there are people who don’t know how to read or something
and listen to it.” Translated from Spanish: [“Los videos me gustan, por qué si la gente no sabemucho con la computadora. Pero con los videos, los está viendo, y hay gente que no sabe leer oalgo y lo escucha.”]“That’s all easy to understand. I could see it here on my phone and I see it well.” Translated from
Spanish: [“Todo eso está fácil de entender. Lo pude ver acá en mi teléfono y lo veo bien. ”]

Design•The style and design of thewebsite was appealing•“It’s very colorful. It’s not boring...bright colors are good.”••“It’s good, gives you, you know, courage.”“This page I am going to share it with my daughters, my friends, so that they can also read it.”Translated from Spanish: [“Esta página se la voy a pasar a mis hijas a mis amistades, para que
ellos también la lean.”]

Motivation••

The website would motivate
people to get screened for
CRCa
Individuals wanted to share thewebsite with others

User test 2: negative feedback and suggestions for change
Content•••

The website should provide•
more clarification about techni- •cal language, particularly in therisk assessment
The content should include•
more information
The content should use lay
language and avoid medical
jargonWhat is Lynch Syndrome? Translated from Spanish: [“Qué es el Síndrome de lynch?”]“The only thing. FITb and FIT-DNAc, they need to explain, what is the difference between those
two. Yeah, everybody knows what colonoscopy is. But they didn’t explain why the FIT and the
FIT-DNA are different.”“I think [the doctor in the video] could have went into a little more detail...She said, if you havethis if you had that...she could have said...well, maybe covered a little more on symptoms.”

Usability and acceptability••

Improve accessibility and usability for those with limited
technology skills or technology
ownership, particularly in therisk assessment
Correct technical glitch

•
•
•
•

Not all participants requested a CRC screening test [behavioral observation]
Some participants asked the research coordinators to show eNav to them because they were
struggling to navigate the website and the risk assessment on the computer [behavioral observation]
Generally, participants were more comfortable scrolling through the website and navigating therisk assessment on their phones [behavioral observation]“Why don’t you navigate this thing? Because I’ll just tell you what my answer is.”Designhttps://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
Theme

Savage et al
Example quotes or behavioral observations••“I feel like [the font]’s a little small. Translated from Spanish: [“Siento que [la letra] es un poquito
pequeña.”]The website should emphasize •the urgency of screening for
•
CRC
•
•

“I think [the website] could be more powerful.”“Sometimes it’s better to shock people into the reality of the condition...If you value your life,
you have to [get screened].”“These people are a little too soft for the seriousness of what we are talking about.”“I mean you could...have a couple more stories. You know, other people to tell them what theywent through with, that, you know...But maybe with the screening, more testimonials...and how
they are doing now, like the survivor story, experience.”

The website should improve
the style and increase the font
size

Motivation•a

CRC: colorectal cancer.

b

FIT: fecal immunochemical test.

c

FIT-DNA: multitarget stool DNA test.

Solutions to Identified Problems
Since the goal of the second user test was to improve eNav in
preparation for testing the website within an RCT, we
implemented the following solutions to address the major
problems highlighted by participants.

Include Additional Information
To address participants’ desire for more information on selected
topics, we created a glossary with in-text pop-ups throughout
the website to define medical terminology. We also bolded or
highlighted information about symptoms and expanded theinformational content included throughout the website (eg, about
CRC disparities, CRC risk factors, and the differences between
FIT and FIT-DNA).

Update Risk Assessment
We altered the risk assessment language (eg, the descriptionsof CRC symptoms) to be more descriptive. We also added “don’tknow” to the “none of the above” answer options. In addition,
we changed the functionality of the risk assessment so that it ismore user-friendly (eg, automatically directs patients to the topof the next question).

Emphasize Urgency
We emphasized the CRC survivor testimonial by placing it ina more prominent part of the website. We also added questions
about patients’ own motivations for screening to the “Patient
Stories” page.

Improve Content Accessibility
In order to improve accessibility, we moved videos to the topof the website pages and removed large sections of text.

Enhance Style, Design, or Usability
Since many participants seemed more comfortable on theirphones, we improved the presentation of the eNav website onphones. We also increased the font size on the website.

Fix Technical Difficulties
Participants encountered minor technical difficulties (eg, anerror page); we collaborated with the developers to fix thesetechnical glitches on the back end of the website.
https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

Discussion
Principal Findings
This study conducted iterative user-testing of a digital patientnavigation tool, which aims to increase CRC screening uptake
among patients receiving care at FQHCs. The eNav intervention,
informed by the HBM, provides critical navigation support topatients due for CRC screening, including information,
motivational support, decisional support, and reminders. The
eNav website can serve as an information hub, which can beaccessed multiple times at convenient times and locations.
Significantly, patients’ risk assessment answers and, if
applicable, test requests are communicated to the health careteam so that patients’ engagement with eNav is integrated into
their care. It was critically important to user test the eNav in an
FQHC setting to ensure that the website was appropriate for theintended users. See Multimedia Appendix 2 for a screenshot ofa page from the updated, post–user-testing version of the eNav
website.
Overall, our study demonstrated that the eNav website is highly
acceptable and user-friendly. In both user tests, the usability
and acceptability scores (as measured by the SUS and the
Acceptability E-scale) exceeded the suggested benchmarks.
Furthermore, the scores remained above the suggested
benchmarks in the second user test, after significant websitechanges to address earlier user commentary. Moreover, during
both user tests, respondents described the website favorably.
For example, many participants stated that eNav was motivating,interesting, informative, and user-friendly. In both user tests,
participants suggested including more content on the website
to ensure that it conveyed all necessary information about CRC
screening and the seriousness of CRC. In addition, respondents
gave suggestions on how to improve the website’s usability,
accessibility, and overall appeal. We used participants’qualitative feedback to inform modifications to the intervention,
especially in terms of content, design, and style, to ensure thatthe website has maximum acceptability and usability.
A key takeaway from the user tests is that some participants
lacked the digital access needed to interact with the eNav
website at home. The literature shows that, although technology
JMIR Form Res 2024 | vol. 8 | e53224 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
use has grown in the last decade, disparities in technology accessand use remain [39]. For example, according to the PEW
Research Center, as of 2021, 43% of families with householdincomes less than US $30,000 in the United States do not havehome broadband services and 41% do not have a desktop or
laptop computer [39]. Indeed, 27% of adults in lower-income
households report using the internet only on their smartphones[39]. In our sample, we found that there was demonstrated
variability in access to and quality of technology. While almost
all of our sample (95%) accessed the internet at home, themajority of participants in both user tests reported someproblems with their internet, including slow internet speed andinterrupted service. Beyond digital access, some participants
explained that they were concerned that members of theircommunities, especially older individuals, would not have theskills needed to navigate the website on their own, particularly
on computers.
To maximize accessibility, including for patients with lower
digital access and skills, and to ensure we are not inadvertently
widening disparities in CRC screening completion, we designed
the eNav intervention to be device agnostic and require minimal
technology skills. Based on the results of the user-testing, we
made additional modifications to the eNav intervention in orderto improve accessibility and usability for patients with limited
digital access and readiness. First, we moved all videos to beeasily accessible at the top of each website page. Furthermore,
and most notably, we removed the login process from thewebsite; patients are now able to access the website directly
and only need to provide their information to complete the risk
assessment. In addition, as previously explained, patients who
do not engage in the eNav intervention will be offered
engagement boosters called “eNav lite” (text-based) and “eNavwith assistance” (navigator-assisted) that can help them request
a CRC screening test. Finally, patients who do not have accessto technological devices at home (eg, computer, smartphone,and tablet) or who would prefer not to interact with eNav athome will have the option to view the website in person, on aclinic-provided device. These modifications are important not

Savage et al
only for patients who struggle with technology access andliteracy but also for patients who may prefer simplified formsof communication. In sum, we made these changes, alongside
the previously mentioned refinements of the content and designof the eNav intervention, to increase the usability, accessibility,
and impact of the intervention for patients with diverse needs.

Limitations
Since we recruited patients from an urban FQHC clinic,
convenience sampling is a limitation of this study. Patients whochose to participate in our research study may not fully represent
the FQHC patient population. Furthermore, all participants wereolder than the age of 50 years, so there was no representationfrom patients in the recently expanded age range for screening(ie, age 45-50 years). A third limitation is that all participantsin both user tests had completed a CRC screening in the past,
although 2 participants in user test 1 and 1 participant in user
test 2 were due for screening at the time of recruitment.
Including patients who had never completed a CRC screening
would have strengthened the study. Included patients, however,were able to draw from their previous experience with CRC
screening to provide valuable feedback. A final limitation of
the study is that participants’ responses may have been impacted
by social desirability. In particular, the participants viewed andinteracted with the eNav intervention with a member of theresearch team present. The presence of the research staff, as
well as the overall study context, may have influenced
participants to respond in a favorable or positive way. In orderto mitigate social desirability bias, we encouraged candor andexplained that the team was specifically seeking negative
feedback and suggestions for change.

Conclusions
The iterative user-testing confirmed the overall usability andacceptability of the eNav website. Furthermore, qualitative andquantitative feedback from participants directly informed
modifications to the intervention. The RCT will formallyevaluate the efficacy of eNav for improving CRC screening
uptake among patients treated at FQHCs.

Acknowledgments
Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health underaward (R01CA248981). The content is solely the responsibility of the authors and does not necessarily represent the official
views of the National Institutes of Health.

Data Availability
The data sets generated and analyzed during this study are not publicly available to protect patient privacy.

Conflicts of Interest
AA is the scientific founder of Rx.Health, and holds the patent for the RxUniverse software platform that is a licensed technology
from the Icahn School of Medicine at Mount Sinai to Rx.Health, Inc (New York). SHI receives consulting fees and researchsupport from the Exact Sciences Corp, research support from Freenome, and consulting fees from Geneoscopy. AA and SHI
were not involved in data gathering or analysis.

Multimedia Appendix 1
Follow-up text messaging content and dosing for eNav intervention.
[PNG File , 419 KB-Multimedia Appendix 1]
https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Savage et alMultimedia Appendix 2
Screenshot of "Colorectal cancer" page on eNav website, including animated video.
[PNG File , 707 KB-Multimedia Appendix 2]

References
1.

2.

3.
4.
5.
6.
7.

8.

9.

10.
11.

12.

13.

14.

15.

16.

17.

18.

19.

Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN
estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71(3):209-249.
[FREE Full text] [doi: 10.3322/caac.21660] [Medline: 33538338]
US Preventive Services Task Force, Davidson KW, Barry MJ, Mangione CM, Cabana M, Caughey AB, et al. Screening
for colorectal cancer: US Preventive Services Task Force recommendation statement. JAMA. May 18,
2021;325(19):1965-1977. [doi: 10.1001/jama.2021.6238] [Medline: 34003218]
Society Ac. Colorectal cancer facts - figures 2023-2025. American Cancer Society. 2023. URL: https://www.cancer.org/
research/cancer-facts-statistics/colorectal-cancer-facts-figures.html [accessed 2024-08-09]
Colorectal cancer is a major public health problem. American Cancer Society. URL: https://nccrt.org/data-progress/ [accessed2024-08-09]
2021 Health center data. Administration HRaS. URL: https://data.hrsa.gov/tools/data-reporting/program-data/national/
table?tableName=Full&year=2021 [accessed 2024-08-09]
80% in every community. American Cancer Society. URL: https://nccrt.org/80-in-every-community-2/ [accessed 2024-08-09]
Prevention CfDCa. New Hampshire colorectal cancer screening program: patient navigation model for increasing colonsocopy
quality and completion. A Replication Manual. 2016. URL: https://ebccp.cancercontrol.cancer.gov/programDetails.
do?programId=27435365 [accessed 2024-08-09]
Christie J, Itzkowitz S, Lihau-Nkanza I, Castillo A, Redd W, Jandorf L. A randomized controlled trial using patient navigation
to increase colonoscopy screening among low-income minorities. J Natl Med Assoc. 2008;100(3):278-284. [doi:10.1016/s0027-9684(15)31240-2] [Medline: 18390020]
Green BB, Wang C, Anderson ML, Chubak J, Meenan RT, Vernon SW, et al. An automated intervention with stepped
increases in support to increase uptake of colorectal cancer screening: a randomized trial. Ann Intern Med. 2013;158(5 Pt
1):301-311. [FREE Full text] [doi: 10.7326/0003-4819-158-5-201303050-00002] [Medline: 23460053]
Jandorf L, Gutierrez Y, Lopez J, Christie J, Itzkowitz SH. Use of a patient navigator to increase colorectal cancer screening
in an urban neighborhood health clinic. J Urban Health. 2005;82(2):216-224. [doi: 10.1093/jurban/jti046]
Lasser KE, Murillo J, Lisboa S, Casimir AN, Valley-Shah L, Emmons KM, et al. Colorectal cancer screening among
ethnically diverse, low-income patients: a randomized controlled trial. Arch Intern Med. 2011;171(10):906-912. [doi:10.1001/archinternmed.2011.201] [Medline: 21606094]
Myers RE, Bittner-Fagan H, Daskalakis C, Sifri R, Vernon SW, Cocroft J, et al. A randomized controlled trial of a tailorednavigation and a standard intervention in colorectal cancer screening. Cancer Epidemiol Biomarkers Prev. 2013;22(1):109-117.
[FREE Full text] [doi: 10.1158/1055-9965.EPI-12-0701] [Medline: 23118143]
Percac-Lima S, Grant RW, Green AR, Ashburner JM, Gamba G, Oo S, et al. A culturally tailored navigator program forcolorectal cancer screening in a community health center: a randomized, controlled trial. J Gen Intern Med.
2009;24(2):211-217. [FREE Full text] [doi: 10.1007/s11606-008-0864-x] [Medline: 19067085]
Ritvo PG, Myers RE, Paszat LF, Tinmouth JM, McColeman J, Mitchell B, et al. Personal navigation increases colorectal
cancer screening uptake. Cancer Epidemiol Biomarkers Prev. 2015;24(3):506-511. [doi: 10.1158/1055-9965.EPI-14-0744][Medline: 25378365]
Jandorf L, Braschi C, Ernstoff E, Wong CR, Thelemaque L, Winkel G, et al. Culturally targeted patient navigation forincreasing african americans' adherence to screening colonoscopy: a randomized clinical trial. Cancer Epidemiol Biomarkers
Prev. 2013;22(9):1577-1587. [FREE Full text] [doi: 10.1158/1055-9965.EPI-12-1275] [Medline: 23753039]
Roland KB, Milliken EL, Rohan EA, DeGroff A, White S, Melillo S, et al. Use of community health workers and patientnavigators to improve cancer outcomes among patients served by federally qualified health centers: a systematic literature
review. Health Equity. 2017;1(1):61-76. [FREE Full text] [doi: 10.1089/heq.2017.0001] [Medline: 28905047]
Jandorf L, Stossel LM, Cooperman JL, Graff Zivin J, Ladabaum U, Hall D, et al. Cost analysis of a patient navigation
system to increase screening colonoscopy adherence among urban minorities. Cancer. 2013;119(3):612-620. [FREE Fulltext] [doi: 10.1002/cncr.27759] [Medline: 22833205]
Ladabaum U, Mannalithara A, Jandorf L, Itzkowitz SH. Cost-effectiveness of patient navigation to increase adherence withscreening colonoscopy among minority individuals. Cancer. 2015;121(7):1088-1097. [FREE Full text] [doi:10.1002/cncr.29162] [Medline: 25492455]
Thomas Craig KJ, Willis VC, Gruen D, Rhee K, Jackson GP. The burden of the digital environment: a systematic review
on organization-directed workplace interventions to mitigate physician burnout. J Am Med Inform Assoc. 2021;28(5):985-997.
[FREE Full text] [doi: 10.1093/jamia/ocaa301] [Medline: 33463680]

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
20.

21.

22.
23.

24.

25.
26.

27.

28.
29.
30.
31.
32.
33.

34.

35.

36.
37.
38.
39.

Savage et al

Huf SW, Asch DA, Volpp KG, Reitz C, Mehta SJ. Text messaging and opt-out mailed outreach in colorectal cancer
screening: a randomized clinical trial. J Gen Intern Med. 2021;36(7):1958-1964. [FREE Full text] [doi:10.1007/s11606-020-06415-8] [Medline: 33511567]
Solonowicz O, Stier M, Kim K, Kupfer S, Tapper E, Sengupta N. Digital navigation improves no-show rates and bowel
preparation quality for patients undergoing colonoscopy: a randomized controlled quality improvement study. J Clin
Gastroenterol. 2022;56(2):166-172. [FREE Full text] [doi: 10.1097/MCG.0000000000001497] [Medline: 33471486]
Trivedi PD, Geng E, Mehta D, Garg S, Rogers J, Kakkar S, et al. Sa650 digital navigation for colonoscopy: an automatedintervention to decrease colonoscopy no-shows. Gastroenterology. 2021;160(6):S-582. [doi: 10.1016/s0016-5085(21)02088-6]
Coronado GD, Rivelli JS, Fuoco MJ, Vollmer WM, Petrik AF, Keast E, et al. Effect of reminding patients to complete
fecal immunochemical testing: a comparative effectiveness study of automated and live approaches. J Gen Intern Med.
2018;33(1):72-78. [FREE Full text] [doi: 10.1007/s11606-017-4184-x] [Medline: 29019046]
Richardson-Parry A, Silva M, Valderas J, Donde S, Woodruff S, van Vugt J. Interactive or tailored digital interventions toincrease uptake in cervical, breast, and colorectal cancer screening to reduce health inequity: a systematic review. Eur J
Cancer Prev. 2023;32(4):396-409. [FREE Full text] [doi: 10.1097/CEJ.0000000000000796] [Medline: 37144585]
Champion VL, Skinner CS. The health belief model. In: Glanz K, Rimer BK, Viswanath K, editors. Health Behavior and
Health Education: Theory, Research, and Practice. Hoboken, New Jersey. Jossey-Bass; 2008:45-65.
Miller SJ, Sly JR, Alcaraz KI, Ashing K, Christy SM, Gonzalez B, et al. Equity and behavioral digital health interventions:strategies to improve benefit and reach. Transl Behav Med. 2023;13(6):400-405. [FREE Full text] [doi: 10.1093/tbm/ibad010][Medline: 36940409]
Inadomi JM, Vijan S, Janz NK, Fagerlin A, Thomas JP, Lin YV, et al. Adherence to colorectal cancer screening: a randomized
clinical trial of competing strategies. Arch Intern Med. 2012;172(7):575-582. [FREE Full text] [doi:10.1001/archinternmed.2012.332] [Medline: 22493463]
Alliance CC. Take the screening quiz. Colorectal Cancer Alliance. URL: https://quiz.getscreened.org/# [accessed 2024-09-08]
Coalition NYCCCC. Colon cancer prevention risk assessment and screening form. NYC Health. URL: https://www.nyc.gov/
assets/doh/downloads/pdf/cancer/colon-cancer-screening-form.pdf [accessed 2024-08-09]
Anderson M. Technology device ownership: 2015. Pew Research Center. 2015. URL: https://www.pewresearch.org/internet/
2015/10/29/technology-device-ownership-2015/ [accessed 2015-10-29]
Perrin A. Mobile technology and home broadband 2021. Pew Research Center. 2021. URL: https://www.pewresearch.org/
internet/2021/06/03/mobile-technology-and-home-broadband-2021/ [accessed 2015-10-29]
Brooke JB. A quick and dirty usability scale. In: Jordan PWT, Weerdmeester BA, McClelland AL, editors. Usability
Evaluation in Industry. London. Taylor and Francis; 1996.
Tariman JD, Berry DL, Halpenny B, Wolpin S, Schepp K. Validation and testing of the acceptability E-scale for web-based
patient-reported outcomes in cancer care. Appl Nurs Res. 2011;24(1):53-58. [FREE Full text] [doi:10.1016/j.apnr.2009.04.003] [Medline: 20974066]
Haun J, Luther S, Dodd V, Donaldson P. Measurement variation across health literacy assessments: implications forassessment selection in research and practice. J Health Commun. 2012;17 Suppl 3:141-159. [doi:10.1080/10810730.2012.712615] [Medline: 23030567]
Wallace LS, Rogers ES, Roskos SE, Holiday DB, Weiss BD. Brief report: screening items to identify patients with limited
health literacy skills. J Gen Intern Med. 2006;21(8):874-877. [FREE Full text] [doi: 10.1111/j.1525-1497.2006.00532.x][Medline: 16881950]
Maguire M, Delahunt B. Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars. All
Irel J High Educ. 2017;9(3).
Hyzy M, Bond R, Mulvenna M, Bai L, Dix A, Leigh S, et al. System usability scale benchmarking for digital health apps:
meta-analysis. JMIR Mhealth Uhealth. 2022;10(8):e37290. [FREE Full text] [doi: 10.2196/37290] [Medline: 35980732]
Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact.
2008;24(6):574-594. [doi: 10.1080/10447310802205776]
Vogels EA. Digital divide persists even as Americans with lower incomes make gains in tech adoption. Pew ResearchCenter. 2021. URL: https://www.pewresearch.org/short-reads/2021/06/22/
digital-divide-persists-even-as-americans-with-lower-incomes-make-gains-in-tech-adoption/ [accessed 2024-08-09]

Abbreviations
CRC: colorectal cancer
FIT: fecal immunochemical test
FIT-DNA: multitarget stool DNA test
FQHC: Federally Qualified Health Center
HBM: Health Belief Model
HIPAA: Health Insurance Portability and Accountability Act
RCT: randomized control trial
https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 14(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Savage et al

SUS: System Usability Scale

Edited by A Mavragani; submitted 03.10.23; peer-reviewed by W LaMendola, M Jimbo; comments to author 24.01.24; revised version
received 02.05.24; accepted 03.05.24; published 25.09.24
Please cite as:
Savage LC, Soto-Cossio LE, Minardi F, Beyrouty M, Schoonover J, Musella J, Frazier M, Villagra CN, Sly JR, Erblich J, Itzkowitz
SH, Jandorf LH, Calman NS, Atreja A, Miller SJ
The Development of a Digital Patient Navigation Tool to Increase Colorectal Cancer Screening Among Federally Qualified HealthCenter Patients: Acceptability and Usability Testing
JMIR Form Res 2024;8:e53224
URL: https://formative.jmir.org/2024/1/e53224
doi: 10.2196/53224
PMID:©Leah C Savage, Luz Estefhany Soto-Cossio, Francesca Minardi, Matthew Beyrouty, Julie Schoonover, Jay Musella, Michaela
Frazier, Cristina N Villagra, Jamilia R Sly, Joel Erblich, Steven H Itzkowitz, Lina H Jandorf, Neil S Calman, Ashish Atreja, SarahJ Miller. Originally published in JMIR Formative Research (https://formative.jmir.org), 25.09.2024. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR
Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on
https://formative.jmir.org, as well as this copyright and license information must be included.

https://formative.jmir.org/2024/1/e53224

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e53224 | p. 15(page number not for citation purposes)