Using Smart Devices to Implement an Evidence-based eHealth System for Older Adults
Phase: NA
Conditions: Chronic Pain
Sponsor: University of Wisconsin, Madison

AHRQ ET Smart System Protocol_7.20.2022 
 
 
 
 
Using Smart Devices to Implement an Evidence-based eHealth System for Older 
Adults 
 
NCT04798196 
 
Protocol document date: July 20, 2022 
 
 

 
2 
 
 
Using Smart Devices to Implement an Evidence-based eHealth System for Older Adults 
 
Principal Investigators: David H. Gustafson, Ph.D. 
 
University of Wisconsin - Madison 
 
Industrial Engineering 
 
1513 University Avenue, ME Bldg. Suite 4109 
 
Madison, WI 53706 
 
608-263-4882; Dhgustaf@wisc.edu 
 
Marie-Louise Mares, Ph.D. 
University of Wisconsin - Madison  
Communication Science 
6140 Vilas Hall 
 
Madison, WI 53706 
 
608-263-2350; mares@wisc.edu 
 
Co-Investigators:    
Jane Mahoney, Ph.D. 
 
University of Wisconsin - Madison 
 
School of Medicine and Public Health 
610 Walnut St, Room 957d  
Madison, WI 53726 
 
608-262-5077; jm2@medicine.wisc.edu 
 
 
Bilge Mutlu, PhD 
 
University of Wisconsin - Madison 
Computer Sciences 
6381 Computer Sciences 
1210 W Dayton Street 
Madison, WI 53706 
608-262-6635; bmutlu@wisc.edu 
 
Study Coordinators: 
Gina Landucci, BS 
Department: Industrial Engineering 
1513 University Avenue, ME Bldg. Room 4111 
Madison, WI 53706 
608-890-1440; gina.landucci@wisc.edu 
 
Klaren Pe-Romaskho, MS 
Department: Industrial Engineering 
1513 University Avenue, ME Bldg. Room 4155A 
Madison, WI 53706 
608-263-3322; kspe@wisc.edu 
  
Sponsor:  
DHHS, Agency for Health Care Research and Quality (AHRQ) 
 
 

 
3 
 
TABLE OF CONTENTS      
Study Summary .............................................................................................................................................. 3 
1.0  Background & Significance   .................................................................................................................. 3 
2.0    Study Objectives .................................................................................................................................... 6 
3.0    Selection of Subjects ........................................................................................................................... 10 
4.0  Methods .............................................................................................................................................. 11 
5.0    Measures and Data Collection ............................................................................................................ 14 
6.0    Statistical Considerations .................................................................................................................... 15 
7.0    Records to be Kept .............................................................................................................................. 16 
Appendix A: CHESS Data and Safety Monitoring Plan ................................................................................. 17 
8.0  References ............................................................................................................................................ 22 
 
 
 

 
4 
 
STUDY SUMMARY 
The primary purpose of this study is to investigate whether voice-activated "smart" technology increases 
adoption and sustains use of an evidence-based electronic health intervention (Elder Tree, or ET) for older adults 
with multiple chronic conditions, and thereby improves its potential to widely enhance quality of life and health 
outcomes. ET is an existing intervention providing tools, motivation, and support on a computer platform to help 
older adults manage their health. An AHRQ-funded randomized controlled trial (RCT) found that ET improved 
quality of life and other factors among high users of primary care with multiple chronic conditions such as 
diabetes and hypertension. However, many people did not use it extensively, which is a common problem with 
all web apps. 
 
ET is based on the extensively tested Comprehensive Health Enhancement Support System (CHESS).1-7 ET is a 
“walled garden” free of ads, with design features based on older users' feedback as well as best-practice 
principles such as uncluttered screens and large type.8 Like all CHESS systems, ET uses computers to deliver key 
elements of successful interventions: long duration, ongoing outreach, prompts, monitoring, cognitive 
reframing, action planning, problem solving, self-tailoring, and peer support.  
 
The central question is, can the unique accessibility characteristics of smart devices substantially increase and 
sustain the impact of ET among older adults? Toward that end we propose a pragmatic, 3-arm RCT comparing 
the laptop version (ET-LT) to a smart system version (ET-SS) to Control or Usual Care (UC) with no technology. 
The goal is to assess whether ET-SS is better at 1) sustaining use of ET and 2) improving long-term QOL, health, 
and health service use (reducing readmissions) among older patients with multiple chronic conditions.9  
 
1.0  
BACKGROUND & SIGNIFICANCE 
Multiple chronic conditions (MCCs) are costly and pervasive among older adults. MCCs account for 90% of 
Medicare spending, and 65% of Medicare beneficiaries have 3 or more chronic conditions; 23% have 5 or more. 
MCCs are often addressed in primary care, where time pressures force a focus on medication and lab results 
rather than self-management skills. Patients often struggle with treatment adherence and the emotional and 
physical burdens of self-management and health tracking.10 Chronic conditions reduce quality of life (QOL) and 
increase loneliness, which exacerbate those conditions.11-13  
 
ET laptop intervention. As reported in a paper under review, an earlier AHRQ-funded laptop version of ET 
significantly increased QOL. In a current NHLBI clinical trial, our team is testing a version of ET for patients with 
MCCs that adds a report to clinicians (CR) documenting changes in health status. Health-tracking data reported 
by patients to ET are sent to clinicians before patients’ appointments to help make those visits more effective 
and efficient. In a lung cancer RCT of CHESS,7 the CR reduced symptom distress by more than 50% over 12 
months (p<.001). We added the CR to ET because of these compelling results and are testing hypotheses that 
patients assigned to ET, vs. an attention control, will have better QOL, fewer symptoms, and better condition-
specific lab scores. The AHRQ-funded trial of our earlier ET found high and sustained engagement, relative to 
reports for other health apps.14-17In months 1-6, 89% of all participants used ET a mean of 44.84 days. However, 
in just 4 months, pages viewed dropped by 53%. Overcoming barriers to sustained, in-depth use is a critical 
challenge. Discussions with older adults found two main reasons why use dropped off: 1) The little hassles of 
going to and turning on the computer, remembering and entering a password, and opening a service created 
barriers to use. 2) Many older adults face barriers due to limitations such as arthritis and impaired vision. 
 
Enter the "smart system." Smart speakers are voice-activated devices connected to the Internet. Because they 
are used by talking and listening rather than typing and reading, many barriers associated with laptop use are 

 
5 
avoided. One can, for example, access a smart version of ET from their easy chair. They would not need to get 
up and go to the computer, type, and read.  And, smart systems provide easy access to other Internet services. 
 
Smart technology makes things convenient, as well as entertaining. Smart speakers are so new there has been 
little study of their general use, let alone health-related effects. That said, a 6-month pilot study of the Amazon 
Alexa with 18 older adults in a retirement community18found that 100% said Alexa made their lives easier and 
71% felt more connected to friends, family, and community. Uses included setting alarms and timers and 
listening to music, weather, news, and other information. This suggests older adults will prefer smart speakers. A 
small test of smart speakers with 12 veterans receiving rehabilitation services found that 100% reported the 
speaker "useful in their daily lives" and that they would be "very likely to continue using" it for a wide range of 
services, including relaxation, information, alerts and reminders, and even companionship.19 Recent media 
reports indicate a groundswell in the use of speakers among the elderly and people with vision loss, tremors, 
mobility problems, dementia, and more.20-24  
 
In fact, the acceptance of smart speakers has been nothing less than extraordinary. In the few years since its 
introduction, 90 million U.S. adults have placed one in their home—a faster adoption rate than TV or the 
Internet.25,26 Industry estimates suggest that by 2020 over 30% of Internet browsing will occur on such 
devices.27 Recently Epic, a leader in medical record software, announced that within 4 years data entry will be 
audio based. 
 
We believe smart devices are ready for dissemination for behavior change. Worth noting is a study conducted 
by the NPR and Edison Research group in which smart speaker owners reported behavior changes in use of 
radio (39% said they use radio less), TV (30%), and smartphones (34%). The study also shows that 83% of 
smart speaker users are using that device at least as much as in the first month they bought it, with 70% using 
it daily.28,29 This contrasts with dramatic reductions in the use of smartphone apps over time. 
 
What makes smart speakers so innovative is their ease of use. Patients can ask questions and get answers 
immediately spoken back to them. They can hear and respond to messages, request audio services such as 
guided meditation, record health data, and more, just by talking. In theory, these capacities also exist on 
smartphones, but speakers have been developed for natural interactivity, incorporating chatbots that mimic 
conversational turn-taking. Smart speakers can be used by patients whose limited vision or dexterity makes it 
difficult to use a mobile phone or laptop. Improvements in artificial intelligence and voice recognition hold the 
promise of overcoming these difficulties more than ever before, assuming the necessary data are available. 
 
Smart displays. Speakers do have limitations. Without a visual display, it can be hard (particularly for those with 
memory challenges) to recall information they have heard but not seen. A smart speaker could link users to a 
laptop when they want information conveyed visually, but this is inconvenient. Smart displays are specifically 
designed to address such issues, including screen magnification for those with partial vision. 
 
The Center for Health Enhancement Systems Studies (CHESS). CHESS is a research center at the University of 
Wisconsin–Madison. CHESS, or Comprehensive Health Enhancement Support System, is also the name of the 
platform we developed to help people cope with life’s challenges. We have conducted 16 RCTs of our eHealth 
CHESS systems and published more than 190 peer-reviewed works, as well as over 100 about organizational 
change and implementation engineering. In randomized trials, CHESS significantly improved: 1) childhood 
asthma control and parent coping1; 2) QOL and cost of care in HIV patients2; 3) QOL and self-efficacy in breast 
cancer patients, including elderly women3; 4) physical activity for survivors of colon cancer30; 5) risky drinking6; 
6) caregiver coping for patients with dementia31; and 7) caregiver burden, symptom distress, and median length 

 
6 
of survival in lung cancer patients.7  That experience bodes well for the success of this project. 
 
Preliminary data. Our AHRQ-funded Center of Excellence in Active Aging developed ET to help older adults 
remain independent. Our completed RCT involved 310 older adults from 3 Wisconsin counties (urban, rural, and 
suburban).32 Among high users of primary care, patients assigned to ET, compared to patients in the control 
group, experienced greater improvements in mental QOL (OR = 0.31, p = .006), social support provided (OR = 0.25, 
p = .002), social support received (OR = 0.16, p = .011), and depression (OR = -0.20, p = .037).  These findings suggest 
that ET improves health when used by patients with high primary care use. The question is how to maximize ET 
use. In our ongoing NHLBI clinical trial (1-R01 HL134146), we are examining the impact of a laptop-delivered ET 
on QOL, of health, and healthcare use for older patients with 3 or more chronic conditions. Now we are planning 
to compare it with ET delivered by a smart display. 
 
Pilot testing the smart speaker and smart display. We placed a Google Mini speaker in the homes of 11 
older adults for 5 weeks. At the end of the test, we interviewed the older adults to learn how they used and felt 
about the device. During that time, staff contacted them by phone to suggest something new to try (e.g. make a 
phone call, set up a reminder, use the Internet); no one reported trouble following through on the suggestion. 
All felt the Mini was easy to use, and 10 of the 11 "strongly agreed" it would improve older-adult quality of life. A 
common criticism of the speaker was the periodic difficulty of asking a question in a way the Mini would 
understand, a problem that can be addressed by a smart display showing how the question was understood. 
Also, reviewers felt it would help to review comments later made by the technology or read them to confirm 
what was heard, both of which are possible with the display. 
 
In a separate pilot, 10 older adults were given an opportunity to observe and interact with the Lenovo smart 
display. All had used the Google Mini. All felt the display was a significant advance over the smart speaker and 
that they would find the display progressively more valuable as they aged. In particular, they noted things a 
speaker can’t do, such as allowing the user to dig more deeply into a topic with lists of options and prompts 
rather than a single response. For example, if users were to ask a smart speaker about their high school, they 
might get a 10-second audio response, but with the display they would get that response plus options to see a 
picture of the school, learn about graduating class size, and more. Participants liked that the display could show 
printable answers, eliminating the need to take notes, and that they could go back to repeat information. In 
general, they felt having a visual component made many smart features easier to understand and follow. All felt 
the display was easier to use than a laptop computer because of its ease of access. 
 
Innovation. ET-SS is highly innovative for the following reasons.  
• One location -  wide access. ET currently operates on a laptop because many older adults have difficulty 
using smartphones. Although portable, laptops tend to sit in one place for older adults. For frail older 
adults, even walking to the computer can be a challenge. With our proposed system, smart devices will be 
placed in multiple areas, giving easy, low-cost communication with ET without needing to walk to or carry 
the device. 
• Tactile input -  two-way voice activation. Most eHealth apps, including ET’s laptop version, need patients to 
see content on a screen and respond by typing or touching options on-screen. Physical limitations, 
common with elderly (vision, motor control, hand pain), can make this difficult. Smart speakers provide bi-
directional, voice-activated interactions. Patients ask out loud, listen to answers, and follow up verbally. 
• Multi-step activation - one-step activation. Laptops require multiple steps to engage a desired feature. 
With a smart speaker, a patient can simply say, "Hey ET, I'll answer my health survey now." Voice 
authentication replaces a laptop’s burdensome activation, login, and even site navigation. 
 

 
7 
• Medical focus - whole-person focus. All chronic diseases share certain features. Patients feel frustrated, 
discouraged, and lonely. They face continual, ongoing ups and downs. NIH funded us to develop one 
program that would address these common needs shared by all patients with MCCs. ET's focus is therefore 
the whole person, not the management of any particular disease. In our completed trial, ET improved QOL 
among those with high levels of primary care use across a wide range of conditions. Older adults in that 
RCT reported that ET increased happiness by being fun, not just utilitarian. In our pilot studies, all were 
enthusiastic about the smart system's potential, not only for health tasks (e.g. medication reminders) but 
also for music and social interaction. In a national survey of more than 1800 adult users,32 top requests 
included jokes, games, and music, underscoring the expectation of pleasure. We predict the smart 
system's ease of use will increase sustained use of ET's content, prompt more overall use and higher 
quality of life. 
• Groundbreaking. This will be the first large RCT of a smart system that is likely to take the health field by 
storm. The knowledge gained could significantly improve future designs and implementations of smart 
devices. 
 
2.0   
STUDY OBJECTIVES 
Overview of methods. We propose a 3-arm RCT to examine whether delivering ET on the smart system (ET- SS) 
will increase frequency and sustainability of use and improve quality of life, when compared to ET on the laptop 
(ET-LT).  A Control group (ET-C), which will receive no technology (smart system or laptop) or ET, will allow us to 
assess whether both groups that receive the ET intervention (albeit on different platforms) show equally 
positive effects relative to control, or whether ET delivered via one platform is more efficacious relative to the 
control and other intervention group.  We will create ET-specific commands so that the smart system operates 
well with ET (e.g. “Hey Google, I'm ready to take my survey”). The RCT itself will begin by identifying eligible 
patients and sending them invitation letters from their primary care provider (PCP). Patients who respond will 
be screened and recruited into the study, baseline data will be collected, and patients will be randomly 
assigned to a study arm, provided with devices if applicable, and trained. The intervention will last 8 months. 
Data will be collected at 0, 4, and 8 months. The final months of the study will be spent analyzing and 
interpreting qualitative and quantitative data and disseminating results through publications, presentations, 
and real-world field tests. 
 
Specific Aim 1: Enhance the smart system platform to deliver ET. 
Smart system technology. Users activate a smart system with a wake phrase (e.g. “Hey, Elder Tree"). Voice 
authentication gives individuals exclusive access to their content. All smart systems have security measures to 
prevent wiretapping, and data transmitted to data centers (e.g. Google) are encrypted. ET-related requests will 
be saved, allowing us to measure smart system use. 
 
ET description. ET provides tools, motivation, and social support to help patients manage chronic pain and other 
chronic conditions. ET is consistent with Self-Determination Theory, which asserts that satisfying 3 basic 
psychological needs contributes to adaptive functioning: competence (feeling effective, not overwhelmed), 
social relatedness (feeling connected to others, not isolated), and intrinsic motivation (feeling autonomous, not 
coerced). 
 
Competence services include: 
• Pain learning modules: interactive lessons that will be available to participants weekly and will focus on 
understanding chronic pain, learning coping skills, and suggested action steps. 
• Easing distress: Tools such as cognitive reframing,33 self-calming with games, guided progressive relaxation 
recordings, deep breathing, and mindfulness meditations. 

 
8 
• Reminders: Schedule reminders (e.g. medications, appointments, blood pressure readings). Prompts will 
automatically ask patients if they intend to carry out the action, a question shown to improve adherence.34  
• Tracking: Weekly survey of health-tracking questions (e.g. sleep, mood, falls). Patients can view trend graphs 
over time. 
• Health Library: High-quality articles and videos on topics such as exercise, pain, sleep, anxiety, falls and 
balance, grief and loss, nutrition, and talking with your doctor. Vetted by Dr. Mahoney. 
 
Social relatedness services include: 
• Regularly scheduled social “Meetups” with research staff and guest speakers via Google Meet. 
• Community Discussion Boards: Monitored, online, anonymous discussion groups for support and 
engagement. Research consistently indicates that participating in social interactions on CHESS predicts 
positive outcomes, including quality of life.35,36  
• Private Messages: Email-like function for ET members to communicate privately with one another. 
 
Intrinsic motivation services include: 
• Thought of the Day: Daily motivational prompts to boost happiness and resilience. 
• Personal stories: Audio/video of patients’ advice/experiences (e.g. what to avoid, tips for coping). 
• Journal: Daily writing prompts based on positive psychology.37  
 
Clinician report (CR): 
• The CR displays data from patients about general health variables relevant to all chronic conditions but rarely 
collected in clinical medicine (e.g. sleep, relationship quality, falls, missed medication). The variables were 
determined by practitioners in geriatrics and family medicine in the UW Health system. Data are displayed in 
at-a-glance graphics based on statistical process control,38 allowing a clinician to identify problem areas and 
successes worth discussing.39 When coupled with traditional clinical data (e.g. blood pressure), the CR adds an 
important dimension to the data set. The CR summary of trends is sent prior to appointments, facilitating 
more effective visits. Our lung cancer test of the CR found that issues were addressed more quickly, reducing 
the need for subsequent intensive clinician involvement.40  
 
Design considerations.  
How ET-SS will protect privacy. We created several ways to protect the privacy of older adults and 
families and to alert when our system is recording. We will help users understand: 
• 
When recording is happening, and when it is off. ET-SS will not record during initial interactions or after 
users have indicated they are done. We will add a highly visible indicator light that turns on during audio 
recording. Users can turn off our program at any time. These features have been well received during 
past studies. Users are comfortable with the recording process, the indicator light, the transparency of 
our procedures, and the ability to control when recording is happening. 
• 
How recorded data will be used. Audio recordings will be copied and stored on a secure server that only 
the research team can access. Researchers may use the speech data in publications only if all identifiable 
information has been removed and will quote users only with their permission. Audio recordings not 
related to the study will be deleted. Participants can review what was recorded and consent to whether 
the recording can be used in research. These statements require additional initials on the consent form. 
• 
What happens if someone leaves the recording on? ET-SS will automatically shut down due to inactivity. 
After 3 minutes, it asks if the user is “still there.” If no response within 30 seconds, the audio recorder is 
disabled. Users can pause audio recording by pressing a dedicated “pause” button. 
 
Potential negative impacts and how are they addressed. A potential negative is that older adults may 
disengage because the novelty has worn off. ET-SS will use two designs to promote long-term interaction: 1) 

 
9 
using information from past interactions during current dialogue41 and 2) revealing a personality.42 To do so, it 
will keep track of previous sessions. Each session starts with ET-SS reminding users of their previous activity. 
Also, over time, the system will reveal details of its own fictional personality during reading sessions (e.g. its 
favorite color is blue). These personal comments are designed to help build rapport during human–computer 
interactions.43 Some older adults are ambivalent about artificial characters building an emotional connection. 
There is some research on how social robots should deliver on their social interaction potential.44,45 We are 
exploring ways in which ET-SS could serve as a facilitator to engage family and friends.46,47  
 
Design decisions already made to limit frustration with technical problems during long-term in-home use.  
1) ET-SS is modeled after typical human behaviors that can help older adults interact with the system in socially 
positive ways.46,48 2) We will closely monitor and act on problems with ET-SS. This monitoring will include 
regular email check-ins with users to conduct routine maintenance checks. We will also coach users on simple 
procedures for resetting the system as needed. 3) We will include proven ways to mitigate the impact of 
breakdowns49 including recovery strategies and expectation setting. 
 
Dealing with ways ET-SS might interfere with older adults' lives. Co-designing with older adults will help us 
anticipate problems and design preventive solutions. Current designs include features that prevent interference 
with daily life, e.g. an automated option to avoid quiet time disturbances.43,50 To ensure that privacy is 
respected, we provide, as noted above, a clear indication of when the device is recording and the opportunity 
to stop it at any time. We also use information about the user to personalize interactions but only with their 
knowledge and consent.51 Concerns have been raised about using a device that is always listening. While 
microphones are always on, smart systems do not retain data until the wake word is used. Recordings will be 
stored on a distant server. We will periodically (or when requested) offer to delete those data.52  
 
Ensuring patient safety. Mental health professionals are not allowed to break confidentiality except when 
there is imminent danger. In non-emergency situations patients will be encouraged to call their doctor or a 
trusted colleague. If we find that a patient is facing an emergency, we will inform the primary care team, 
following the treatment program’s standard operating procedures for serious clinical situations. 
 
Specific Aim 2: Conduct a randomized clinical trial to test the following hypotheses. 
Figure 1 describes the 3-arm RCT. Baseline covariates include race/ethnicity, education, baseline outcomes, life 
stresses, health service use, and technology use. 
1. Primary outcomes: For 
those in ET-SS (vs. ET-LT vs. 
Control): pain interference 
will decrease and psychosocial 
quality of life will increase. 
2. Secondary outcomes: ET-SS 
(vs. ET-LT vs. Control) will 
increase ET use at 8 months, 
increase physical quality of 
life, well-being, and 
communication with doctor. 
Pain intensity, 30-day hospital 
readmission rates, health 
distress, loneliness, and 

 
10
irritability will decrease. We also plan to compare ET-LT to the UC control, given the unique focus of chronic pain 
in this study. 
3. Mediators: Self-Determination Theory constructs of competence, social relatedness, and intrinsic motivation, 
together with exercise and pain catastrophizing, will mediate the effects of study arms on outcomes. 
Additionally, use of ET at 4 months will mediate effects of study arms (ET-LT vs. ET-SS) on 8-month ET use. 
4. Moderators: ET-SS (vs. ET-LT vs. Control) will show greater improvements for a) women (vs. men), b) patients 
with more (vs. fewer) chronic conditions, and c) patients with fewer barriers to using technology for the primary 
and secondary outcomes. 
5. Exploratory analyses: We will measure level of use of each ET service, how it differs between the two ET arms, 
and perceptions of Elder Tree and the Clinician Report. We will also measure alcohol use, smoking, and falls. 
 
ET implementation and operation.  
Our Center developed the NIATx change model, which has been used by more than 3,000 behavioral health 
treatment agencies.53 NIATx led to our extensive research on models to predict and explain implementation 
success,54 ingredients of successful change,55 and diffusion of innovations.56 We will use this research to guide 
the implementation and operation of this study. 
 
Initiate. Physician change leaders and clinic administrators have already been appointed in the 6 UW clinics we 
propose to use: 20 S. Park, 884 eligible patients; Yahara, 602 eligible patients; Odana, 485 eligible patients; 
Northeast,181 eligible patients; UW East, 376 eligible patients; Sun Prairie, 117 eligible patients. We have 
active collaborations with those clinics. In fact, we recently completed recruitment of 346 subjects from UW 
clinics to an RCT four months ahead of schedule. The clinic administrators are familiar with our research team 
and UW Health clinic operations and have a strong relationship with clinical staff. Our research staff has 
engaged clinic staff to understand the clinic flow and refine our implementation process. We will regularly 
follow-up with clinic leaders, using the 15-item Organizational Change Manager,57 to measure and monitor 
implementation fidelity. 
 
Prepare the environment. Clinic staff and patients will be asked to advise on SS design (already done for the 
LT) and implementation. We have personally experienced what it is like to be a patient at these clinics. This 
“walk- through”58 helps us tailor LT and SS operations to each clinic. Clinic staff will specify how operations will 
be handled to minimize frustration (e.g. in a given clinic we may decide to deliver the CR by fax, not email, 3 
days before a scheduled patient visit). Clinic staff will help finalize processes for patient recruitment, data 
collection, and feedback about the study. We are experienced in community-based participatory research, 
most recently via our Center of Excellence in Active Aging, where we engaged 53 of 72 Wisconsin counties and 
hundreds of volunteers to implement the first version of ET.32 We will use the British NHS Sustainability Index 
(which we developed) at baseline and every 4 months to identify and reduce barriers to sustainability.59  
 
Refine systems. In the first 12 months, our team of clinicians, engineers, and scientists will make design 
modifications to integrate the smart display and associated smart speaker. Our information technology team is 
already addressing those issues, building on our previous experience with digital assistants and wireless 
devices. Dr. Mahoney will guide creation of symptom management tips, health trackers, and self-management 
materials. Some of the data collected by our technologies will be used for just-in-time tailoring of resources to 
participants., including 1) modifications to data entered at setup, 2) health behaviors, and 3) health status data. 
 
Make ET easy to use. We'll do this by 1) designing interactions with the SS so they require no training (e.g. 
assessments/alerts) and 2) continuing our commitment to a simple, intuitive interface for ET. 
 

 
11
Initiatives to sustain ET use. 1) Monitors (lay people selected for communication skills and empathy) will 
keep ET appealing by removing inappropriate posts and calming any conflict that may occur. Our project 
director will be the lead monitor, reviewing use data and sending encouraging notes. 2) Ice breakers will 
encourage use, e.g., by asking people to find common points in their background (“Where were you on the 
day of the first moon landing?”). 3) Refreshers will encourage patients to rate, comment on, and suggest ways 
to improve ET. 4) Under guidance of Dr. Mahoney, we will update ET as new information about relevant 
conditions becomes available. 5) We avoid using unsustainable financial incentives. 
 
Training and support. To support adoption of ET, we will adapt the experience of Front Porch Communities and 
Services in their 2017 test of Amazon Echo.18Our adaptation will include bimonthly phone-based workshops to 
cover use of devices, answer questions, and address technical support issues. The workshops will differ by 
whether the participant has ET-LT or ET-SS. Family members will be encouraged to join in to build enthusiasm. 
Topics include: 1) overview and setting up ET, and using our help line; 2) ET features; 3) open 
call-in times to answer questions; 4) linking to high quality resources outside of ET via the Internet; 5) accessing 
audiobooks; 6) trying a new ET feature; 7) setting reminders; 8) discussion groups and weekly health-tracking 
questionnaires (for the CR); and 9) making calls and connecting with friends. While there will be some overlap 
on topics, the Front Porch participants welcomed that duplication. We will also check in with participants 
during the first 3 weeks and send out a monthly newsletter to participants. 
 
3.0  
SELECTION OF SUBJECTS 
Patient eligibility/inclusion criteria. Eligible patients will: 
1. Be ≥60 years old;  
2. Have been treated in UW Health clinics in the last 12 months, with no plans to leave during the study period 
(only relevant for UW Health patients, not patients recruited from the community); 3. Have a chronic pain 
diagnosis AND a medical diagnoses of three or more of the following common chronic conditions: COPD, 
asthma, diabetes, hyperlipidemia, hypertension, ischemic heart disease, atrial fibrillation, heart failure, stroke, 
BMI 30+, cancer, chronic kidney disease, depression, osteoporosis, arthritis, or dizziness/falls/loss of vestibular 
function;  
4. Be willing to share healthcare use (e.g. 30-day readmissions) in EHRs (only relevant for UW Health patients, 
not patients recruited from the community); and 
5. Allow researchers to share information about a patient’s health status with their PCP. 
 
Patient exclusion criteria. Patients are not eligible if they: 
1. Require an interpreter 
2. Have a medical diagnosis of any of the following: 
• Alzheimer’s 
• Schizophrenia/other psychotic disorder 
• Dementia 
• Autism spectrum disorder 
• Known terminal illness with less than 6 months to live 
• Acute medical problem requiring immediate hospitalization 
 
Recruitment process.  
UW Health Family Medicine and General Internal Medicine Clinics: UW's Clinical and Health Informatics Institute 
(CHI2) will use clinic records to identify patients meeting the eligibility criteria described above and send eligible 
patient data (including first and last name, address, birth date, age, UW clinic location, and primary care doctor) 
to the UW Office of Clinical Trials (OCT) via REDCap. Potential participants will receive an opt-in letter describing 

 
12
the study and consent form from the OCT, plus a stamped return letter inviting contact from the study team.  
 
Community Recruitment: We will supplement recruitment at UW Health with community efforts to increase the 
racial diversity of our patient population. Through collaborations with the African American Health Network 
(AAHN), Rebalanced-Life Wellness Association, African American Opioid Coalition (AAOC), and the Community-
Academic Aging Research Network (CAARN) we plan to reach out to black churches, community centers, 
fraternity and sorority health service programs, and other organizations in the Madison and Beloit areas. We will 
distribute a recruitment flyer and consent with a return card/opt-in and conduct Zoom and in-person community 
sessions when possible to introduce the study and invite participation.  
 
Access Community Health Centers (ACHC): In addition to our community recruitment efforts, working with 
ACHC, will help us reach a larger number African Americans and other participants of color, as well as expand 
our reach to underserved patients. We will work directly with ACHC staff to disseminate the recruitment flyer, 
consent, and return card/opt-in to potentially eligible patients.  
 
Regardless of recruitment method, when a return card/opt-in is received, study staff will call and assess 
eligibility; provide a study overview that includes potential benefits and risks, study procedures, and 
compensation; thoroughly walk through informed consent; and address questions. The baseline survey will be 
mailed. Patients will be given adequate time to make the decision whether or not to participate and in some 
cases a follow up phone call will be scheduled after the patient has time to review the survey and consent form. 
 
Once a patient verbally confirms they want to participate, a home visit will be scheduled to collect the baseline 
survey, randomize, set up equipment, and train. Note that all study staff that conduct home visits will be fully 
vaccinated for Covid-19, will wear face masks, and socially distance as much as possible while in the patient’s 
home.  If no visit is desired, once the completed baseline is received via mail, equipment will be shipped, and 
technology setup and training will be conducted by phone. The baseline survey should take 20-30 minutes. 
Measures will be the same in all three arms to avoid differential drop-out. We will document those who choose 
not to participate and why, following CONSORT standards. 
 
4.0 
METHODS  
Development Phase Usability Testing 
The development phase will involve 16 adults age 60 or over with little to no experience using a smart system 
who are willing to participate in a 10-day usability test of the smart system. Feedback from the usability testing 
will help inform final development and refinements of the system before the RCT begins. Specific goals include:  
 
• How to make system set up easy for the user 
• How new users learn to use the system 
• What may go wrong in the users’ interaction with the system 
• Overall perceptions of ElderTree features (e.g. are they helpful, easy to use) 
• How “politeness” prompts in the system affect use (e.g. “It’s time to do the survey” or “Let’s do 
something together”) 
 
Usability subjects will be recruited via word of mouth by our research team and community partners – research 
staff will reach out to past study participants that indicated interest in further opportunities, the CAARN group led 
by Co-I Dr Jane Mahoney will spread the word through their networks – they will provide basic info (can text or 

 
13
email info sheet and consent with permission) and invite those interested to call the study team. A consent and 
info sheet will be mailed, texted, or emailed (with subject permission) and a second phone call will be scheduled 
to go through informed consent.  
 
Usability subjects will be randomized to receive: 
1. ET-SS: Subjects will receive a smart system, internet hotspot, and access to ElderTree 
2. ET-SS + “Politeness” Prompts: Subject will receive a smart system, internet hotspot, and access to ElderTree 
with “politeness prompts designed to help guide user interactions. 
 
Regardless of group, subjects participating in the usability testing will be asked to: 
• Complete an intake visit or phone call. If in person, research staff will deliver and set up the smart system, 
train you how to use it, and explain the “tasks” you will be asked to complete during the testing period. If via 
phone, the smart system will be mailed to you and research staff will talk you through set up and training 
over the phone. The intake visit or phone call will take about an hour and will be audio-recorded. The 
recording will then be transcribed, removing all personal identifying information (e.g. name) and the 
recording will be destroyed. 
• Complete recommended “tasks” and engage with ElderTree on the smart system during the 10-day testing 
period. Tasks include completing the ElderTree weekly survey, posting a message in the discussion group, 
writing a journal entry, and viewing a lesson on chronic pain. 
• 
Allow the research team to collect information on how you use ElderTree. This will be collected 
automatically on the smart system and will include what services you used, when, and for how long, what 
prompts you say to trigger the services, and any system failures. 
• Complete survey questions and an exit interview with research staff over the phone at the end of the testing 
period to collect your feedback on the system. The survey questions and exit interview will take about an 
hour and will be audio-taped for transcription as described above. 
 
Potential Risks are similar to those for the RCT and include: 
• There could be a breach of confidentiality that could result in disclosure of research data outside of the 
study team. To prevent this, only limited research staff will have access to subject names, addresses, 
phone numbers, and other identifiable information. All data will be stored securely and de-identified for 
analysis. Transcripts from the audio-recordings will not contain identifiable information and the 
recordings will be destroyed. 
• The potential for subjects to receive and act on bad information or misinterpret accurate information. To 
mitigate the risk of providing inaccurate or harmful information to patients, all ET content will be 
screened by Dr. Mahoney. ET information will be accessible by individuals who have low literacy skills to 
reduce the risk of misinterpretation. Additionally, messages exchanged within the ET discussion groups 
will be monitored to make sure the information is accurate and that study participants are using the 
system for its intended purpose. 
• During study activities we may discover behavior that raises concern about elder abuse, harm to self or 
others. If we see anything that suggests that patient subjects or others face imminent risk of harm, we 
will contact appropriate others to intervene (e.g., community mental health resources and/or police). 
There is no cost to usability subjects to participate. If subjects complete the 10-day testing period and the final 
survey and exit interview, they will be paid $25.  

 
14
 
Research staff will collect the smart system and internet hotspot at the end of the 10-day testing period. 
 
 
RCT Patient Recruitment 
The RCT will involve 291 patients age 60 and older, recruited over 18 months, and randomized into three groups: 
1. ET-SS: Patients will receive treatment as usual (TAU) + Smart System + Internet access + ElderTree 
2. ET-LT: Patients will receive TAU + laptop + Internet access + ElderTree 
3. Control: Patient will receive TAU 
 
Patient Screening 
For patients recruited via UW Health: Research staff will call UW Health patients that return the opt-in letter and 
ask the following screening questions to confirm eligibility: 
Vision, hearing, 
motor impairment 
Do you have a vision, hearing, or motor impairment that would 
make it hard for you to use a computer or a voice-activated 
system? 
 
“No” = eligible 
UW Health 
Do you have any plans to change your healthcare from UW Health 
to another provider in the next 8 months?  
 
“No” = eligible 
Pain DURATION 
Have you had pain on some or most days, that has lasted at least 
12 weeks? 
 
 
“Yes” = eligible 
Pain FREQUENCY 
In the past 3 months, how often did you have pain? 
Never    Some days   Most days 
Every day 
 
Never = not eligible 
Pain INTENSITY 
In the past 7 days how would you rate your pain on average? 
0 = No pain – 10 = Worst imaginable pain 
 
3 - 10 = eligible 
 
For patients recruited via the community: Research staff will call patients that return the community flyer return 
card/opt-in and ask the following screening questions to confirm eligibility: 
 
Assess eligibility via self-report – inclusion criteria: 
1. Are you ≥60 years old? 
2. Have you had a chronic pain diagnosis in the last 6 months AND a medical diagnoses of three or more of the 
following common chronic conditions: COPD, asthma, diabetes, hyperlipidemia, hypertension, ischemic heart 
disease, atrial fibrillation, heart failure, stroke, BMI 30+, cancer, chronic kidney disease, depression, 
osteoporosis, arthritis, or dizziness/falls/loss of vestibular function;  
3. Would you be willing to allow us to share information about your health status with your PCP? 
 
Patient exclusion criteria.  
4. Do you require an interpreter? 
5. Do you have a medical diagnosis of any of the following: Alzheimer’s, Schizophrenia/other psychotic disorder, 
Dementia, Autism spectrum disorder, Known terminal illness with less than 6 months to live, Acute medical 
problem requiring immediate hospitalization? 
 
Additional Screening Questions: 

 
15
Vision, hearing, 
motor impairment 
Do you have a vision, hearing or motor impairment that would 
make it hard for you to use a computer or a voice-activated 
system? 
 
“No” = eligible 
Pain DURATION 
Have you had pain on some or most days, that has lasted at least 
12 weeks? 
 
 
“Yes” = eligible 
Pain FREQUENCY 
In the past 3 months, how often did you have pain? 
Never    Some days   Most days 
Every day 
 
Never = not eligible 
Pain INTENSITY 
In the past 7 days how would you rate your pain on average? 
0 = No pain – 10 = Worst imaginable pain 
 
3 - 10 = eligible 
 
 
Patient Consent  
Patients must agree to allow monitoring of their ElderTree use and for data to be collected from their EHR (UW 
Health patients only), as detailed below. The patient will be given the right not to share specific information and 
will retain the right to revoke their permission at any point. 
The consent process will inform potential subjects of:  
(1) the nature and purpose of the study 
(2) the types of data that will be collected from the EHR (by the CHI2 – UW Health patients only)  
• Chronic conditions 
• health care utilization: hospitalizations (admission and discharge), number of ER visits, urgent care visits, PC visits, 
specialty care visits, long-term care visits, and rehab visits 
(3) the types of data that will be collected from ElderTree  
• Weekly survey data (missed medications, falls, thinking and memory, mood, healthy meals/snacks/drinks, 
physical activity, quality time with others, sleep, pain, and balance) 
• Use data 
(4) the types of data that will be shared with their primary care physician - a graph that charts weekly scores 
related to the quality of sleep, nutrition, physical activity, memory, falls, moods, balance, pain, medication 
adherence and quality time spent with others- for UW Health patients only. 
(5) study risks and measures taken to mitigate 
(6) their right to leave the study at any time 
(7) the timeline of the study 
Verbal consent will be documented on the recruitment form and in REDCap and a copy of the IRB-approved 
consent form will be mailed to participants for their records. All study data outside of REDCap will be stored in a 
locked cabinet at the Center for Health Enhancement Systems Studies at the UW.  
The research team will always have a member available during standard operating hours for participants to 
contact with questions or issues. If a patient declines to participate, a research team member will determine 
reasons for non-participation so we can examine how patients who opt out differ from those who choose to 
participate.  

 
16
 
Patient Randomization  
Once informed consent and baseline measures have been obtained from patients, they will be randomized to ET 
+ SS, ET + LT, or the Control group, stratified on gender, site (UW clinic vs community), and number of chronic 
conditions (4-5 vs. 6+).  
 
Setup and Training. Researchers will introduce patients in each arm to their technology and place the laptop or 
smart display in the preferred location. After explaining the implications, the smart display will go where 
patients least want a camera. Patients will play a 5-minute introductory video/audio recording that explains ET-
LT or ET-SS. During setup, information is entered to tailor ET, such as medications, contacts, and activities of 
interest. During training, all patients add a post to the discussion group. A week later, they are encouraged to 
play the intro again and select another ET service to learn. Thus, patients can gradually increase the services they 
use. If necessary, UW staff will text or email patients to encourage them and resolve issues. Periodically, 
thereafter, a brief intervention will feature a different service, based on past use (“we see that you have not 
used the XXX service. We believe it is worth a try because…”). 
 
Costs to participants = $0. Patients in the ET-LT and ET-SS groups will get technology (laptop or smart system 
respectively), and Internet service (hot spot) for 8 months, and will be paid $10 to complete each of the three 
surveys ($30 total). Patients in the Control group will be paid $30 to complete each of the three surveys ($90 
total). 
 
Privacy and Confidentiality 
To mitigate the risk of patient breaches of confidentiality, all subjects will be assigned a unique code number. 
Subject code numbers and all study data will be maintained by the UW project director and stored in UW ICTR 
Research Electronic Data Capture (REDCap). Members of the research team will be able to view de-identified 
individual and clinic-level aggregations of variables. When all study activities are complete identifiable 
information will be destroyed. De-identified study data will be stored on the secure servers for potential 
future unspecified research for which new IRB submissions will be initiated. 
 
Potential Risks 
The principal risks to participants include:  
• The potential for subjects to receive and act on bad information or misinterpret accurate information. To 
mitigate the risk of providing inaccurate or harmful information to patients, all ET content will be 
screened by Dr. Mahoney. ET information will be accessible by individuals who have low literacy skills to 
reduce the risk of misinterpretation. Additionally, messages exchanged within the ET discussion groups 
will be monitored to make sure the information is accurate and that study participants are using the 
system for its intended purpose. 
• There is a risk that information provided in ET will be used to the detriment of the subjects. All 
participants will receive information and guidelines on Internet safety and security during initial training 
in both the ET-LT and ET-SS group.  
• If participants engage in the social components of ET (e.g. discussion group, live Meetups), they could get 
upset with opinions or comments from other participants. Research staff monitor discussions daily and 
facilitate live meet-ups. Appropriate action will be taken if questionable posts/comments occur. 
Participants may choose not to participate in social components on ElderTree if they wish. 
 

 
17
• There could be a breach of confidentiality that could result in disclosure of research data outside of the 
study team. To prevent this, all subjects will be assigned a blind code number. Data collected from clinic 
records will have the name removed and a code number attached. Project staff who have access to the 
data will not have access to subject names. ET will automatically collect data on how often and for how 
long participants use the specific services within ET. This information will be collected by subjects’ ET 
code number only and will not be attached to real names or identities. 
• Internet service will be stopped after 8 months. Subjects may feel some loss when they no longer have 
this service. Participants will receive notice in advance of internet termination, but will be allowed to 
keep their equipment in case they acquire internet service on their own. 
• On surveys participants will be asked personal questions related to past or current behaviors and 
experiences, such as struggles with health issues, depression, etc. that could produce anxiety, distress, 
embarrassment, or feelings of sadness. Participants can opt not to answer certain questions if they 
choose. 
• During study activities we may discover behavior that raises concern about elder abuse, harm to self or 
others. If we see anything that suggests that patient subjects or others face imminent risk of harm, we 
will contact appropriate others to intervene (e.g., State authorities, clinic, and/or police). 
Unanticipated Events  
Should any unanticipated problems arise the UW research team will work with the Health Sciences IRB and 
the ICTR Data Monitoring Committee to immediately address the problem (see Appendix A for detailed Data 
and Safety Monitoring Plan). 
Inappropriate use of ElderTree 
It is anticipated that ElderTree could be used inappropriately. A research staff member will review and delete 
any messages deemed inappropriate (i.e nudity, threats, racism, bigotry). A research staff member will then 
follow up with the author of the inappropriate content. Discussion group guidelines have been developed and 
will be reviewed with participants as part of training.  
Reasons for removing a subject from study 
The PIs have discretion to remove a subject from the study for any reason that is in the best interest of the 
individual as well as others on the study. Possible reasons for removal may include: 
• Inappropriate use of ElderTree as discussed above. 
• Repeated loss or destruction of laptop or smart system. 
• Inability to use the technology. In some cases, despite repeated technical support, the use of the laptop 
or smart system by the participant may not be possible and has the potential to cause distress. In these 
cases, the technology will be removed and the participant offered the option of completing the mailed 
surveys only for the duration of the study. 
 
 
 
 
 
 
 
 
 
 

 
18
 
5.0      MEASURES AND DATA COLLECTION 
Participants will be tracked for a total 
of 8 months. Data will be collected 
from: electronic health records (EHR), 
participant surveys (at months 0, 4, and 
8 months), qualitative interviews, and 
ElderTree use data. Table 2 lists 
quantitative and qualitative measures, 
data sources, psychometrics, and 
references to validation studies. 
 
Quantitative. Surveys conducted at 
months 0, 4, and 8 are expected to take 
20-30 minutes and will assess 
psychosocial and physical QOL, pain, 
30-day readmits, health distress, well-
being, loneliness, communication with 
doctors, irritability, alcohol use, 
smoking, falls, and hypothesized 
mediators of competence, relatedness, 
motivation, exercise, and pain 
catastrophizing. Surveys will be mailed 
to patients with a self-addressed 
stamped envelope. Patients will be 
phoned if they do not respond after 2 
weeks. Patients can contact study staff 
for more detail if questions arise. 
Contact with patients can be requested 
by staff if data integrity or compliance 
issues are detected in ongoing data 
review. Survey data will be entered into 
REDCap. 
 
 
For ET use outcomes, keystrokes (for ET-LT), voice commands (for ET- SS), and time on system will be 
collected continuously. Data about ET use in both arms will be collected in time-stamped log files. The primary 
measure of use will be logons per week. Additional use measures will be: number of ET services used, pages 
viewed, messages posted, and weekly surveys completed.  
 
Data about healthcare use and number of chronic conditions will come from patient EHRs. Medication 
adherence will come from patients' responses to the weekly health survey on ET. 
 
Qualitative. Interviews will be conducted from prepared scripts by people unaffiliated with the study, trained 
and monitored by Prof. Mares. Data will be gathered from 32 patients, 16 in each arm, balanced by sex and 
number of chronic conditions (4-5 vs. 6+). Half of each group will be interviewed at 4 months, the other half at 8 
months. This balances the need for insight into patient experience with the need to avoid confounding effects of 
interviews with ET effects. They will be asked about insights critical to ultimate ET dissemination: barriers to use, 
Table 2. Measures Used to Evaluate ET-SS vs. ET-LT vs. Control 
Endpoint 
Measure 
# Qs 
Source 
Psychometrics 
Primary Outcomes 
Pain Interference 
Promis-29 v2.1 
4 
Patient 
α range =  
.88 - .9760 
Psychosocial QOL 
Promis-29 v2.1 
12 
Patient 
N/A 
Secondary Outcomes 
ET use @ 8 mos. 
# logons 
N/A 
ET 
N/A 
Physical QOL 
Promis-29 v2.1 & Late Life 
Function and Disability 
Instrument 
53 
Patient 
N/A 
Pain Intensity 
Promis-29 v2.1 & Lorig 
3 
Patient 
N/A 
30 day readmits 
Admissions 
1 
Patient 
N/A 
Health distress 
Lorig Health Distress Scale 
4 
Patient 
α = .8761 
Well-being 
WHO Well-Being Index 
5 
Patient 
α = .9362 
Loneliness 
NIH Toolbox 
5 
Patient 
α = .9363 
Comm w/doctors Lorig Comm w. Physicians 
3 
Patient 
α = .8961 
Irritability 
Brief Irritability Test (BITe) 
5 
Patient 
α > .8864 
 
 
 
 
 
Mediator Candidates 
ET use @ 4 mos. 
# page views w/o nav. 
N/A 
ET 
N/A 
Competence 
Disease Self-Efficacy 
6 
Patient 
α = .9165 
Relatedness 
Bonding 
5 
Patient 
α = .8566 
Motivation 
Tx Self-Regulation subset 
4 
Patient 
N/A 
Exercise 
Scale created for study 
4 
Patient 
N/A 
Pain Catastroph. Pain Catastrophizing Scale 
13 
Patient 
α = .87 
Moderator Candidates 
Sex 
Baseline survey 
1 
Patient 
N/A 
Chronic conditions 
Total # 
N/A 
EHR 
N/A 
Barriers to tech 
Created for study 
2 
Patient 
N/A 
Exploratory Questions 
ET use by service 
# pageviews per service 
N/A 
ET 
N/A 
ET/CR perception Interviews; Scale created for 
study 
9 
Pt/cl staff 
N/A  
Medication 
ET Weekly Survey: ET arms 
1 
ET 
N/A 
Alcohol use 
AUDIT-C 
3 
Patient 
α = .9167 
Smoking 
# per day 
1 
Patient 
NA 
Falls 
# past 6 months 
2 
Patient 
NA 
Other Measures 
Demographics 
Race/ethnicity, education 
2 
Patient 
N/A 

 
19
technical issues that arose, whether and how ET fit into their day, what could be done to make ET better, 
reactions to the device(s), and how ET came up in appointments with their doctor. Clinicians will encounter ET 
only through the CR. We will record and analyze clinicians' comments and questions during meetings where we 
introduce the study and the CR to the clinics, and we will ask about anticipated barriers and benefits. We will 
document the protocols that clinics use to receive and disseminate the CR to clinicians. We will interview 5 
clinicians at the end of data collection about their experiences with and perceptions of the CR. We will compare 
the qualitative and quantitative results to obtain further insights. If we find QOL improvements in the ET-SS 
quantitative results, for example, we will frame our qualitative interviews of both patients and providers at 8 
months to help us understand why and how the effects  occurred. 
 
Data quality. Weiskopf et al.68 offer quality assessment guidelines for using EHRs based on data correctness, 
completeness, and currency, which we will follow in assessing our data. We will use data related to diabetes, 
obesity, kidney function, lung function, hyperlipidemia, and hypertension found in Epic’s EHR (Table 2). We 
believe these data are fit for use because: 1) it is usual at UW Health to measure blood pressure in older 
patients with at least two consecutive readings at the clinical visit69 (otherwise, we will take the average of the 
last two measures, an approach recommended in national guidelines70) and 2) BMI will be taken from older 
adults with MCCs, a population whose BMI data typically have high levels of completeness71 (weight measured 
at each visit; height measured annually). The opportunity for human error in recorded lab scores is very low 
because lab test data are automatically entered into the EHR by the lab. The data come from patients who are 
currently in treatment, thus current. Hence, we believe our data can be treated as complete and correct. 
 
6.0   
STATISTICAL CONSIDERATIONS 
Power for primary outcome. Our primary outcome that we are using to power the study is reduction in pain 
interference. Based on PROMIS validation studies with chronic pain samples, a difference of 3 points or more 
between study arms was considered to indicate a clinically meaningful difference72. The PROMIS website also 
gives the mean and standard deviation of the T-score metric (M = 50, SD = 10). We powered the analysis to be 
able to detect a 3-point difference (an effect size of d = .30) between control and ET-LT, and then the same 
magnitude of effect between ET-LT and ET-SS (an effect size of d = .60 for control versus ET-SS). Across 10,000 
LMEM simulations, a post-attrition-N = 255, we would have power >80% to detect the study arm X time 
interaction. 
 
Missingness. Data are likely to be missing not-at-random. This may lead to biased parameter estimates. 
Pattern-mixture modeling will test the sensitivity of our intervention analysis to missing data assumptions,73,74 
by conducting sensitivity analyses after imputing missing data with a range of clinically plausible values using 
assumptions for missing data (e.g. best or worst case, with and without multiple imputation).75,76 With regard to 
survey data, in the previous ET trial, 353 of 390 participants (90.51%) completed the 6-month survey and 310 of 
390 (79.49%) completed the 12-month survey. In completed surveys, data were missing on about 2% of core 
items. We expect similar rates in this study, that is, about 87% survey completion at 8 months. Thus, we 
increased the total sample size after attrition to 291 to increase the likelihood of detecting effects on the 
outcomes with 2 moderators: sex and number of chronic conditions. 
 
Quantitative analysis plan. Intention-to-treat will be the data primary analyses. 
Assumptions and randomization effect. We will report descriptive statistics for all demographic and clinical 
variables in both arms to ensure that randomization produced comparable groups; if not, variables with 
significant differences will be included as covariates in a sensitivity analysis. Sensitivity analyses (with and 
without covariate adjustment) will determine the robustness of covariate-related error control. All outcome 
variables will be examined using standard summary statistics, visualizations, and tests for normality and 

 
20
homoscedasticity. Data will be transformed for continuous outcomes that do not meet the assumptions of a 
normally distributed outcome. We will test for a main effect of clinic site. If significant, we will add clinic site to 
our models to control for its variance.  
 
Effectiveness of control vs. ET-LT vs. ET-SS. Linear mixed effects models (LMEM)—which account for 
dependence among successive observations for the same participant and can address incomplete data—will be 
used to examine effects of study arm (ET-SS vs. ET-LT vs. control, a between-subjects factor) on our outcomes 
over time. We will conduct specific treatment X time contrasts both between, and within, groups to test time-
based effects. For binary, count, and other non-normal data, generalized mixed effects models will be used. 
 
Mediation and moderation effects. Structural equation modeling (SEM) will explore the effects of mediation 
on the relation between study arms and our two primary outcomes. We anticipate that the impact of study 
arm on pain interference and psychological QOL will be mediated by SDT (competence, relatedness, and 
motivation), exercise, and pain catastrophizing at 4 months. SEMs involving ET use will be run separately for 
just the ET-SS and ET-LT arms. 2) We anticipate that sex, number of chronic conditions, and barriers to 
technology will moderate the indirect paths of the mediation models (see Figure 1, logic diagram). 
 
Impact of interim analyses on Type 1 error for whole sample analysis. Findings that are significant in the 
interim but not the final analyses will be treated as non-significant. The final analysis will examine all data 
over time. For analyses that address the same hypothesis a Holm-Bonferroni correction will be applied.  
 
Qualitative analyses. For each data set, a coding scheme of key themes will be constructed, based on the 
research questions (perceived benefits, barriers to use) and examination of the data. Each scheme will be pilot 
tested, then 2 trained coders will code an overlapping subsample of 20% of content. Once reliability is 
established (minimum Krippendorff’s alpha of .80 per category), the coders will work independently in NVivo. 
 
7.0 
RECORDS TO BE KEPT  
• Subject Intake  
• Subject Demographics 
• Subject Consent Forms 
• HIPAA Authorization Form 
• Baseline and follow up survey data 
• Coded ET use data   
• Coded patient health care utilization   
• Coded transcripts from interviews with patients 
 
 
 
 
 
 
 
 
 
 
 

 
21
 
Appendix A 
CENTER FOR HEALTH ENHANCEMENT SYSTEMS STUDIES 
Data and Safety Monitoring Plan 
 
Data and Safety Monitoring Committee 
The Center uses the Data Monitoring Committee (DMC) located in the University of Wisconsin’s Institute for 
Clinical and Translational Research (ICTR). The ICTR DMC will provide services to ensure appropriate measures 
are in place to promote subject safety, research integrity and compliance with federal regulations and local 
policies. The DMC members will review protocol-specific reports created by statisticians using data pulled 
from the Research Electronic Data Capture (REDCap) data management tool. These standard reports will 
include an overview of study objectives, a review of actual and projected accrual rates, an evaluation of 
patient demographics for balance of randomization, and a summary of the number and seriousness of adverse 
events. An interim analysis of study results may be performed and source documents may be reviewed to 
allow the board to independently judge whether the overall integrity and conduct of the protocol remain 
acceptable based on data provided and reported by the Principal Investigators. The board will make 
recommendations to the Principal Investigator that could include actions of continuation, modification, 
suspension, or termination. 
 
In providing oversight for the conduct of this study, the ICTR DMC will meet annually during the 5-year study. 
Additional meetings may be scheduled as determined by the DMC or as requested by the PIs. 
 
The predefined stopping points for this study will include excess rates of hospital re-admissions or quality of 
life determinates, exceeding 30% of pre-test values in both cases. We will record events that are deemed to 
be ‘possibly, probably, or definitely’ related to the study intervention, and communicate to the AHRQ, DMC 
and UW IRB in accordance with the following guideline. 
 
What Event is Reported 
When is Event Reported 
By Whom is Event 
Reported 
To Whom is Event 
Reported 
Fatal or life-threatening 
unexpected, suspected 
serious adverse reactions 
Within 7 calendar days of initial  
receipt of information 
Investigator 
Internal IRBs 
NHLBI and/or DMC 
Sponsor or designee FDA (if IND study) 
Non-fatal, non-life-
threatening unexpected, 
suspected serious adverse 
reactions 
Within 15 calendar days of initial 
receipt of information 
Investigator 
Internal IRBs/ 
Institutional Officials 
NHLBI and/or DMC 
Sponsor or designee 
FDA  
All participating 
investigators 
Unanticipated adverse 
device effects 
Within 10 working days of investigator 
first learning of effect 
Investigator 
Internal IRBs 
NHLBI and/or DMC 
Sponsor or designee FDA (if IDE study) 
Unanticipated Problem 
that is not an SAE 
Within 14 days of the investigator 
becoming aware of the problem 
Investigator 
Internal 
IRBs/Institutional 
Officials, 
NHLBI and/or DMC 
All Unanticipated 
Problems2 
Within 30 days of the IRB’s receipt of 
the report of the UP from the 
IRB 
OHRP 
Investigator 
Internal IRBs and DMC 

 
22
Data Security and Privacy Monitoring Plan 
In accordance with the Health Insurance Portability and Accountability Act (HIPAA), CHESS has adopted the 
following Data Security and Privacy Monitoring Plan. 
 
I. Appointment of Center Data Security Officers. 
A. Gina Landucci and Matt Wright are the security officers for the Center. 
B. Responsibilities of the security officers include: 
• Developing information technology (IT) security policies 
• Increasing security awareness and providing training for all Center faculty and Team Members 
• Providing virus protection for IT resources 
• Maintaining security patches on computing equipment 
• Developing and implementing back up procedures 
• Performing periodic vulnerability scanning on computers 
• Reviewing and updating firewall strategies and policies 
• Enhancing the physical security of IT resources 
• Performing annual review of staff policies and procedures contained in this document 
• Conducting formal HIPAA Risk Assessment Inventory per requirements of the UW HIPAA 
Office; last review completed summer 2018. 
 
II. Policy for Orientation and Training 
A. All Center students, faculty, and staff are required to complete the UW-Madison 2019-20 HIPAA 
Privacy & Security Training course online which is offered in Canvas at 
https://canvas.wisc.edu/courses/173269. Contact Judy Ganch if you do not have the HIPAA 
training on your Canvas dashboard. Please email Judy when you have completed the training. You 
will need to retake this course annually. 
B. All Center students, faculty, and staff that are involved in human subjects research are required 
to complete the UW-Madison CITI Human Subjects training online at 
https://my.gradsch.wisc.edu/citi/index.php. Register with your NetID. Select the training for the 
UW Social & Behavioral Course. Please email Judy Ganch when you have completed the training. 
You will need to take a refresher course every three years. 
C. All Center students, faculty, and staff that are involved in human subjects research are 
required to complete the UW-Madison CITI Human Subjects training online at 
https://my.gradsch.wisc.edu/citi/index.php. Register with your NetID. Select the training for 
the GCP—Social and Behavioral Research Best Practices for Clinical Research. Please email 
Judy Ganch when you have completed the training. You will need to take a refresher course 
every three years. 
D. Successful completion of the CITI and HIPAA training is required before any team member is 
allowed access to Protected Health Information (PHI) or Research Data. 
E. All Center Team Members are required to complete training with Judy Ganch on Center 
security procedures and policies. 
F. Sign (on the last page) the “Center Data Security Policy and Privacy Monitoring Plan” upon 
completion of this training and return to Judy Ganch. 
 
III. Workstation Policy 
A. All workstations will require login with a unique user name and password. 
investigator. 

 
23
B. All workstations are required to be joined to the ENGR domain under the control of the College 
of Engineering. Access to specific folders on the Center network will be approved by the Center 
Management team or the study Project Director. 
C. All workstations are required to use anti-virus software that can be remotely administered from the 
College of Engineering domain. 
D. Users will log-out from or lock workstations when leaving them unattended. 
E. Screen savers will be configured to require a password and to activate after ten minutes 
of workstation inactivity. 
F. Users requiring remote access to the Center network will only do so with computers 
specifically certified by a Center Data Security Officer. 
G. Study coordinators or support personnel using desktops in private or public areas must have 
screens adjusted so that visitors cannot read the screen upon entering the space in case PHI or 
other confidential information is displayed. 
H. Access to the network or center work stations along with any on-line systems will be 
terminated immediately following an employee’s last day of work at the Center. 
 
IV. Password Policy 
A. Users will require a password to access any computer connecting to the Center network. 
B. Passwords must meet the requirements of the Computer Aided Engineering department (CAE). 
CAE password construction help can be found here: 
https://kb.wisc.edu/cae/page.php?id=8143. 
C. Ideally the best practice is to re-set your password on a semi-annual basis. 
D. Passwords may not be stored in proximity to the workstation and may not be shared by others. 
 
V. Policy for the Use of Email 
A. Patient Identifiable, Confidential, or Personnel Data may only be included in an 
encrypted attachment and should never be sent in the body or subject line of an email 
message. 
B. Team Members who need to send encrypted attachments should contact a Center Data Security 
Officer to schedule a training session. 
 
VI. Policy for Storage, Retrieval, and Disposal of Protected Information 
A. Effective 1/1/14 any new studies must use the REDCap environment for study 
participant information. Contact Matt for start-up and access information. 
B. Any Patient Identifiable, Confidential, or Personnel Data in electronic form will be stored on 
secure servers only and may not be stored on individual workstations, laptops, or any other 
endpoint devices. Currently, the only place such information can be stored is the R: Drive for 
studies prior to 1/1/14 or REDCap. 
C. Study coordinators will have printers in their office for printing any materials that include PHI 
and/or names of study participants. 
D. All paper-based files will be stored in locked rooms inside locked file cabinets with limited access. 
E. Any offices that contain PHI must be locked when leaving a room. 
F. Servers containing Patient Identifiable, Confidential or Personnel Data must be located 
within physically secured server rooms which can only be accessed by authorized personnel. 
G. The Center Data Security Officers will be responsible for assigning and restricting access to shared 
resources on Center servers. 
H. Patient Identifiable, Confidential, and Personnel Data as a general rule may not be copied to or 
stored on the Center’s publicly accessible servers at any time. However, in some studies, patient’s 
name, disease state, and physician are stored within Center applications only accessible via a 

 
24
secure connection using a codename/password. 
I. Remote access to files on secure Center servers will be provided in a very limited case only through 
a connection from a certified Center Workstation (see Center Workstation policy above for details). 
J. Storage media containing Patient Identifiable, Confidential or Personnel Data will be 
rendered unusable before disposal. 
K. All back up media will be stored in locked rooms with limited access. 
 
VII. Policy for the Use of Endpoints (e.g. Workstation Smartphone, Laptop, Thumb Drive, External 
Hard Drive) Accessing Patient Identifiable, Confidential, and Personnel Data 
A. Staff are not allowed to access or store any PHI data on any endpoint unless the endpoint is 
provided by a Center Data Security Officer ensuring that the endpoint is protected and secured. In 
addition, any PHI data residing on an endpoint must also be encrypted with the oversight of 
a Center Data Security Officer. 
B. All endpoints accessing PHI must be owned and provisioned by the Center. 
C. Patient Identifiable, Confidential, and Personnel Data files may not be transported from the 
Center unless the device/medium is monitored for physical security at all times and the data is 
encrypted on the mobile device. 
D. Remote access to PHI data must be conducted using a secure encrypted end-to-end connection 
implementing modern security best practices either over HTTPS or using the WiscVPN connection. 
 
VIII. Policy Governing the Storage and Use of Audiovisual Materials 
A. Audiovisual media containing Patient Identifiable, Confidential, and Personnel Data are governed 
by the same policies and procedures that apply to handling and use of computerized data, including 
disposal, storage and access to media. 
B. Such audiovisual media may not be transported from the Center unless the material is 
monitored for physical security at all times. 
 
IX. Policy Governing the Transmission of Information via Fax 
A. All outgoing correspondence via fax must be stripped of confidential information. 
B. Before confidential information is transmitted to the Center via fax, sender must notify the 
appropriate Center Team Member to ensure the recipient is available to pick up the fax 
document. 
 
X. Field Hardware Policy 
A. Device provided for study participants will require log-in information to access the device. 
B. Upon their return, all Center study computers, smartphones, or other devices that have been 
used in the field will have all data wiped from the hard drive in compliance with DOD standards. 
C. Field computers will be stored at the Center in a wiped state. 
 
XI. Study Participant Information 
A. Center Team Members will not share or talk about confidential information regarding the CHESS 
study participant with anyone who is not directly involved in the management of the CHESS Project. 
B. Confidential or other sensitive information regarding the study participant cannot be left in 
an unsecured place where others may see it. 
C. Copies of written correspondence about the study participants with anyone other than CHESS 
management cannot be provided, unless specifically authorized to do so by the Project Director. 
D. Access to study participant’s PHI will be approved by the Project Director in coordination with 
the PI for the project. As noted in Section VI, studies beginning after 1/1/14 must use REDCap 

 
25
to enter participant information electronically. 
 
 
XII. User Responsibilities 
A. All Center Team Members are responsible for adhering to the Center Data Security and Privacy 
Monitoring Plan Policies at all times. In addition, all Center Team Members are responsible for 
adhering to the UW-Madison IT policies detailed at https://it.wisc.edu/about/office-of-the-
cio/it- policies/ at all times as well. 
B. All Center Team Members will be given a comprehensive briefing session upon hire. 
C. Usernames and passwords are not to be shared with others. 
D. No equipment may be connected to the Center network without specific prior certification 
by a Center Security Officer. 
E. Specifically, no wireless access points may be deployed or connected to the network under any 
circumstances. 
F. The Center Data Security Officers will maintain records to insure that all Center Team Members 
have been briefed. 
G. The Center Data Security Officers will provide periodic refresher sessions to all Center Team 
Members. 
 
XIII. In case of a PHI breach, Center Security Officer and/or PI reports incident to UW HIPAA Security 
Office, the UW Health Sciences IRB or other IRB of record, the ICTR Data Monitoring Committee, 
and sponsor as required. 
 
 
 

 
26
References  
1. 
Gustafson D, Wise M, Bhattacharya A, Pulvermacher A, Shanovich K, Phillips B, Lehman E, Chinchilli V, 
Hawkins R, Kim JS. The effects of combining Web-based eHealth with telephone nurse case management 
for pediatric asthma control: a randomized controlled trial. JMIR. 2012;14(4):e101. PMCID: PMC3409549. 
2. 
Gustafson DH, Hawkins R, Boberg E, Pingree S, Serlin RE, Graziano F, Chan CL. Impact of a patient- 
centered, computer-based health information/support system. Am J Prev Med. 1999;16(1):1-9. PMID: 
9894548. 
3. 
Gustafson DH, McTavish F, Hawkins R, Pingree S, Arora N, Mendenhall J, Simmons GE. Computer support 
for elderly women with breast cancer. JAMA. 1998;280(15):1305. PMID: 9794300. 
4. 
Gustafson DH, Hawkins R, Pingree S, McTavish F, Arora NK, Mendenhall J, Cella DF, Serlin RC, Apantaku 
FM, Stewart J, Salner A. Effect of computer support on younger women with breast cancer. J Gen Intern 
Med. 2001;16(7):435-45. PMCID: PMC1495237. 
5. 
Gustafson DH, Hawkins R, McTavish F, Pingree S, Chen WC, Volrathongchai K, Stengle W, Stewart JA, Serlin 
RC. Internet-based interactive support for cancer patients: are integrated systems better? J Commun. 
2008;58(2):238-57. PMCID: PMC3144782. 
6. 
Gustafson DH, McTavish FM, Chih MY, Atwood AK, Johnson RA, Boyle MG, Levy MS, Driscoll H, Chisholm 
SM, Dillenburg L, Isham A, Shah D. A smartphone application to support recovery from alcoholism: a 
randomized clinical trial. JAMA Psychiatry. 2014;71(5):566-72. PMCID: PMC4016167. 
7. 
Gustafson DH, DuBenske LL, Namkoong K, Hawkins R, Chih MY, Atwood AK, Johnson R, Bhattacharya A, 
Carmack CL, Traynor AM, Campbell TC, Buss MK, Govindan R, Schiller JH, Cleary JF. An eHealth system 
supporting palliative care for patients with non-small cell lung cancer: a randomized trial. Cancer. 
2013;119(9):1744-51. PMCID: PMC3684251. 
8. 
Johnson J, Finn K. Designing user interfaces for an aging population: towards universal design. 1st ed. 
Cambridge, MA: Morgan Kaufmann; 2017. 
9. 
Gruneir A, Fung K, Fischer HD, Bronskill SE, Panjwani D, Bell CM, Dhalla I, Rochon PA, Anderson G. Care 
setting and 30-day hospital readmissions among older adults: a population-based cohort study. CMAJ. 
2018;190(38):E1124-e33. PMCID: PMC6157496. 
10. 
Snoek FJ, Pouwer F, Welch GW, Polonsky WH. Diabetes-related emotional distress in Dutch and US diabetic 
patients: cross-cultural validity of the problem areas in diabetes scale. Diabetes Care. 2000;23(9):1305-9. 
11. 
Burholt V, Scharf T. Poor health and loneliness in later life: the role of depressive symptoms, social 
resources, and rural environments. J Gerontol B Psychol Sci Soc Sci. 2014;69(2):311-24. PMCID: 
PMC3968864. 
12. 
Luo Y, Hawkley LC, Waite LJ, Cacioppo JT. Loneliness, health, and mortality in old age: a national 
longitudinal study. Soc Sci Med. 2012;74(6):907-14. PMCID: PMC3303190. 
13. 
Perissinotto CM, Stijacic Cenzer I, Covinsky KE. Loneliness in older persons: a predictor of functional 
decline and death. Arch Intern Med. 2012;172(14):1078-83. PMCID: PMC4383762. 
14. 
Krebs P, Duncan DT. Health app use among US mobile phone owners: a national survey. JMIR Mhealth 
Uhealth. 2015;3(4):e101. PMCID: PMC4704953. 
15. 
Perez S. Nearly 1 in 4 People Abandon Mobile Apps after Only One Use. San Francisco: TechCrunch; 2016 
May 31. https://techcrunch.com/2016/05/31/nearly-1-in-4-people-abandon-mobile-apps-after-only- one-
use/. Accessed June 4, 2020. 
16. 
Perro J. Mobile Apps: What’s a Good Retention Rate? Localytics; 2017 March 21. Updated March 22, 2018. 
http://info.localytics.com/blog/mobile-apps-whats-a-good-retention-rate. Accessed June 4, 2020. 
17. 
Endeavor Partners. Inside Wearables: How the Science of Human Behavior Change Offers the Secret to 
Long-Term Engagement. Endeavor Partners; 2014 Jan. https://blog.endeavour.partners/inside-wearable- 
how-the-science-of-human-behavior-change-offers-the-secret-to-long-term-engagement-a15b3c7d4cf3. 
Accessed September 19, 2019. 
18. 
Mizak A, Park M, Park D, Olson K. Amazon “Alexa” Pilot Analysis Report. Front Porch Center for Innovation 

 
27
and Wellbeing; 2017 Dec. http://fpciw.org/wp-content/uploads/sites/15/2017/12/FINAL- DRAFT-Amazon-
Alexa-Analysis-Report.pdf 
19. 
Wallace T, Morris JT. Smart speaker usability by military service members with mTBI and PTSD. The Journal 
on Technology and Persons with Disabilities. California State University, Northridge. 2018. 
http://scholarworks.csun.edu/bitstream/handle/10211.3/202991/JTPD-2018-ID16-p127-139.pdf 
20. 
Capan F. Why Amazon device is a gift for healthcare. Med Mark Media. 2016;51(1):20. http://media.mmm-
online.com/documents/188/january_2016_viewpoint_46793.pdf 
21. 
St. John A. Amazon Echo Voice Commands Offer Big Benefits to Users With Disabilities. Consumer Reports, 
Inc.; 2017 Jan 20. https://www.consumerreports.org/amazon/amazon-echo-voice-commands- offer-big-
benefits-to-users-with-disabilities/. Accessed June 4, 2020. 
22. 
Thompson AC. Persuading Your Older Parents to Take the Smart Home Leap. CNET 2017 April 11. 
https://www.cnet.com/news/how-to-have-the-tech-talk-with-your-aging-parents/. Accessed June 4, 2020. 
23. 
Woyke E. The Octogenarians who Love Amazon's Alexa. Cambridge, MA: MIT Technology Review; 2017 
June 9. https://www.technologyreview.com/s/608047/the-octogenarians-who-love-amazons-alexa/. 
Accessed June 4, 2020. 
24. 
DailyCaring Editorial Team. Amazon Echo for Dementia: Technology for Seniors. DailyCaring, LLC; 2018. 
https://dailycaring.com/amazon-echo-for-dementia-technology-for-seniors/. Accessed September 19, 
2019. 
25. 
Perez S. 47.3 Million U.S. Adults have Access to a Smart Speaker, Report Says. San Francisco: TechCrunch; 
2018 Mar 7. https://techcrunch.com/2018/03/07/47-3-million-u-s-adults-have-access-to-a-smart-speaker-
report-says/. Accessed June 8, 2020. 
26. 
National Public Radio, Edison Research. The Smart Audio Report. Winter 2018. National Public Media LLC, 
2019. https://techcrunch.com/2018/03/07/47-3-million-u-s-adults-have-access-to-a-smart-speaker-
report-says/. 
27. 
Gartner, Inc. Newsroom Press Release: Gartner Reveals Top Predictions for IT Organizations and Users in 
2017 and Beyond. Stamford, CT: Gartner, Inc.; 2016 Oct 18. 
https://www.gartner.com/en/newsroom/press-releases/2016-10-18-gartner-reveals-top-predictions-for-
it- organizations-and-users-in-2017-and-beyond. Accessed June 4, 2020. 
28. 
National Public Radio, Edison Research. The Smart Audio Report. Fall/winter. National Public Media LLC, 
2017. https://www.npr.org/about-npr/735821111/npr-and-edison-research-release-the-smart-audio-
report-spring-2019. 
29. 
National Public Radio, Edison Research. The Smart Audio Report., Spring 2019. National Public Media LLC, 
2019. https://www.npr.org/about-npr/735821111/npr-and-edison-research-release-the-smart-audio-
report-spring-2019. Accessed June 8, 2020. 
30. 
Mayer DK, Landucci G, Awoyinka L, Atwood AK, Carmack CL, Demark-Wahnefried W, McTavish F, 
Gustafson DH. SurvivorCHESS to increase physical activity in colon cancer survivors: can we get them 
moving? J Cancer Surviv. 2018;12(1):82-94. PMCID: PMC5911378. 
31. 
Gustafson Jr DH, Gustafson Sr DH, Cody OJ, Chih MY, Johnston DC, Asthana S. Pilot test of a computer-
based system to help family caregivers of dementia patients. J Alzheimers Dis. 2019;70(2):541-52. PMID: 
31256126. 
32. 
Gustafson Sr. DH, McTavish F, Gustafson DH, Jr., Mahoney JE, Johnson RA, Lee JD, Quanbeck A, Atwood 
AK, Isham A, Veeramani R, Clemson L, Shah D. The effect of an information and communication technology 
(ICT) on older adults' quality of life: study protocol for a randomized control trial. Trials. 2015;16(1):191. 
PMCID: PMC4417513. 
33. 
Bodenheimer T, Berry-Millett R. Care management of patients with complex health care needs. Policy. 
2009;1:6. 
34. 
Webb TL, Sheeran P. Does changing behavioral intentions engender behavior change? A meta-analysis of 
the experimental evidence. Psychol Bull. 2006;132(2):249-68. PMID: 16536643. 

 
28
35. 
Namkoong K, Shah DV, Han JY, Kim SC, Yoo W, Fan D, McTavish FM, Gustafson DH. Expression and 
reception of treatment information in breast cancer support groups: how health self-efficacy moderates 
effects on emotional well-being. Patient Educ Couns. 2010;81 Suppl:S41-7. PMCID: PMC2993816. 
36. 
Han JY, Shah DV, Kim E, Namkoong K, Lee SY, Moon TJ, Cleland R, Bu QL, McTavish FM, Gustafson DH. 
Empathic exchanges in online cancer support groups: distinguishing message expression and reception 
effects. Health Commun. 2011;26(2):185-97. PMCID: PMC3551338. 
37. 
Seligman ME, Csikszentmihalyi M. Positive psychology: an introduction. In: Csikszentmihalyi M, ed. Flow 
and the Foundations of Positive Psychology. Netherlands: Springer; 2014:279-98. 
38. 
Wheeler DJ, Chambers DS. Understanding Statistical Process Control. Knoxville, TN: SPC Press; 1992. 
39. 
Parekh AK, Goodman RA, Gordon C, Koh HK. Managing multiple chronic conditions: a strategic framework 
for improving health outcomes and quality of life. Public Health Rep. 2011;126(4):460-71. PMCID: 
PMC3115206. 
40. 
Gustafson DH, DuBenske LL, Atwood AK, Chih M-Y, Johnson RA, McTavish F, Quanbeck A, Brown RL, Cleary 
JF, Shah D. Reducing symptom distress in patients with advanced cancer using an eAlert system for 
caregivers: pooled analysis of two randomized clinical trials. JMIR. 2017;19(11):e354. PMCID: 
PMC5705856. 
41. 
Leite I, Martinho C, Paiva A. Social robots for long-term interaction: a survey. Int J Soc Robot. 
2013;5(2):291-308. 
42. 
Gockley R, Bruce A, Forlizzi J, Michalowski M, Mundell A, Rosenthal S, Sellner B, Simmons R, Snipes K, 
Schultz AC, eds. Designing robots for long-term social interaction. 2005 IEEE/RSJ International Conference 
on Intelligent Robots and Systems; 2005 Aug 2-6; Edmonton, Alberta: IEEE; 2005:1338-43. 
43. 
Sinha T, Cassell J, editors. We click, we align, we learn: Impact of influence and convergence processes on 
student learning and rapport building. Proceedings of the 1st Workshop on Modeling INTERPERsonal 
SynchrONy And infLuence; 2015 Nov 13; Seattle, WA: ACM; 2015:13-20. 
44. 
Spencer G. Much More than a Chatbot: China's Xiaoice Mixes AI with Emotions and Wins Over Millions of 
Fans: Microsoft; 2018 Nov 1. https://news.microsoft.com/apac/features/much-more-than-a-chatbot- 
chinas-xiaoice-mixes-ai-with-emotions-and-wins-over-millions-of-fans/. Accessed June 4, 2020. 
45. 
Michaelis JE, Mutlu B, eds. Someone to read with: Design of and experiences with an in-home learning 
companion robot for reading. Proceedings of the 2017 CHI Conference on Human Factors in Computing 
Systems; 2017 May 6-11; Denver, CO: ACM; 2017:301-12. 
46. 
Kidd CD, Taggart W, Turkle S, eds. A sociable robot to encourage social interaction among the elderly. 
Proceedings 2006 IEEE International Conference on Robotics and Automation; 2006 May 15-19; Orlando, 
FL: Wada K, Shibata T. Social effects of robot therapy in a care house–change of social network of the 
residents for one year. JACIII. 2009;13(4):386-92. 
47. 
Wada K, Shibata T. Social effects of robot therapy in a care house–change of social network of the 
residents for one year. JACIII. 2009;13(4):386-92.  
48. 
Wada K, Shibata T. Living with seal robots—its sociopsychological and physiological influences on the 
elderly at a care house. IEEE Trans Robot. 2007;23(5):972-80. 
49. 
Lee MK, Kiesler S, Forlizzi J, Srinivasa S, Rybski P, eds. Gracefully mitigating breakdowns in robotic services. 
2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI); 2010 Mar 2- 5; Osaka, 
Japan: IEEE; 2010:203-10. 
50. 
Šabanović S. Robots in society, society in robots. Int J Soc Robot. 2010;2(4):439-50. 
51. 
Leite I, Lehman JF, eds. The robot who knew too much: toward understanding the privacy/personalization 
trade-off in child-robot conversation. Proceedings of the The 15th International Conference on Interaction 
Design and Children; 2016 Jun 21-24; Manchester, United Kingdom: ACM; 2016:379-87. 
52. 
Barrett B. What Amazon Echo and Google Home Do With Your Voice Data. New York: Wired Magazine; 
2017 Nov 24. https://www.wired.com/story/amazon-echo-and-google-home-voice-data-delete/. Accessed 
June 4, 2020. 

 
29
53. 
McCarty D, Gustafson DH, Wisdom JP, Ford J, Choi D, Molfenter T, Capoccia V, Cotter F. The Network for 
the Improvement of Addiction Treatment (NIATx): enhancing access and retention. Drug Alcohol Depend. 
2007;88(2-3):138-45. PMCID: PMC1896099. 
54. 
Gustafson DH, Brennan PF, Hawkins RP. Investing in e-health: What it takes to sustain consumer health 
informatics. In: Hannah KJ, Ball MJ, eds. Health Informatics. New York: Springer Science+Business Media, 
LLC; 2007. 
55. 
Gustafson DH, Hundt AS. Findings of innovation research applied to quality management principles for 
health care. Health Care Manage Rev. 1995;20(2):16-33. PMID: 7607882. 
56. 
Hoffman KA, Ford JH 2nd, Choi D, Gustafson DH, McCarty D. Replication and sustainability of improved 
access and retention within the Network for the Improvement of Addiction Treatment. Drug Alcohol 
Depend. 2008;98(1-2):63-9. PMCID: PMC2607248. 
57. 
Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Developing and testing a model to 
predict outcomes of organizational change. Health Serv Res. 2003;38(2):751-76. PMCID: PMC1360903. 
58. 
NIATxNPO. NIATx process improvement: conducting a walk-through. CHESS/NIATx, University of 
Wisconsin-Madison; 2009. 
59. 
Maher L, Gustafson D, Evans A. Sustainability, Model and Guide. Leicester, England: British National Health 
Service Modernization Agency; 2004. 
60. Amtmann D, Cook KF, Jensen MP, Chen WH, Choi S, Revicki D, Cella D, Rothrock N, Keefe F, Callahan L, Lai 
JS. Development of a PROMIS item bank to measure pain interference. Pain. 2010;150(1):173–182. 
61. Stanford Chronic Disease Self-Management Study. Psychometrics reported in: Lorig K, Stewart A, Ritter P, 
González V, Laurent D, & Lynch J, Outcome Measures for Health Education and other Health Care 
Interventions. Thousand Oaks CA: Sage Publications, 1996, pp.24,40. 
62. Allgaier AK, Kramer D,Saravo B, Mergi R, Fejtkova S, & Hegerl U. Beside the Geriatric Depression Scale: the 
WHO-Five Well-being Index as a valid screening tool for depression in nursing homes. International Journal 
of Geriatric Psychiatry. 2013:28(11), 1197-1204. 
63. Cyranowski JM, Zill N, Bode R, Butt Z, Kelly MA, Pilkonis PA, & Cella D. Assessing social support, 
companionship, and distress: National Institute of Health (NIH) Toolbox Adult Social Relationship Scales. 
Health Psychology. 2013:32(3), 293. 
64. Holtzman S, O’Connor BP, Barata PC, & Stewart DE. The Brief Irritability Test (BITe) A Measure of Irritability 
for Use Among Men and Women. Assessment. 2015:22(1), 101-115. 
65. Stanford/Garfield Kaiser Chronic Disease Dissemination Study. Psychometrics reported in: Lorig KR, Sobel 
DS, Ritter PL, Laurent D, Hobbs M. Effect of a self-management program for patients with chronic disease. 
Effective Clinical Practice. 2001:4, 256-262. 
66. Gustafson DH, Hawkins RP, McTavish FM, Pingree S, Chen WC, Volrathongchai K, et al. Internet-based 
interactive support for cancer patients: Are integrated systems better? Journal of 
Communication. 2008;58(2):238–257. doi: 10.1111/j.1460-2466.2008.00383. 
67. 
Jeong HS, Park S, Lim SM, Ma J, Kang I, Kim J, Kim EJ, Choi YJ, Lim JH, Chung YA, Lyoo IK, Yoon S, Kim JE. 
Psychometric Properties of the Alcohol Use Disorders Identification Test-Consumption (AUDIT-C) in Public 
First Responders. Subst Use Misuse. 2017 Jul 3;52(8):1069-1075. doi: 10.1080/10826084.2016.1271986. Epub 
2017 Mar 21. PMID: 28323556. 
68. Weiskopf NG, Bakken S, Hripcsak G, Weng C. A data quality assessment guideline for electronic health 
record data reuse. EGEMS (Wash DC). 2017;5(1):14. PMCID: PMC5983018 
69. Oladipo I, Adedokun A. Comparison of the average of five readings with averages from fewer readings for 
automated oscillometric blood pressure measurement in an outpatient clinic. Korean Circ J. 
2013;43(5):329-35. PMCID: PMC3675307. 
70. Weber MA, Schiffrin EL, White WB, Mann S, Lindholm LH, Kenerson JG, Flack JM, Carter BL, Materson BJ, 
Ram CV, Cohen DL, Cadet JC, Jean-Charles RR, Taler S, Kountz D, Townsend RR, Chalmers J, Ramirez AJ, 
Bakris GL, Wang J, Schutte AE, Bisognano JD, Touyz RM, Sica D, Harrap SB. Clinical practice guidelines for 

 
30
the management of hypertension in the community: a statement by the American Society of Hypertension 
and the International Society of Hypertension. J Clin Hypertens (Greenwich). 2014;16(1):14-26. PMID: 
24341872. 
71. Turner LR, Harris MF, Mazza D. Obesity management in general practice: does current practice match 
guideline recommendations? Med J Aust. 2015;202(7):370-2. PMID: 25877119. 
72. Meaningful change for PROMIS® - estimates for magnitude of changethat corresponds to “important” 
change are published. (n.d.) Health Measures. https://www.healthmeasures.net/score-and-
interpret/interpret-scores/promis/meaningful-change 
73. 
Hedeker D, Gibbons RD. Longitudinal Data Analysis. Hoboken, NJ: John Wiley & Sons; 2006. 
74. 
Enders CK. Missing not at random models for latent growth curve analyses. Psychol Methods. 
2011;16(1):1-16. PMID: 21381816. 
75. Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. 
Int J Med Inform. 2004;73(11-12):781-95. PMID: 15491929. 
76. 
White IR, Horton NJ, Carpenter J, Pocock SJ. Strategy for intention to treat analysis in randomised trials 
with missing outcome data. BMJ. 2011;342:d40. PMCID: PMC3230114.