A Technology-Driven Intervention to Improve Early Detection and Management of Cognitive Impairment
Phase: NA
Conditions: Dementia, Cognitive Impairment
Sponsor: HealthPartners Institute

Clinical Interventional Study Protocol 
A TECHNOLOGY-DRIVEN INTERVENTION TO 
IMPROVE EARLY DETECTION AND MANAGEMENT 
OF COGNITIVE IMPAIRMENT  
Co-Principal Investigators: 
Leah Hanson, PhD  
Senior Investigator, HealthPartners Institute 
Rebecca Rossom, MD, MS  
Senior Investigator, HealthPartners Institute 
Associate Professor, University of Minnesota 
Supported by: 
The National Institute on Aging 
R33AG069770 
NCT05723523

Version 5.0 
May 8, 2025
TABLE OF CONTENTS  
 
 
 
 
 
 
 
 
 
 
 
Page 
A Technology-Driven Intervention to Improve Early Detection and Management of 
Cognitive Impairment ................................................................................................................   
Summary of CHANGES ...........................................................................................................iv 
PRÉCIS ..................................................................................................................................... v 
Study Title ............................................................................................................................... v 
Objectives ............................................................................................................................... v 
Design and Outcomes ............................................................................................................. v 
Interventions and Duration ...................................................................................................... v 
Sample Size and Population ................................................................................................... v 
STUDY TEAM ROSTER ............................................................................................................ 1 
Study objectives ...................................................................................................................... 2 
1. 
BACKGROUND AND RATIONALE ................................................................................... 3 
1.1 
Background on Condition, Disease, or Other Primary Study Focus ............................ 3 
1.2 
Study Rationale .......................................................................................................... 4 
2. 
STUDY DESIGN ................................................................................................................ 6 
3. 
SELECTION AND ENROLLMENT OF PARTICIPANTS .................................................... 8 
3.1 
Inclusion Criteria ......................................................................................................... 8 
3.2 
Exclusion Criteria ........................................................................................................ 9 
3.3 
Study Enrollment Procedures ..................................................................................... 9 
5.  STUDY INTERVENTIONS ...................................................................................................10 
5.1 Interventions, Administration, and Duration .....................................................................10 
5.2 
Handling of Study Interventions .................................................................................12 
5.3 
Concomitant Interventions .........................................................................................12 
5.3.1 
Allowed Interventions ..........................................................................................12 
5.3.2 
Required Interventions ........................................................................................12 
5.3.3 
Prohibited Interventions ......................................................................................12 
5.4 
Adherence Assessment .............................................................................................12 
6. 
STUDY PROCEDURES ....................................................................................................13 
6.1 Schedule of Evaluations ..................................................................................................13 
6.2 
Description of Evaluations .........................................................................................13 
6.2.1 
Screening ..............................................................................................................13 
6.2.2 
Patient accrual and index visit assessment .......................................................14 
NCT05723523

6.2.3 
Follow-up Visits ....................................................................................................14 
6.2.4 
Completion/Final Evaluation ...............................................................................14 
6.2.5 
Clinician surveys ..................................................................................................14 
6.2.6 
Chart audits ..........................................................................................................15 
6.2.7 
Patient / caregiver dyad interviews .....................................................................15 
7. 
SAFETY ASSESSMENTS ................................................................................................15 
7.1 Specification of Safety Parameters ..................................................................................15 
7.2 
Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters ....16 
7.3 
Clinician Feedback and Safety Events .......................................................................16 
7.3.1 
Clinician Feedback ...............................................................................................16 
7.3.2 
Safety events ........................................................................................................17 
7.3.3 
Follow-up for Safety Events ................................................................................18 
7.4 
Safety Monitoring .......................................................................................................18 
8   INTERVENTION DISCONTINUATION ................................................................................18 
9 
STATISTICAL CONSIDERATIONS ..................................................................................18 
9.1 
General Design Issues ..............................................................................................18 
9.2 
Sample Size and Randomization ...............................................................................19 
9.2.1 
Treatment Assignment Procedures ....................................................................19 
9.2.2 
Sample Size Justification ....................................................................................19 
9.3 
Interim analyses and Stopping Rules .........................................................................21 
9.4 
Outcomes ..................................................................................................................21 
9.4.1 
Primary outcome ..................................................................................................21 
9.4.2 
Secondary outcomes ...........................................................................................21 
9.5 
Data Analyses ...........................................................................................................22 
9.5.1 
Primary analysis ...................................................................................................22 
9.5.2 Secondary analyses ...................................................................................................22 
9.5.3. Secondary outcomes ................................................................................................23 
10. DATA COLLECTION AND QUALITY ASSURANCE ........................................................23 
10.1 Data Collection Forms ...................................................................................................23 
10.1.1 Missing Data .............................................................................................................24 
10.2 
Data Management .....................................................................................................24 
10.3 Quality Assurance .........................................................................................................24 
10.4 Protocol Deviations .......................................................................................................24 
10.5 Monitoring .....................................................................................................................25 
10.6 Regulatory Files ............................................................................................................25 
10.7 Reporting to Sponsor ....................................................................................................25 
10.8 Audits ............................................................................................................................25 
10.9 Study Documentation ....................................................................................................25 
11. PARTICIPANT RIGHTS AND CONFIDENTIALITY ..........................................................25 
NCT05723523

11.1 Institutional Review Board (IRB) Review .......................................................................25 
11.2 
Informed Consent Forms ...........................................................................................26 
11.3 
Participant Confidentiality ..........................................................................................26 
11.4 
Study Discontinuation ................................................................................................26 
12. ETHICAL CONSIDERATIONS ..........................................................................................26 
12.1 Statement of Compliance ...........................................................................................26 
12.2 Investigator Assurances .............................................................................................27 
12.3 Inclusion of Women and Minorities ...........................................................................27 
12.4 Prisoner Certification .....................................................................................................27 
13. COMMITTEES ..................................................................................................................27 
14. PUBLICATION OF RESEARCH FINDINGS .....................................................................28 
15. SUPPLEMENTS / Appendices ........................................................................................28 
15.1 
Procedures schedule .................................................................................................28 
15.2 Informed Consent Document .........................................................................................29 
15.2.2 Survey Invitation and Consent for Primary Care Clinicians ..................................30 
15.3 Summary Results of Primary Care Clinician Pilot Surveys ............................................30 
15.4 Clinical Decision Support Content for Diagnosis and Management of CI .......................33 
15.5 REPLICATION SITE PROCEDURES ............................................................................37 
16. REFERENCES ..................................................................................................................43 
 
 
 
NCT05723523

SUMMARY OF CHANGES 
Summary of Changes from Protocol Version 4 dated July 17, 2023 
to Protocol Version 5 dated May 08, 2025 
 
Page #, Section # 
Change Description 
Rationale/Justification 
Title page, footer, table 
of contents 
Updated date and version number 
New protocol submitted 
Pg 7 
Clarification on 
intervention/observation period 
length for HealthPartners 
Changed paragraph to state “up to 
24 month intervention period” to be 
consistent with the language in 
paragraph immediately following 
Pg 7: Replication 
Update on intervention/observation 
period length for OCHIN 
The observation period length has 
changed to approximately 6 months 
due to some delays in go-live at 
OCHIN. 
Pg 15 
Clarification on chart audit 
procedures 
Clarification in language regarding 
chart audit procedures to state that 
we will be pulling data related to 
completion of assessments in the 
accrued population as one aspect of 
the chart audit, and then manually 
auditing charts of 50-100 patients in 
each group to assess the presence 
of care plans. 
Pg 28: OCHIN Appendix 
Updated study roster to match the 
OCHIN study personnel document 
previously submitted 
No new personnel were added to 
OCHIN’s study team beyond what 
was documented in the OCHIN 
study personnel documented dated 
10/16/2024. Table was updated so 
that the protocol matched what was 
in this document. 
 
 
 
 
NCT05723523

PRÉCIS 
Study Title 
A Technology-Driven Intervention to Improve Early Detection and Management of Cognitive 
Impairment 
Objectives  
Primary Objective: To evaluate the effect of the clinical decision support (CDS) tool for 
identifying and managing cognitive impairment (CI) on rates of CI detection in a pragmatic 
cluster-randomized trial of primary care clinics randomized to intervention (CI-CDS) or usual 
care (UC). 
Secondary Objectives: To assess the extent to which the CI-CDS system increases clinician 
confidence in diagnosing and managing CI, or changes in healthcare utilization costs.   
Design and Outcomes   
This pragmatic clinic-randomized trial will evaluate the effectiveness of the CI-CDS system in 
improving care for patients with elevated risk for CI by making it easier for primary care clinicians 
to assess, diagnose, and manage individuals with CI. The study will randomize clinics 1:1 to 
continue with UC or to deploy the CI-CDS intervention.  
The primary outcome is CI detection, defined as an ICD-10 diagnostic code for CI (dementia or 
mild cognitive impairment [MCI]) documented at outpatient or inpatient encounters or added to 
the problem list during the observation period among patients with elevated risk for CI. 
Secondary outcomes are clinician-reported confidence in CI care management, assessed via 
pre-/post-surveys, and health care utilization among patients with HealthPartners insurance.  
Interventions and Duration  
The CI-CDS user interface will alert primary care clinicians when patients are at elevated risk 
for CI, prompting diagnostic screening exams, and, if indicated, CI diagnoses, quick orders (e.g., 
referrals, procedures, lab assessments, medications), patient education materials (e.g., 
diagnoses, legal documents, community resources), and CI management (e.g., diagnostic 
assessments, lab values, medications). The intervention will be active at HealthPartners for two 
years.   
Sample Size and Population  
Eligible patients are primary care patients >65 years of age with no CI diagnosis at the time of 
an index visit at a randomized clinic and have elevated risk of CI. Elevated risk is 1) an abnormal 
score on any comprehensive cognitive assessment; 2) a positive screen (MiniCog score <3) 
with no follow-up comprehensive cognitive assessment; or 3) >=15% risk of a CI diagnosis in 
the next 3 years as estimated by the MC-PLUS algorithm developed in the R61 phase. 
Preliminary data suggest at least 3000 patients at randomized HealthPartners primary care 
clinics will meet eligibility criteria over a 1-year accrual period.
NCT05723523

STUDY TEAM ROSTER  
Investigator 
Role 
Contact Information 
Main responsibility/Key 
Roles 
Leah 
Hanson, PhD 
Co-Principal 
Investigator 
HealthPartners Neuroscience Center 
295 Phalen Blvd, MS41200A 
Saint Paul, MN 55130 
Phone: 651-495-6352 
Fax: 651-495-6375 
Leah.R.Hanson@HealthPartners.com 
• 
Share leadership with Dr. 
Rossom 
• 
Assure adherence to 
protocol 
• 
Ensure ongoing 
communication with 
IRB/funding agency 
Rebecca 
Rossom, 
MD, MSCR 
Co-Principal 
Investigator 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55425 
Phone: 952-883-5466 
Rebecca.R.Rossom@HealthPartners.com 
• 
Share leadership with Dr. 
Hanson 
• 
Assure adherence to 
protocol 
• 
Liaison to HealthPartners 
medical group 
Lauren 
Crain, PhD 
Co-
Investigator 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55425 
Phone: 952-967-5354 
Lauren.A.Crain@HealthPartners.com 
• 
Variable definition input 
• 
Data collection planning  
• 
Data safety plan  
• 
Statistical analysis plan  
Meghan 
JaKa, PhD, 
MS 
Co-
Investigator 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55425 
Phone: 952-967-5208 
Meghan.M.JaKa@HealthPartners.com 
• 
Dyad interviews 
• 
Surveys 
• 
Dissemination and 
manuscript 
Michael 
Maciosek, 
PhD 
Co-
Investigator 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55425 
Phone: 952-967-5073 
Michael.V.Maciosek@HealthPartners.com  
• 
Economic analysis 
• 
Dissemination and 
manuscript 
Patrick 
O’Connor, 
MD, MA, 
MPH 
Co-
Investigator 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55425 
Phone: 952-967-5034 
Patrick.J.OConnor@HealthPartners.com 
• 
Assist in CDS design and 
implementation 
• 
Assist in interpretation of 
data and preparation of 
manuscripts 
Bethany 
Crouse, PhD 
Project 
Manager 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55455 
Bethany.K.Crouse@HealthPartners.com 
• 
Maintain regulatory 
documentation and 
reports to regulatory 
agencies 
• 
Support project activities 
Heidi 
Ekstrom, MA 
Project 
Manager 
HealthPartners Institute 
8170 33rd Ave S, MS2112R 
Bloomington, MN 55455 
Heidi.L.Ekstrom@HealthPartners.com 
• 
Support project activities 
• 
Liaison to replication site 
Aleta Svitak, 
MS 
Project 
Manager 
HealthPartners Neuroscience Center 
295 Phalen Blvd, MS41200A 
Saint Paul, MN 55130 
Aleta.L.Svitak@HealthPartners.com 
• 
Support project activities 
• 
Support ongoing R61 
activities 
NCT05723523

STUDY OBJECTIVES 
The overarching aim of this study is to implement a clinical decision support (CDS) tool for 
identifying and managing cognitive impairment (CI) in a pragmatic clinic-randomized trial to 
evaluate its impact on CI detection, clinician perceptions and healthcare utilization among 
patients with elevated risk for CI in primary care clinics. 
In the R61 phase of this study, our team developed and built the CI-CDS system that will 
make CI a priority to address at primary care office visits for patients with elevated risk for CI. 
The CI-CDS system is comprised of two interdependent processes that gather and assemble 
volumes of electronic health record (EHR) data to 1) offer clinicians tailored, point-of-care 
suggestions and tools for diagnosing and managing mild cognitive impairment (MCI) and 
dementia, and 2) estimate the likelihood of a future dementia diagnosis. The CI algorithms 
extract and process clinical data elements (e.g., medications, diagnoses, laboratory values, 
screening results) from up to 5 years prior to provide clinicians with suggested actions to take 
during visits with patients at risk for CI that are informed by comprehensive, up to date 
information. The MC-PLUS prediction model gathers data from EHR sources beyond 
screening results (e.g., utilization patterns, laboratory testing history, diagnosis history from 
the prior 2 years) to accurately estimate 3-year risk of a dementia diagnosis. The CI algorithms 
and MC-PLUS were added to an existing successful web-based CDS platform that is 
seamlessly integrated in the HealthPartners EHR, resulting in the CI-CDS system. CI-CDS 
was developed and implemented in partnership with primary care leadership to ensure 
seamless integration with primary care workflows and inclusion of preferred instruments, 
materials, and care recommendations. Our team also successfully designed and piloted a 
survey of primary care clinicians during the R61 phase to identify clinician-reported barriers to 
diagnosing cognitive impairment and managing care, as well as the potential utility of a system 
such as CI-CDS. These survey findings are summarized in Section 15.3.  
In this R33 phase of this study, we will conduct a pragmatic clinic-randomized trial to evaluate 
the effectiveness of the CI-CDS system in improving care for patients with elevated risk for CI 
by making it easier for clinicians to assess, diagnose, and manage cognitively impaired 
individuals. The study will randomize clinics to implement the CI-CDS system (CI-CDS) or to 
continue with usual care (UC). In the intervention clinics, the CI-CDS will use data stored in 
the EHR to identify patients with elevated risk for CI; assemble treatment recommendations 
tailored to each eligible patient’s current needs; display these recommendations to primary 
care clinicians via the CI-CDS user interface; and store analytic data from all targeted visits. 
In UC clinics, the CI-CDS will run invisibly in the background to identify eligible patients, 
assemble tailored treatment recommendations, and store analytic data from all targeted visits. 
The specific aims of the R33 phase of this trial are: To evaluate the effect of the CI-CDS 
system on rates of CI detection, clinician confidence, and healthcare utilization costs in a 
pragmatic cluster-randomized trial of primary care clinics randomized to CI-CDS or UC. 
H1: Patients with elevated CI risk at index visits in CI-CDS compared to UC clinics will 
have significantly higher rates of CI detection as indicated by EHR documentation of CI 
diagnosis in up to 24 months of follow up.  
H2: Clinicians at CI-CDS compared to UC clinics will have significantly more confidence 
in diagnosing and managing CI, as assessed through clinician surveys.  
H3: Among a subset of insured patients with elevated CI risk at index visits, those at CI-
CDS clinics will have significantly lower healthcare utilization costs related to emergency 
NCT05723523

room and inpatient visits in the follow-up period compared to similar patients at UC clinics. 
In addition to hypothesizing that the CI-CDS will increase the likelihood that CI is recognized 
among accrued patients in the intervention clinics, we will assess the extent to which the CI-
CDS system increases the likelihood that clinicians more actively take appropriate clinical 
actions to manage patients with CI or elevated risk for CI.   
One mechanism by which the CI-CDS system is thought to be effective in encouraging 
clinicians to take more action is by reducing perceived barriers to CI care. We will explore this 
possibility by surveying clinicians from CI-CDS clinics about how useful or accessible they 
found the CI-CDS system, and the extent to which the system supported them in their 
intentions to treat patients with CI or elevated risk of CI. We will also assess clinician attitudes 
and perceptions of control, perceived norms, and barriers to CI care among clinicians in CI-
CDS relative to UC clinics. 
Treatment effectiveness may vary as a function of contextual factors or patient characteristics. 
Qualitative semi-structured interviews will be conducted with newly diagnosed patient/care 
partner dyads to explore perspectives on experience with the assessment and diagnosis 
process as well as preparedness for subsequent management. Learnings will be used to 
inform future implementation and to design extensions of the CI-CDS related to disease 
management. In secondary analyses, contextual factors and patient characteristics will be 
assessed for the extent to which the CI-CDS system is differentially effective (i.e., treatment 
heterogeneity) across patient subgroups, or relatedly, whether the CI-CDS intervention can 
reduce pre-intervention disparities in CI care.  
1. BACKGROUND AND RATIONALE  
1.1 Background on Condition, Disease, or Other Primary Study Focus 
Cognitive impairment (CI) is an urgent global health problem affecting more than 47 million 
people.[1] The prevalence of Alzheimer’s disease (AD) and related dementias (ADRD) is 
expected to triple by 2050 at an annual estimated cost of $818 billion.[2, 3] While CI is 
generally not curable, many interventions can transform the lives of people with CI and their 
families through maximizing cognition, decreasing distress associated with symptoms and 
crises, educating patients and caregivers and improving quality of life, but timely diagnosis is 
essential for these interventions.[1] In addition, family members and caregivers report a lower 
quality of life and have higher rates of medical problems, depression, anxiety and work 
absence.[4-6] Unfortunately, the diagnosis of CI is often not made until the disease is well 
advanced and the patient is in crisis, resulting in poor quality of life for both the individuals 
with CI and their relatives.[4, 7-9] In fact, most people with CI and their families are relieved 
when a diagnosis is finally made, and most people report wanting to be told if they were 
thought to have CI.[10] The problem is compounded by a care delivery system that is already 
stressed, with competing demands and limited access to specialists.[11] Primary care 
clinicians often lack the confidence or time to evaluate CI, tending instead to refer to 
specialists, even though many patients do not follow through with such referrals.[12-14] Aside 
from time and reimbursement, other commonly reported barriers to CI diagnosis and care 
include a lack of consensus for what constitutes adequate evaluation, which assessment tools 
to use, what type of brain imaging is most appropriate, and when referrals to specialists are 
indicated. [15-17] Given these ambiguities, it is not surprising that many clinicians report they 
would be moderately to extremely likely to treat more patients with CI rather than referring 
them to specialists if decision support tools were available. 
NCT05723523

1.2 Study Rationale 
Barriers to CI Detection and Care: The United States Preventive Service Task Force does 
not recommend for or against CI screening due to insufficient evidence that screening leads 
to benefits for patients and caregivers.[16] In the absence of universal screening, most 
clinicians rely on case finding based on observed behaviors or concerns raised at the time of 
encounters.[18] To assess for CI, there are over 22 cognitive screening tests available.[19-
21] The MiniCog is an example of a simple and brief cognitive screening tool, with sensitivity 
ranging between 76-100% in different study settings and specificity ranging between 27-
85%.[22, 23] However, there is evidence that screening alone is insufficient to alter patterns 
of clinician practice for subsequent CI detection, evaluation, referrals, or treatments for CI. For 
example, relevant physician action only occurred in 17% of patients with a positive MiniCog 
screen in primary care.[24] Similarly, in our health system, there were no documented clinical 
actions in two-thirds of patients with a positive MiniCog screen completed at Annual Medicare 
Wellness (AMW) exams and neurology visits.[25] Reported primary care barriers to CI care 
management include lack of perceived control over diagnosis and care, inadequate time, 
difficulty accessing specialists, low reimbursement, difficulties connecting with social services, 
and need for clinical decision support (CDS).[12, 13, 26] Many clinicians rely solely on history 
and routine examination to make the diagnosis without using standardized CI testing, even 
though only 4% of clinicians said testing would not be of value.[27] If decision support tools 
were available, the features they said they would value most were choice of screening 
assessment, guiding the diagnostic process, and guidance for patient management and 
ongoing care.[27] Current EHR systems do not actively give clinicians many tools for CI 
diagnosis and care management. A CDS system that could selectively alert clinicians as to 
when cognitive assessments are warranted, along with CDS recommendations and tools to 
guide them, could lead to greater clinician confidence to assess, diagnose and manage CI 
and could address some of the time barriers.  
Our previous work developing and implementing CDS to improve care outcomes is guided by 
Wagner’s well-vetted and much-used Chronic Care Model, which posits that optimal care 
occurs when a “prepared practice team” encounters an activated patient.[28] It is no surprise 
that Wagner suggests that innovative use of information systems are foundational elements 
of successful care.[29, 30] Without, the complexity required to deliver evidence-based and 
appropriate CI (or other chronic disease) care in a timely way is an almost superhuman 
task.[31, 32] CI-CDS can prepare the clinician for the visit and present a set of evidence-
based options related to CI diagnosis and care, thus promoting shared decision making and 
patient-centered care.[33] 
How a Machine Learning Approach Could Help Overcome Barriers: The MiniCog is a 3-
minute screening instrument with fairly high rates of false positives (specificity of 27-85%). 
[16, 22] If the MiniCog is positive, more detailed testing is recommended to assess potential 
CI, provide more information about affected cognitive functions, and to establish a basis for 
tracking change with tools such as the Montreal Cognitive Assessment (MoCA), Mini Mental 
Status Exam (MMSE), or the St. Louis University Mental Status Examination (SLUMS).[24, 
34, 35] Longitudinal testing can provide insight in some circumstances, with rates of declines 
on some of these tests predictive of AD or MCI.[36] Prediction models using machine learning 
have been advocated as a way to more accurately predict risk. One newer prediction model, 
called eRADAR, with good discrimination (C statistic = 0.78, 95% CI: 0.76-0.81) and validation 
(C = 0.81, 95% CI: 0.78-0.84), was developed using a prospective cohort of research 
participants undergoing standardized cognitive testing every 2 years. It predicted 
undiagnosed dementia from available EHR data on the research cohort diagnosed with 
NCT05723523

dementia.[37] In our application, we suggested that a prediction model using CI screening 
data in combination with EHR data, developed and validated in a cohort of patients from a 
primary care setting, could have even greater predictive accuracy. Furthermore, while the 
eRADAR model predicts incident undiagnosed dementia in a research population, there may 
be additional clinical value to detecting future risk of being diagnosed with dementia in real 
world settings. In the R61 phase of this study, we developed and tested a prediction model 
for CI that attained a similar predictive value (C statistic = .80, 95% CI: .79, .81) as the 
eRADAR in adults in primary care settings. The MC-PLUS model estimates risk of dementia 
diagnosis in 3 years at a single visit using commonly available EHR data and requiring no 
prior cognitive screening.   
Components of Evidence-Based CI Management for Developing CDS Content: Despite 
the lack of a “one-size fits all” approach to CI management, there is broad consensus and 
solid evidence that multidomain assessment and management are needed for optimal 
management of CI over time. These include:  
1. Diagnostic Assessments: A cognitive history with input from a knowledgeable 
independent source is key to identifying and diagnosing CI. In addition, the American 
Academy of Neurology recommends routine neuroimaging as part of all workups for 
CI.[38] MRI is often preferred because of greater sensitivity and ability to differentiate 
dementia subtypes, such as vascular dementia.[39, 40] Other assessments routinely 
recommended to search for possible confounding problems, well within the score of 
primary care practice, include medication review for cognition-impairing medications and 
nutritional, endocrine, and metabolic disorders that can contribute to CI.[1]  
2. Needs Assessment: A needs assessment can identify cognitive, emotional, 
psychological, and social needs as well as safety risks that if addressed can enable 
patients to have more autonomy and maximize potential for remaining in the community 
for as long as possible.[41, 42] Risks may include medication noncompliance, fall, fire, 
driving safety, and vulnerability to financial scams.[41] Driving safety is often one of the 
most contentious and difficult challenges for both clinicians and caregivers, and many 
clinicians need to be made aware of resources available to address this effectively.[43]  
3. Caregiver and/or proxy identification: It is helpful for patients to choose a proxy or 
someone who can enact pre-specified wishes or choices consistent with their values 
while they retain some ability to consider future scenarios.[44, 45] In addition, keeping 
an updated release of information on file is recommended.  
4. Medications for cognition and affective, psychotic or behavioral symptoms: 
Cognition enhancing medications may be helpful in mild to moderate dementia when 
tolerated and potentially improve everyday functioning.[46] Depression, apathy, anxiety, 
and sleep disturbances are very common.[47] Psychotic symptoms [48] such as 
delusions and visual hallucinations (most common in Lewy body dementia) occur in 
about 18% of patients with dementia.[49, 50] Use of antipsychotic medications aren’t 
generally recommended for non-distressing symptoms that can be managed with 
caregiver education or other classes of medications.[51] Agitation and behavioral 
disturbance are also common, complicate family care, [52-54] and can increase the cost 
of care by 12%.[55] Measures to address them such as caregiver education and 
behavioral management training along with careful prescribing of medications, if 
systematically implemented, can mitigate the higher costs associated with behavioral 
disturbances.[55] Management can include coaching caregivers’, communication skills, 
NCT05723523

use of antidepressants, or interventions to increase pleasant activities or social 
engagement.[1, 47] [56-59]  
5. Support for caregivers: Family caregivers are the most important resource for many 
patients living with CI. Family caregivers suffer more often from depression, anxiety, 
work absence, and lower quality of life.[5, 6, 60] Care should be directed to support the 
needs of both patients and caregivers.  
Features of Successful CDS: In reports issued 25 years ago, the Institute of Medicine 
asserted that EHR technology would lead to substantial improvements in quality of medical 
care in America.[61, 62] Since that time, private and public investments in EHR systems have 
totaled hundreds of billions of dollars, with most CDS interventions until very recently unable 
to demonstrate improvements in chronic disease outcomes other than process measures.[63-
71] Much of what has been learned in the last 25 years about how to implement effective CDS 
in primary care settings for chronic diseases, such as diabetes and hypertension, can inform 
the design, implementation, and maintenance of CDS systems designed to address the 
complexity of CI care. The CDS platform we developed (prior to inclusion of CI-CDS) was 
informed by extensive input from primary care clinicians who would be users of the technology 
and by our careful analysis of numerous antecedent failed CDS efforts to improve chronic 
disease care in primary care settings.[63, 65, 66, 72-75] We deduced from this analysis that 
a CDS strategy must have the following key features in order to be successful: (a) CDS must 
be available in real-time at the point of care and be based on all available clinical data;[71] (b) 
CDS must be sophisticated and not limited to general prompts or reminders;[63, 66, 76] (c) 
CDS must be integrated into the routine workflow of the primary care clinician and use rooming 
staff and nurses to the fullest extent possible; (d) CDS should be provided to both patients 
and clinicians and implemented in a way that engages patients and caregivers and promotes 
shared decision-making;[77-79] (e) CDS must increase clinician efficiency and save them 
clicks or time; (f) CDS use must be sustained at high rates to affect clinical outcomes. High 
CDS use is achieved through clinic staff training, ongoing communication, and feedback to 
care teams on their use rates.  
2. STUDY DESIGN 
This pragmatic clinic-randomized trial will evaluate the effectiveness of the CI-CDS system in 
improving care for patients with elevated risk for CI by making it easier for clinicians to assess, 
diagnose, and manage cognitively impaired individuals. The study will be conducted in at least 
30 HealthPartners primary care clinics, where at least 3000 patients with elevated risk for CI 
are estimated to receive care.  
NCT05723523

Clinics will be randomized equally to the CI-CDS 
or 
UC 
treatment 
group 
using 
simple 
randomization. CI-CDS clinics will fully implement 
the CI-CDS system.  UC clinics will silently 
implement the CI-CDS system at the same time 
as CI-CDS clinics so that it operates undetectably 
in the background. Following intent-to-treat 
principles, all eligible patients will be attributed to 
the treatment group to which their clinic is 
randomly assigned.  
The 12-month patient accrual period will begin at 
the same calendar time in all clinics when the CI-
CDS system is implemented; full implementation 
in CI-CDS clinics and partial implementation in UC 
clinics. The up to 24-month intervention period will 
begin in all clinics at the same time as the accrual 
period..   
Each patient will accrue into the study at their 
index visit. The index visit is the first visit during the accrual period at a randomized clinic at 
which a patient meets all intervention eligibility criteria. The index visit date will mark the 
beginning of each accrued patient’s observation period. Patients with an index visit near the 
beginning of the accrual period will have up to a 24-month observation period, and those with 
an index visit near the end of the accrual period will have an approximate 12-month 
observation period.  
Patients who have an index visit, and thus have accrued into the study, may have additional 
visits at randomized clinics prior to the end of the intervention period. The CI-CDS system is 
programmed to assess intervention eligibility at all visits at all clinics in both treatment groups 
for the duration of the intervention period, and to display tailored CI-CDS for eligible post-
index visits in CI-CDS clinics only.   
Replication: In addition to the primary effectiveness trial, the CI-CDS system will be 
implemented and evaluated at OCHIN, a network of safety net health centers clinics across 
the country. organized under a 501(c)(3) designation. Across the country, OCHIN’s highly 
customized, centrally hosted EpicCare ambulatory system EHR is used in a broad network of 
primary care safety net settings including Federally Qualified Health Centers (FQHCs) and 
FQHC lookalikes, rural health centers (RHCs), school-based health centers, and county 
health departments. The replication site study design will be similar in that service areas with 
constituent primary care clinics will be randomized equally to a CI-CDS or UC treatment group. 
The CI-CDS system will be fully implemented in CI-CDS clinics and silently implemented in 
UC clinics (i.e., working in the background to collect data but not visible to clinicians). Patients 
will accrue into the study at index visits over a 6-month accrual period, with an approximately 
6-month observation period for the last accrued patient, resulting in an ~12-month intervention 
period. See replication site-specific protocol details in Section 15.5. 
NCT05723523

3. SELECTION 
AND 
ENROLLMENT 
OF 
PARTICIPANTS 
Primary care clinics. HealthPartners leaders have 
agreed to enroll at least 30 primary care clinics from the 
Twin Cities metropolitan area into this clinic-
randomized effectiveness trial. The co-PIs will 
collaborate with HealthPartners leaders to identify 
specific clinics to enroll with the intention that study 
clinics are representative of primary care environments 
at this and other health systems. Clinics that are likely 
to close or transfer ownership during the study period 
will be excluded from participation. All study clinics are 
currently using a version of the CDS platform for non-
CI-related clinical domains with high use rates.  
The co-PIs will collaborate with the replication Lead Co-
Investigator to identify candidate clinics to enroll with 
the intention that they be representative of primary care 
environments at the replication site and other health 
systems. The replication site team will collaborate with 
health system leaders to identify the specific clinics to 
enroll. Replication site clinics are also using a version 
of the CDS platform for non-CI-related clinical domains. 
Primary care clinicians. All primary care clinicians 
practicing in clinics randomized to the intervention will 
be able to use the CI-CDS in their care of patients. To 
be study-eligible, a primary care clinician must be a 
family physician, general internist, or adult-care non-
obstetric nurse practitioner (NP) or physician assistant (PA) working at a randomized clinic. 
Eligible primary care clinicians will be invited to complete pre-and post-implementation 
surveys assessing their comfort and confidence diagnosing and managing CI, perceived 
barriers to providing CI care (H2), and in intervention clinics, their perceptions of the CI-CDS. 
The clinician survey process will not be implemented at the replication site.  
Patients. The study populations whose data will be included in the primary effectiveness 
analyses and in the replication analyses are the patients who have index visits at the CI-CDS 
and UC clinics during the accrual period at the primary and replication sites. Patients will be 
automatically screened by the CI-CDS algorithms for intervention eligibility at all primary care 
visits at the randomized clinics during the intervention period. Patients who have a visit that 
meets the minimal intervention eligibility criteria during the accrual period will be considered 
accrued into the study and included in the analyses to maximize generalizability.  
3.1 Inclusion Criteria  
Patients are accrued into the study upon completion of an index visit. All of the following 
intervention eligibility criteria must be met on the visit date for the visit to be an index visit:  
 
Primary care office visit at a randomized clinic during the accrual period, and 
Definitions  
Intervention Period 
 CI-CDS implementation date 
through a predetermined time at 
which the interface will be turned 
off in the intervention clinics  
Accrual Period 
 CI-CDS implementation date 
through a predetermined time at 
which no more people will be 
included in the analytic sample 
Index Visit  
 Varies by patient 
 First visit during accrual period 
at which the CI-CDS identifies 
that the patient is study eligible 
 Patient is accrued into study at 
index visit 
Observation Period 
 Varies by patient 
 Index visit date through a 
predetermined time at which 
outcome data will no longer be 
accumulated 
NCT05723523

 
Patient is age 65 or over, and 
 
Patient has no CI diagnosis documented in the EHR prior to the visit, and 
 
Patient has  
o Any abnormal score on a comprehensive cognitive assessment (MoCA, MMSE or 
SLUMS) in the prior 18 months, or  
o MiniCog score <3 in the prior 18 months and there is no evidence of a subsequent 
comprehensive cognitive assessment (MoCA, MMSE, SLUMS), or  
o No cognitive assessment in the past 18 months and risk of a dementia diagnosis 
in the next 3 years >=15% as calculated by the MC-PLUS algorithm developed in 
the R61 phase, and 
 
First visit during the accrual period at which all prior inclusion criteria are met. 
3.2 Exclusion Criteria  
Any of the following exclusion criteria will prevent a visit from being intervention eligible, and 
therefore not considered for index visit status:  
 
Patient has received active parenteral chemotherapy within the last year 
 
Patient has stage 4 or equivalent cancer diagnosis 
 
Patient is enrolled in hospice care or palliative care programs 
All women and members of racial or ethnic minority groups and their subpopulations who 
meet the above eligibility criteria will be included in accordance with the National Institutes of 
Health (NIH) Policy on Inclusion of Women and Minorities as Participants in Research 
Involving Human Subjects.  
3.3 Study Enrollment Procedures  
All primary care visits that take place at the randomized clinics after the CI-CDS system is 
implemented will be screened for intervention eligibility. The web platform in which the CI-
CDS is housed is programmed to assess whether CDS content should be displayed each time 
the vitals section of the EHR is closed by extracting data elements from the EHR and 
supplying them to algorithms that assess eligibility. For the CI-CDS, this means that the 
algorithms run in all randomized clinics at all primary care visits. They confirm that the person 
is age-eligible, search for prior cognitive screening and assessment data, run the MC-PLUS 
algorithm to estimate risk of CI, and confirm that the patient does not already have a diagnosis 
of CI and that none of the exclusion criteria apply. In short, intervention eligibility is 
systematically assessed at all visits with all patients at the randomized clinics where CI-CDS 
is running. The platform documents all eligibility assessments (i.e., eligible and ineligible) so 
that the characteristics of visits and patients screened for eligibility can be quantified as each 
eligibility criterion is applied. Patients will be accrued into the study on the date of their first 
visit during the accrual period that meets all intervention eligibility criteria (i.e., index visit), and 
followed for the duration of the observation period.  
NCT05723523

We have received a waiver of informed consent for patients from the HealthPartners Institute 
Institutional Review Board (IRB) because the study is minimal or less than minimal risk 
compared to the risk associated with any primary care encounter. This was similarly granted 
for the R61 Phase. The evidence-based care recommendations in the CI-CDS intervention 
are from national guidelines and approved by primary and neurology care delivery leadership. 
Moreover, this pragmatic intervention could not be conducted if written informed consent were 
required to be obtained in the context of busy community-based primary care clinic 
environments. Note that patients who have requested non-participation in research studies 
will be excluded from all analyses. 
Primary care clinicians in the intervention clinics will be encouraged but not required to use 
the CI-CDS with eligible patients, so that the decision to use or not use CI-CDS at a given 
clinical encounter is up to the clinician. All primary care clinicians at the primary site will be 
asked to consent to and complete pre-and post-implementation surveys and may receive 
modest compensation for completing these surveys. We have received a waiver of written 
informed consent for clinicians from the IRB for the intervention because (a) the CI-CDS is 
based on current national standards of CI identification and management and does not make 
any treatment recommendations that are not accepted as community standard of care, and 
(b) consenting clinicians would compromise the external validity of the study by introducing 
selection effects. Measures to protect the privacy of clinicians participating in this study are 
described further below.  
5.  STUDY INTERVENTIONS 
5.1 Interventions, Administration, and Duration  
The CI-CDS system is a system-level intervention comprised of 1) a passively operating web 
service that gathers, analyzes, and retains information pertinent to every web service call, and 
2) a user interface that presents a summary of algorithm results to front end users. The web 
service documents every automatically triggered web service call made in all randomized 
clinics so that data pertinent to all visits from all intervention eligible and accrued patients will 
be identically collected and available across treatment groups. The user interface will function 
as the intervention delivery vehicle and be operative only in intervention clinics. The CI-CDS 
system will therefore operate passively in all randomized clinics, but the user interface will 
only be turned on in clinics that are randomly assigned to the CI-CDS group.  
The CI-CDS system will be implemented in all randomized clinics to use data stored in the 
EHR to invisibly identify visits that are eligible for the CI-CDS intervention, assemble treatment 
recommendations tailored to the needs of each patient at eligible visits and store analytic data 
from all visits. In clinics randomized to the intervention group, the CI-CDS will display the 
treatment recommendations to primary care clinicians and document clinician actions via the 
CI-CDS user interface. The CI-CDS user interface will be disabled in clinics randomized to 
the control group. 
The CI-CDS system is programmed to support primary care clinicians in assessing patients 
with elevated risk for CI and then diagnosing and managing care for patients with CI. The CI-
CDS user interface will provide updated clinical recommendations at primary care encounters 
for patients with elevated CI risk or with a CI diagnosis. The interface will enable the user to 
administer diagnostic screening exams (e.g., Montreal Cognitive Assessment (MoCA), Patient 
Health Questionnaire (PHQ-9)), place quick orders (e.g., referrals, procedures, lab 
assessments, medications), accurately diagnose CI, provide patient education materials (e.g., 
NCT05723523

diagnoses, legal documents, community resources), and manage CI (e.g., visualize trends in 
screening exams, lab values, medications). See Section 15.4 for more detailed content 
information. This approach presents patients and their primary care clinicians multiple 
encounter opportunities to consider an evolving array of timely, evidence-based treatment 
options to improve CI care. Operationally, implementing CI-CDS at clinics requires a series of 
distinct steps:   
Step 1: Data Exchange. The EHR securely exchanges data with the web service at every 
encounter of adult patients aged 65 or older and is currently triggered by blood pressure entry 
at CI-CDS and UC clinics. Several measures are in place to ensure security of protected 
health information. Data transfer to the web service uses a Simple Object Access Protocol 
(SOAP) with Secure Sockets Layer (SSL) encryption over a Hypertext Transfer Protocol 
Secure (HTTPS) computer network. There is a double firewall in the web service; once data 
flows through the initial web service firewall, the data crosses another firewall into a new 
secure pathway that also employs SOAP, SSL, and HTTPS for processing. The data are sent 
through a batch server for more efficient processing, all within the double firewall web service. 
A unique study ID unrelated to subject-specific information is assigned at the index visit, saved 
in the EHR, and exchanged with the web service at subsequent encounters. Limited data is 
saved in an analytic data set for study purposes using the study ID to reduce the risk of breach 
of confidentiality.  
Step 2: Algorithm Driven Assessment. The web service will use EHR data and the MC-
PLUS prediction tool to identify individuals with elevated CI risk at primary care encounters 
and to create a flag which is returned to the EHR for use in a “best practice alert” (BPA) pop 
up to rooming staff. The algorithms will also identify patients with CI who have outstanding 
care opportunities even if they are not eligible for inclusion in the primary analyses. The data 
is run through coded algorithms within the web service to determine the content of clinical 
care suggestions that will be displayed in engagement tools for step 3 and to inform the 
content within the online CDS interface, outlined in Section 15.4.  
Step 3: Patient/Caregiver and Clinician Engagement at Primary Care Office Visits. At 
visits with patients with elevated CI risk or with identified CI-related care opportunities, the 
rooming staff receives an immediate BPA to print CDS engagement materials. Using this 
sequence of steps successfully implemented in previous studies, the rooming staff will print 
the materials from a link on the BPA and give the lay version to the patient/caregiver to review 
while waiting for the clinician. A professional version is provided to the clinician to review 
before entering the exam room. This approach has been well-liked by our clinicians to help 
them be prepared to engage patients in their care needs before the clinician-patient 
interaction.  
Step 4. Online CDS for Clinicians with More Detail and Tools to Promote Efficiency. 
Clinicians can opt to view CDS and more detailed materials and tools online through an activity 
that is seamlessly integrated into the EHR and can be viewed in real time for any patient at 
any time they choose. Newer EHR features make it easier for the web-based application to 
be interactive with the end user. As the user interacts with the CDS interface, a summary is 
generated that captures patient CI risk or diagnosis, results of additional assessments 
completed, and actions recommended. This summary can then be copy and pasted into the 
clinician’s progress note. Our team includes very experienced EPIC programmers employed 
by our research institute who take full advantage of new functionality as it becomes available 
to make the CI-CDS interface and interactive features as efficient as possible for clinicians.  
NCT05723523

5.2 Handling of Study Interventions  
The CI algorithms that reside in the CI-CDS system consistently and universally assess 
automatically generated web calls for intervention eligibility. When a patient is deemed eligible 
for CI-CDS content, the algorithms use up to 5 years of clinical information stored in the EHR 
to assemble the intervention content (treatment recommendations) that should be presented 
to the clinician. Display rules determine whether the visit is occurring in a clinic randomized to 
present the intervention content to the clinician (i.e., intervention) or not (i.e., control). Trace 
data documenting the results of each of these sets of algorithms are stored in the web service. 
The study team retrospectively extract the trace data from the web service to know how many 
index visits or intervention-eligible visits have occurred and with how many patients, and to 
know the content assembled for each of these visits. 
The study team is responsible for developing and testing the CI-CDS system algorithms. Once 
programmed, though, all intervention delivery activities are managed by the CI-CDS system 
algorithms with no involvement from the study team. The CI-CDS system is an intervention 
that supports clinicians by efficiently aggregating and processing a broad spectrum of patient-
focused data and then offering the clinician guideline-based recommendations based on 
those data. The clinicians, who are not part of the study team, have discretion with respect to 
whether and how much of the presented information to act on or share with the patient.    
5.3 Concomitant Interventions 
5.3.1 Allowed Interventions 
As this pragmatic trial occurs in primary care clinics, and the intervention is provided in 
addition to any patient’s routine care, any and all medications and other treatments and 
interventions ordered by a patient’s routine care providers are allowed in the course of this 
trial. 
5.3.2 Required Interventions  
There are no required concomitant interventions in this clinic-based pragmatic trial.  
5.3.3  Prohibited Interventions 
There are no prohibited interventions in this clinic-based pragmatic trial.  
5.4 Adherence Assessment  
Clinician training. All clinician and staff training activities will be conducted collaboratively with 
quality improvement leaders at participating health systems through webinar training sessions 
for CI-CDS clinic staff, including clinicians, nurses, and rooming staff. The training will focus 
on the workflow, scripting when CI engagement tools are introduced to patients or caregivers, 
and the CI-CDS features and content. The training will emphasize that the CI-CDS is meant 
to supplement, not replace clinical care, and that primary care clinicians should use their 
clinical judgement to use the tool or follow its treatment suggestions. Training will be brief 
(about 50 minutes) and modeled after successful training programs conducted in our previous 
CDS studies. The webinar will be taped and available online for asynchronous viewing, and 
written materials will also be available. We will also provide information about the study and 
interventions regularly in our online newsletters and communications. Rooming staff printing 
NCT05723523

procedures for CDS tools are incorporated into the standard care model process that is part 
of all new staff training with periodic refreshers to existing staff.  
Tracking CI-CDS use. Multiple processes are measured and tracked throughout the 
intervention period to assess CDS use as described by elements in the NIH Treatment Fidelity 
framework including delivery, receipt and enactment of intervention dose and content.[80] 
Print rates by rooming staff will be measured, grouped by clinic and clinician, and reported 
monthly to clinic leadership. The goal of printing the patient and clinician engagement 
materials will be set at ≥70% of targeted encounters. Using this strategy, we have maintained 
print rates for the existing CDS engagement interfaces at over 75% for many without financial 
incentives. We chose 70% as the target use rate given that at any given encounter, there may 
be more pressing issues or psychosocial factors that should be addressed instead, such as 
an acute care issue or death of a spouse, and clinicians are to use their clinical judgement 
about when use of the CI-CDS is appropriate. If necessary, supplemental communication and 
training will be provided to clinics and/or clinicians who have low print rates relative to other 
clinics or clinicians. We will also track clinician use of online CDS features by monitoring click 
rates on tools, quick orders, and educational materials. However, as clinicians are encouraged 
to use their clinical judgement to determine when and how to use the CI-CDS tool, we will not 
be providing clinic- or clinician-level feedback on click rates.  
6. STUDY PROCEDURES 
6.1 Schedule of Evaluations 
All patient-focused data used to evaluate the effectiveness of the CI-CDS system on primary 
and secondary outcomes will derive from the CI-CDS system itself (e.g., eligibility criteria at 
index visit), the EHR (e.g., CI diagnosis, patient characteristics at index visit) or health care 
claims databases (e.g., paid amounts for emergency room visits or inpatient stays). There is 
no scheduled prospective data collection directly from accrued patients. Rather, data 
elements that document the process, outcomes and costs of care delivery will be recorded in 
these electronic resources as care is delivered to patients, with the timing of care delivery 
driven by clinician and patient judgment. This clinic-based pragmatic trial will not direct the 
timing, manner or amount of care delivered to any accrued patient according to a pre-defined 
schedule.  
6.2 Description of Evaluations 
6.2.1 Screening  
The CI-CDS system is programmed to evaluate intervention eligibility each time a web service 
call is made from a randomized clinic during the accrual and intervention periods. Eligibility 
screening will therefore be consistently carried out and thoroughly documented for all patient 
visits as part of routine care at all randomized clinics. All primary care visits during the accrual 
period may be characterized with respect to basic patient demographics (e.g., age, sex, 
racial/ethnic group) and all eligibility criteria. Eligibility criteria are automatically extracted from 
the EHR when rooming staff enter a measured blood pressure into the EHR and the result of 
eligibility assessment is documented in less than one second. We have received a waiver of 
consent for use of the CI-CDS during clinical care because the CDS is consistent with 
recommended care guidelines and because conduct of the study would not be feasible without 
this waiver of consent.  
NCT05723523

6.2.2 Patient accrual and index visit assessment 
Index visit 
All primary care visits at randomized clinics during the accrual period will be assessed for 
intervention eligibility. Patients may be screened for eligibility more than once during this time 
period. The first visit at which the CI-CDS algorithms determine that a patient is intervention-
eligible during their clinic’s accrual period will be denoted as the patient’s index visit. The 
patient will be accrued into the study at the index visit. The index visit functions as a baseline 
to the extent that patients will be described / stratified according to characteristics that were 
documented at the beginning of the index visit, prior to potential intervention exposure.  
Index Assessment 
Patient information documented in the EHR or health plan claims databases on or prior to the 
index visit will be used to describe pre-intervention patient characteristics (e.g., age on index 
date, sex, risk of CI diagnosis at index visit), organize patients into groups to assess treatment 
heterogeneity (e.g., race / ethnic group, insurance status) or calculate pre-index values (e.g., 
healthcare utilization in pre-index year). All metrics needed for baseline characterization or 
reference are either already available in historical databases or will be documented in the 
course of care delivery. None will be assessed for solely for research purposes.  
Randomization 
The primary effectiveness trial and the study at the replication site are both cluster-randomized 
pragmatic trials. Accrued patients will be assigned to the randomly assigned treatment group 
(CI-CDS, UC) of the clinic at which their index visit takes place.  
6.2.3 Follow-up Visits 
There is no study-determined visit schedule. Patient visits with their primary care clinicians in 
randomized clinics will take place on a frequency determined by the patient and clinician, 
without any interference from the study team. Primary care visits that take place at any clinic, 
randomized or non-randomized, after the index visit and prior to the end of the observation 
period will be denoted as post-index visits. All index and post-index visits will be documented 
in the EHR so that data documented from each may be extracted and assembled into an 
analytic dataset. 
6.2.4 Completion/Final Evaluation 
There is no study-determined visit schedule, and therefore no predetermined final visit. All 
documentation of care delivery in the EHR or claims database between each patient’s index 
visit and the end of the observation period will be considered for inclusion in analyses, as 
appropriate.  
6.2.5 Clinician surveys  
As part of the R61 phase, primary care clinicians who practice in our 3 pilot clinics completed 
baseline surveys. Results of that survey are summarized in Section 15.3. 
In the R33 phase, a baseline clinician survey conducted at HealthPartners (the primary 
effectiveness site) will serve as a pre-implementation assessment of clinician self-reports of 
NCT05723523

confidence and management of CI. A post-implementation clinician survey will be fielded 9-
12 months after CI-CDS is implemented. Following extensive prior experience, 
HealthPartners Institute’s Center for Evaluation and Survey Research (CESR) will field 
surveys using an initial leadership-endorsed invitation email to all practicing clinicians in the 
randomized clinics followed by as many as seven reminder emails over two months. Using 
this approach and modest monetary incentives, we expect clinician response rates of 
60%.[81] 
6.2.6 Chart audits  
We will replicate procedures described in previous publications to measure rates of actions 
such as completion of cognitive assessments, orders for neuroimaging, referrals for 
neurocognitive testing, specialists, and orders for medications in the accrued population.[25] 
We will also audit the charts of 50-100 patients each in the CI-CDS and UC treatment groups, 
until saturation is reached. Chart audits are an important supplement to the EHR-based data 
to the extent that some clinician actions are difficult to obtain from fixed data fields in the EHR, 
including the presence and quality of care plans, caregiver identification, release of 
information authorization completion, and individualized recommendations for managing 
potentially distressful symptoms and safety issues.  
6.2.7 Patient / caregiver dyad interviews  
We will conduct semi-structured interviews with patient/caregiver dyads from randomized 
study clinics who are at elevated risk, recently screened, or newly diagnosed with CI to explore 
their experiences with screening, assessment, or diagnosis as well as their perceived 
preparedness for subsequent disease management. We will recruit study-accrued patients 
with recent study-eligible primary care index encounters to participate in the interviews. 
Following best-practice in qualitative data collection methods,[82] approximately twenty dyads 
will be recruited to participate. Interview guides will be designed in close partnership with 
qualitative and clinical experts to elicit rich qualitative patient and caregiver perspectives. A 
semi-structured design approach will be used consisting of 4-7 open-ended root questions 
each followed by a set of open-ended probes. Interviewers are trained and experienced in 
qualitative data collection within the health system setting. Interviewers will receive additional 
study-specific training related to CI care and the CI-CDS intervention. Practice interviews and 
on-going supervision will be conducted to ensure consistent, valuable data are collected. 
Detailed field notes will be collected using REDCap and, with permission, interviews will be 
recorded and transcribed. A thematic content analysis will be designed using combined 
inductive and deductive approaches to explore the perceived impact of clinic experience on 
addressed/unaddressed concerns, and resulting benefit/distress from screening, feelings 
about preparedness for future decisions, support, and distress. Learnings will be used to 
understand patient experience and inform future implementation and to design extensions of 
the CI-CDS related to disease management.  
7. SAFETY ASSESSMENTS  
7.1 Specification of Safety Parameters 
This pragmatic trial introduces the CI-CDS system as an intervention delivery tool to clinics 
randomized to an intervention treatment group, where it provides data-informed clinical 
decision support to clinicians as an adjunct to the usual care they provide at primary care 
visits. The CI-CDS system is designed to facilitate the provision of accepted standards of care. 
NCT05723523

With this work, we are not attempting to change the standard of care for managing patients 
with CI in primary care, but rather are attempting to help clinicians achieve this standard of 
care in CI management such as through screening patients for CI or recognizing CI in the 
EHR. Primary care clinicians will be trained that, as with other clinical decision tools, the CI-
CDS system is meant to supplement but not supersede clinical judgment. Clinicians will be 
able to choose to follow or to not follow the guidance of the CDS at any given time for any 
given patient.  
Clinicians will be asked to use the Feedback Tab within the tool to let the study team know of 
questions or potential errors in the CDS. Additionally, clinicians will be trained to let the 
research team know via the Feedback Tab when their clinical judgment is inconsistent with 
the CDS. This feedback will be monitored by the treatment team and the CDS algorithms 
adjusted if indicated. 
Actions suggested by the CI-CDS system are consistent with standard care for patients who 
meet intervention eligibility criteria, and therefore are unlikely to pose specific risks to their 
safety beyond those inherent to primary care. As such, we will not be monitoring safety events 
specific to the CI-CDS intervention or the content it delivers. Similarly, the intervention does 
not involve laboratory measures beyond those assessed in usual primary care, and as such 
we do not specify laboratory safety parameters. We will monitor emergency department visits, 
hospitalizations, suicide attempts, and deaths as safety events among all accrued patients 
from prior to their index visit through the end of the observation period although these events 
are unlikely to be related to the intervention.   
7.2 Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters 
We propose to use passive surveillance of EHR and administrative data sources to monitor 
the occurrence of safety events. Benefits to this approach are that safety events can be 
ascertained identically for all accrued patients in all randomized clinics. Documentation will 
occur in the course of delivering health care or administering health insurance so that records 
should be reasonably accurate and complete with minimal and randomly missing information. 
When care for safety events is delivered outside the HealthPartners care system, health care 
claims can document their occurrence for patients who are also health plan members. Missing 
information due to events for which patients do not seek care or care that escapes 
documentation in the EHR or claims will be assumed to occur equally across treatment 
groups. 
7.3 Clinician Feedback and Safety Events  
7.3.1 Clinician Feedback  
This intervention is being delivered by way of CDS prompts to influence clinician actions to 
incorporate evidence-based best practice standards related to CI. Prior to implementation, we 
will train all intervention clinicians and their rooming staff on the importance of helping us 
identify any clinician-identified safety events or near-misses that may be related to the EHR 
or CI-CDS. We will systematically educate them in identification of potential safety events and 
near-misses and informing us of these events via use of the Feedback Tab in the CDS or 
email. We will also ask clinicians to notify us of any clinical situations where their clinical 
judgment differs from the CI-CDS. 
Use of the Feedback Tab will automatically generate an email that is sent to study team 
members, including investigators and programmers. The study team will then discuss this 
NCT05723523

feedback and any necessary actions, and reply to the clinician to answer the question, discuss 
steps taken to address the issue, or gather additional information if needed to further trouble-
shoot. Clinicians will be asked to submit feedback any time their clinical judgment is 
inconsistent with the CDS tool. Additionally, the emails of study investigators will be listed in 
the training materials for clinicians, and clinicians will be encouraged to contact the study team 
directly with any questions or concerns if they would rather not use the Feedback Tab in the 
CI-CDS. This feedback will be provided to the DSMB at a frequency determined by the DSMB. 
7.3.2 Safety events 
Adverse Event (AE):   
As this is a minimal risk study, with the CI-CDS intervention layered on top of usual care, 
adverse events will not be collected in the context of this trial.  
Safety Events:   
 
emergency department visits 
 
hospitalizations 
 
suicide attempts 
 
death 
We will extract data from the EHR and administrative databases to monitor safety events that 
occurred from 1 year prior to each accrued patient’s index visit date through the end of the 
observation period. The following activities will be carried out at a frequency determined by 
the DSMB. The study statistician will assemble the random, unique study identifier and index 
visit date for all accrued patients. A study programmer will extract data elements needed to 
characterize the patient sample and to quantify rates of emergency department visits, 
hospitalizations, suicide attempts, and deaths. State preliminary manner of death data are 
sent to HealthPartners monthly, with a lag of about a month; these data and final adjudicated 
cause of death data are incorporated into HealthPartners databases upon receipt. The 
statistician will assemble a safety report and provide it to the co-PIs, OCHIN Lead Co-
Investigator and the DSMB.  
The DSMB report will provide information regarding patient accrual, intervention delivery and 
safety events. Patient accrual will be tracked through monthly and cumulative counts of actual 
and expected counts of index visits. Pre-randomization clinic characteristics and index visit 
date patient characteristics will be provided overall and by treatment group within site. CI-CDS 
use will be tracked monthly and cumulatively by treatment arm within site to monitor 
intervention adherence (intervention clinics) and check for contamination (control clinics). 
The following metrics will be calculated for each safety event: total number of events (allowing 
multiple per patient), proportion of accrued patients with at least one event, and the event rate 
in patient-years. Each metric for each event will be compared between the treatment groups 
(intervention vs. control) and over time (post-index vs. 1 year pre-index). Data from the year 
prior to each patient’s index visit will provide baseline data regarding the prevalence of each 
safety event. Post-index data will be compared to baseline data by treatment group to identify 
differential changes in safety event rates.  
NCT05723523

As safety events will be summarized by treatment arm using EHR and administrative data, it 
is not feasible to assess relationship to the study intervention, per se. Instead, the study team 
and DSMB will evaluate safety data at an interval determined by the DSMB to evaluate 
differences in safety event rates across treatment groups.  
7.3.3 Follow-up for Safety Events 
As safety events will be assessed retrospectively via EHR and administrative data and given 
that the study team does not interact directly with patients (but instead they are treated by 
their primary care clinicians), the study team will not be providing follow-up for safety events.  
Instead, we can presume that the primary care clinicians and other medical professionals who 
are providing the patients’ clinical care will manage these events.  
7.4 Safety Monitoring 
This study will be monitored by an NIA-appointed Data and Safety Monitoring Board (DSMB).  
Tables will summarize, within site and by blinded treatment group, (a) metrics pertaining to 
patient accrual; (b) performance of the CI-CDS system; (c) pre-randomization clinic 
characteristics and index visit patient characteristics; and (d) safety measures. Reports will 
be provided to the DSMB at a frequency the DSMB requests. The DSMB will provide an 
opinion on whether there is support for continuation of the trial, evidence that study procedures 
should be changed, or evidence that the trial should be halted for any reason, such as the 
safety of study participants, the efficacy of the treatment under study or inadequate trial 
performance. 
8   INTERVENTION DISCONTINUATION  
Once randomized, all primary care clinics are anticipated to remain enrolled for the duration 
of the study. All accrued patients will be followed for the duration of the observation period 
(i.e., end of study) unless they die or leave the care system. Patients who have opted out of 
research will be excluded from all analyses. 
9 STATISTICAL CONSIDERATIONS  
9.1 General Design Issues  
The specific aims of the R33 phase of this trial are: To evaluate the effect of the CI-CDS 
system on rates of CI detection, clinician confidence, and healthcare utilization costs in a 
pragmatic cluster-randomized trial of at least 30 primary care clinics randomized to CI-CDS 
or UC. 
H1: Patients with elevated CI risk at index visits in CI-CDS compared to UC clinics will 
have significantly higher rates of CI detection as indicated by EHR documentation of CI 
diagnosis in up to 24 months of follow up.  
H2: Clinicians at CI-CDS compared to UC clinics will have significantly more confidence 
in diagnosing and managing CI, as assessed through clinician surveys.  
H3: Among a subset of insured patients with elevated CI risk at index visits, those at CI-
CDS clinics will have significantly lower healthcare utilization costs related to emergency 
room and inpatient visits in the follow-up period compared to similar patients at UC clinics. 
NCT05723523

The primary effectiveness study is a pragmatic clinic-randomized trial that will assess the 
effectiveness of the CI-CDS system to impact outcomes that are important to patients, 
clinicians, and health system. This trial will be followed by a replication study in up to 30 
randomized clinics in a separate health care system (replication site).  The minimal eligibility 
criteria at the clinic and patient level help ensure that the results of these assessments will be 
generalizable to a range of clinical settings and patients. The intervention, the CI-CDS system, 
will be operating in live, primary care settings. At the end of the replication site intervention 
period, the study team will work with replication site leadership to transition the CI-CDS system 
from an experimental technology to the system-wide standard of care if so desired by 
leadership. The transition is readily accomplished by enabling user interface functionality in 
clinics that had been randomized to the UC group and by adding new clinic locations to the 
CI-CDS algorithms. 
 
9.2 Sample Size and Randomization 
9.2.1 Treatment Assignment Procedures 
 
Clinics will be randomized 1:1 to CI-CDS or UC using simple randomization. Randomizing by 
clinic rather than clinician or patient is the most effective way to minimize risk of intervention 
contamination. Patients tend to receive most of their care over time at a single clinic, and 
clinicians tend to practice at a single clinic, so it is reasonable to assert that the risk of 
intervention contamination across clinics is low. However, it is common for patients to see 
more than one clinician in a clinic location over time. Randomizing by clinician is not preferable 
because it would increase the number of patients with multiple visits during the intervention 
period who would be exposed to both treatment groups over the course of these visits. 
Repeated use of the CI-CDS system may alter practice patterns so that clinicians may 
diagnose and manage care for patients with CI in the absence of prompts provided by the 
CDS. Randomizing by patients is not preferable because it would result in care for UC patients 
that was contaminated by changes in clinician practice patterns. 
Each patient will be assigned to the clinic in which his or her index visit takes place, and as 
such will be assigned to the treatment group to which their clinic was randomly assigned. 
Post-index visits may take place in the same or different clinics or treatment groups relative 
to the index visit and may or may not be eligible for the CI-CDS to offer treatment 
recommendations. In keeping with an intent-to-treat principle, all index and post-index visits 
and outcome measures for each patient will be attributed to the treatment group assignment 
of the clinic where the index visit took place. 
The nature of the intervention prevents study personnel, clinic leaders or primary care 
clinicians from being blinded to the treatment group assignment of each clinic. 
9.2.2 Sample Size Justification  
The primary hypothesis test will compare patients in CI-CDS clinics to patients in UC clinics 
on the likelihood that CI is diagnosed during the observation period. We conducted a power 
analysis to estimate the minimum detectable rate of new CI diagnoses given ranges of 
assumptions about analytic sample sizes, proportions of patients currently meeting outcome 
criteria, and clinic intraclass correlation (ICCclin). We used data accrued into the CI-CDS  
NCT05723523

system during the silent pilot phase from the R61 to identify patients who met intervention 
eligibility criteria at clinic visits (sections 4.1, 4.2). Based on the pilot CI-CDS data, we estimate 
that, on average, about n=100 patients in each of at least 30 clinics will likely have an index 
visit in over the course of a 12-month accrual period.  
An EHR-based cohort that consisted of patients who were age >=65 and had a primary care 
visit between September 2021 and August 2022 was assembled to provide estimates of CI 
diagnosis rates in primary care clinics that were candidates for randomization. The proportion 
of patients who had at least one CI diagnosis code at an encounter or listed on their active 
problem list during the 12-month period was 2.3% among all patients in the cohort, and 15.1% 
among those who had a MiniCog<3 during the period. There was substantial variation in CI 
recognition among all patients, ICCclin=0.03, and among those with MiniCog<3, ICCclin=0.04.  
Based on these data-informed estimates, we 
used the following assumptions regarding the 
primary effectiveness study for the power 
analysis: 30, 34 or 38 clinics randomly assigned 
1:1 to CI-CDS or UC, n=100 patients per clinic, 
ICCclin=(0.03, 0.04, 0.05) and diagnosis rates = 
(8%, 12%, 16%) in UC clinics. The correlated 
sample size estimate, N, was divided by the 
design effect (deff; 1+(n-1)*ICC) to estimate an 
equivalent independent patient sample size (i.e., 
Neff=N/deff). The power analysis (power=0.80, 
α2=0.05) estimated the minimum detectable CI-
CDS CI diagnosis rate from a single binary 
predictor in a logistic regression model using the 
downwardly adjusted independent sample size 
estimate.[84]  
Given the median assumptions for number of 
randomized clinics, ICCclin and CI diagnosis rate, 
the study is powered to detect a diagnosis rate of 
19.8% in CI-CDS clinics relative to 12% in UC 
clinics (Table 1). The absolute increase in 
diagnosis rates ranges from 5.6% (38 clinics, 
ICCclin=0.03, 
8%) 
to 
10.2% 
(30 
clinics, 
ICCclin=0.05, 16%) across the range of these 
assumptions.  
A comparable power analysis was conducted for 
the replication site analysis of CI diagnosis rates 
among patients in CI-CDS relative to UC clinics. 
Assumptions about the number of randomized 
clinics (22, 26, 30), patients per clinic (n=60) and CI diagnosis rate (4%, 8%, 12%) were 
changed to acknowledge the likely smaller sample size and shorter observation period relative 
to the primary effectiveness site. Given the median assumptions for number of randomized 
clinics, ICCclin and CI diagnosis rate, the study is powered to detect a diagnosis rate of 16.5% 
in CI-CDS clinics relative to 8% in UC clinics (Table 2). The absolute increase in diagnosis 
rates ranges from 5.5% (30 clinics, ICCclin=0.03, 4%) to 9.7% (22 clinics, ICCclin=0.05, 12%) 
across the range of these assumptions. 
Table 1. Minimum detectable rates of CI 
diagnosis in CI-CDS clinics assuming 
n=100 accrued patients per clinic, 30-38 
clinics, ICCclin = 0.03-0.05 and UC 
diagnosis rates = 8-16%. 
 
 
UC diagnoses (%) 
clinics 
ICCclin 
8% 
12% 
16% 
 
0.03 
14.4 
19.4 
24.2 
30 
0.04 
15.3 
20.4 
25.2 
 
0.05 
16.1 
21.3 
26.2 
 
0.03 
14.0 
18.9 
23.6 
34 
0.04 
14.8 
19.8 
24.6 
 
0.05 
15.5 
20.6 
25.5 
 
0.03 
13.6 
18.5 
23.2 
38 
0.04 
14.4 
19.4 
24.1 
 
0.05 
15.1 
20.1 
24.9 
Table 2. Minimum detectable rates of CI 
diagnosis in replication site CI-CDS clinics 
assuming n=60 accrued patients per clinic, 
22-30 clinics, ICCclin = 0.03-0.05 and UC 
diagnosis rates = 4-12%. 
 
 
UC diagnoses (%) 
clinics 
ICCclin 
4% 
8% 
12% 
 
0.03 
10.7 
16.4 
21.6 
22 
0.04 
11.6 
17.4 
22.7 
 
0.05 
12.4 
18.3 
23.7 
 
0.03 
10.0 
15.6 
20.7 
26 
0.04 
10.8 
16.5 
21.7 
 
0.05 
11.5 
17.4 
22.6 
 
0.03 
9.5 
15.0 
20.1 
30 
0.04 
10.2 
15.8 
21.0 
 
0.05 
10.9 
16.6 
21.8 
NCT05723523

9.3 Interim analyses and Stopping Rules 
There are no plans to conduct an interim analysis. The study team will collaborate with the 
DSMB to define thresholds at which differential rates of safety events by treatment group 
warrant consideration of suspending patient accrual or modifying intervention delivery.  
9.4 Outcomes  
9.4.1 Primary outcome   
The primary outcome will be CI detection (H1). CI detection will be a binary outcome 
calculated from EHR data and defined as occurring when an ICD-10 diagnostic code for CI is 
documented at outpatient or inpatient encounters, or added to the problem list, between the 
index visit date and the end of the observation period, inclusive.  
9.4.2 Secondary outcomes   
A secondary outcome will be clinician-reported confidence in management of CI care (H2). 
The pre-and post-implementation clinician surveys in both study arms will include a series of 
Likert-scale survey items based on previous literature to assess confidence to correctly 
assess and diagnose CI.[14, 27] Items will ask about confidence in diagnosis and 
management of CI care generally and will include items about specific elements such as 
conducting appropriate testing and patient and care partner education. Items will be analyzed 
individually, and composite measures of confidence will be considered. 
Another secondary outcome will be health care utilization among accrued patients with 
HealthPartners insurance (H3). Indicators of any use of emergency room (ER), inpatient or 
both facility types between the index visit and of the observation period, inclusive, will be 
calculated from data extracted from insurance claims databases. Additionally, the combined 
costs of ER and inpatient care delivered will be extracted as paid amounts for facility and 
professional services on ER and inpatient days.  
Patient and clinician characteristics that could moderate the effectiveness of the CI-CDS 
system on these primary and secondary outcomes will also be extracted from the EHR and 
clinician surveys. EHR-based data include patient encounters (number and type), 
demographics (e.g., age, gender, race, ethnicity), social determinants of health (e.g., insurer, 
area deprivation index score for home address, median education for Census tract of home 
address, country of origin, need for interpreter) and comorbidities present at index (e.g., 
diabetes, cardiovascular disease, mental illnesses, substance use). Potential clinician 
moderators include clinician self-reported years in practice, sex, provider type and specialty.  
The effectiveness of the CI-CDS system on several exploratory outcomes will also be 
assessed. These exploratory outcomes include clinical actions documented in patient charts 
or the EHR (e.g., referrals to community resources; documentation of care plans, release of 
information, caregiver identification or recommendations for managing stressful symptoms or 
safety); clinician attitudes (e.g., perceived control, attitudes about CI, perceived norms) or 
perceptions of CI-CDS  (e.g., usefulness, intentions to treat patients with CI) reported in the 
post-implementation clinician survey; and patient or caregiver experiences discussed in the 
patient/caregiver dyad interviews about assessment for CI (e.g., addressed or unaddressed 
concerns about memory; benefit or harm of screening; feelings of preparedness, support, 
distress among newly diagnosed patients).  
NCT05723523

We will use the CI-CDS metadata (e.g., display, print, click rates for CI-CDS features or patient 
educational or resource materials) and the EHR (e.g., Quick Orders for cognitive assessment, 
labs, brain imaging, referrals, medications for cognition benefits or symptom management) to 
track clinician use of the CI-CDS system and compliance with suggested actions.  
9.5 Data Analyses 
9.5.1 Primary analysis  
The H1 sample will consist of primary effectiveness site patients accrued from the CI-CDS 
and UC clinics who are at elevated risk for CI but without a CI diagnosis at index. The primary 
outcome is defined as at least one CI diagnostic code or CI diagnosis entry on the problem 
list at any time between the index visit and the end of the observation period (at least 12 and 
up to 24 months post-index), inclusive. H1 will be tested using a generalized linear mixed 
model (LMM) to account for clustering within randomized clinics and normalize the binary 
outcome via a distribution-appropriate link function (e.g., logit, log). Clinic-randomized 
treatment group and any patient covariates (e.g., age at index visit, sex, MC-PLUS risk score 
at index visit) will be treated as fixed effects. 
Descriptive statistics will be calculated on all pre-intervention patient variables across all 
accrued patients, by treatment group and by clinic to ensure that key patient characteristics 
related to the risk of CI or its diagnosis (i.e., age, sex, comorbidities) are balanced across 
treatment arms. Imbalanced patient characteristics will be considered for inclusion as 
covariates in the primary analyses.  
For H1, we will fit a generalized LMM in which the binary post-index CI diagnosis outcome will 
be predicted via a link function from fixed effects for the clinic-randomized predictor, γ10, and 
patient covariates, γ0*, and a random clinic intercept, vj, such as:  
logit(diagnosisji) = γ00 + γ10CI-CDSj + γ0*(pt covars]ji + [vj]. 
Parameter γ10 is expected to be positive and statistically significant, indicating that accrued 
patients in clinics randomly assigned to CI-CDS are more likely to receive a CI diagnosis in 
the observation period than patients in UC clinics, accounting for fixed effects and random 
clinic variation.  
In a second model, we will add a fixed coefficient that represents time elapsed between the 
index visit and the first EHR-documented CI diagnosis, or zero for those lacking a CI 
diagnosis, and the interaction between treatment group and time. The time covariate will be 
coded as (days since index)/365 so that it quantifies the predicted annual change in CI 
diagnosis rates. The time parameter is expected to be near zero and not statistically 
significant. Should the time elapsed between index and CI diagnosis differ by treatment group, 
log (observation time) may be included as an offset in the primary analysis.  
This same analytic approach will be used to estimate the CI-CDS system treatment effect at 
the replication site. There are no plans for a pooled analysis across the primary effectiveness 
and replication sites, nor to implement alpha-sparing techniques. The determination of CI-
CDS effectiveness will rest on the results of the primary analysis at the effectiveness site.  
9.5.2 Secondary analyses  
Treatment effectiveness may vary as a function of contextual factors or patient characteristics. 
NCT05723523

Secondary H1 analyses will add fixed parameters to the H1 model that assess contextual 
factors (e.g., area deprivation index score for home address, median education for Census 
tract of home address) or patient characteristics (e.g., socioeconomic status, gender, pre-
intervention CI risk) as main effects and in interaction with the CI-CDS indicator to assess 
whether the CI-CDS intervention is differentially effective (i.e., treatment heterogeneity) 
across patient subgroups, or relatedly, whether the CI-CDS intervention can reduce pre-
intervention disparities in CI care. 
9.5.3. Secondary outcomes  
H2 predicts that clinicians practicing in CI-CDS relative to UC clinics will report more post-
implementation confidence in diagnosing and managing CI. We will also use a LMM approach 
for the H2 analyses. The distributional properties of the confidence and management 
composites calculated from post-implementation clinician survey data will inform how to 
specify the distribution and link functions for these variables (e.g., normal-identity, Poisson-
log, binomial-logit, binomial-log, multinomial-cumulative logit). The fixed CI-CDS parameter 
and random clinic intercept from the H1 model will be retained, and a fixed parameter for the 
outcome value reported in the pre-implementation clinician survey will be added. Exploratory 
analyses will assess whether the relationship between CI-CDS and either of these outcomes 
is more pronounced among clinician subgroups (e.g., clinician type).  
H3 predicts that patients accrued from CI-CDS clinics will have lower post-index health care 
utilization. Anticipating a zero-mass of patients with no emergency department or inpatient 
costs, we will employ a 2-part model to test H3.[86] In the first part, the probability that patients 
will experience one or more emergency department visits or inpatient stays will be assessed 
using logistic regression in the same form as was used for H1. The second part will employ a 
generalized linear model (GLM)[87, 88] allowing clustering by clinic and controlling for 
demographics and baseline risk. Such analyses often specify a gamma distribution for health 
care expenditures with a log link function of the explanatory variables. We will choose the 
distribution family based upon the data using a modified Park test and choose the link function 
using a Box-Cox test.[89] If we observe a statistically significant reduction in emergency 
department and inpatient costs in the CI-CDS group, we will use similar methods to assess 
whether total costs of care, including all outpatient care and pharmacy, are also lower. 
Analyses of secondary and exploratory patient outcomes will also follow the same analytic 
approach as the H1 model with appropriate modifications to the link function and error 
distribution. 
 
10. DATA COLLECTION AND QUALITY ASSURANCE 
10.1 Data Collection Forms 
We expect person-based missingness to be extremely rare. Patients are unlikely to be aware 
that their data are being used for this research. They will not be consented and are unlikely to 
request that their data be excluded from analyses. Only patients who have requested that 
their data not be used for research and appear on site-maintained opt out lists will be 
excluded. 
The primary data source for hypotheses 1 and 3 and the safety analyses will be EHR- and 
claims-based data repositories maintained by HealthPartners. The stored EHR data elements 
NCT05723523

required for calculating the primary and secondary outcomes are extracted from live 
production tables. Medical care costs for H3 will come from administrative claims systems that 
are relied upon for reimbursement of medical care services. The absence of documentation 
of a care process, vital sign, or medication should not be interpreted as a missing value but 
rather as indicative of a care process or test not having been performed or medication not 
prescribed within the health system. Likewise, absence of utilization indicated by billing claims 
almost always indicates that the utilization (such as a hospitalization) did not occur. Truly 
missing field-based observations (e.g., CI diagnosis assigned, action not observed) will be 
extremely rare, undetectable, and assumed to be missing at random. The primary data source 
for H2 is a survey developed during the R61 phase. The survey will be administered and data 
stored securely using RedCap. 
10.1.1 Missing Data  
Data elements required for calculating the primary outcomes are extracted from EHR 
production tables. The absence of documentation of a care process, vital sign, or medication 
should not be interpreted as a missing value but rather as indicative of a care process or test 
not having been performed or medication not prescribed within the health system. Truly 
missing observations (e.g. value not available) will be extremely rare, undetectable and 
assumed to be missing at random. The estimation techniques used in random coefficient 
models readily accommodate structural variation across observations in the amount of data 
present (e.g., patients per clinic) and lead to unbiased parameter estimates and accurate 
standard errors when data are missing at random.  
10.2 Data Management  
The CI-CDS itself will house the algorithms, communicate with and display within the EHR, 
and store data required to assess study objectives in a secure analytic database at 
HealthPartners Institute. These data, supplemented by EPIC Clarity data, will be used to 
assess CI-CDS use rates, diagnosis of CI, and hospitalizations and emergency department 
visits. Data collected from the primary care clinicians via surveys will be similarly housed on 
secure servers at HealthPartners Institute.  
10.3 Quality Assurance  
As this study is conducted as a pragmatic clinical trial, with CI-CDS being suggested to 
primary care clinicians in the course of usual care, the training and quality control metrics 
typically found in traditional randomized trials do not apply. Primary care clinicians will receive 
training on use of the CI-CDS and print rates will be monitored and communicated to 
clinicians, as noted above. 
10.4 Protocol Deviations 
Any departure from procedures and requirements outlined in the protocol will be classified as 
either a major or minor protocol deviation. The difference between a major and minor protocol 
deviation has to do with the seriousness of the event and the corrective action required. A 
minor protocol deviation is considered an action (or inaction) that by itself is not likely to affect 
the scientific soundness of the investigation or seriously affect the safety, rights, or welfare of 
a study participant. Major protocol deviations are departures that may compromise the 
participant safety, participant rights, inclusion/exclusion criteria or the integrity of study data 
and could be cause for corrective actions if not rectified or prevented from re-occurrence. 
NCT05723523

Additionally, the Co-PIs are responsible for reviewing the IRB of record’s definition of a 
protocol deviation or violation and understanding which events need to be reported.  
10.5 Monitoring 
All protocol deviations will be monitored for (1) significance, (2) frequency, and (3) impact on 
the study objectives, to ensure that site performance does not compromise the integrity of the 
trial. The Co-PIs will be responsible for developing corrective action plans for both major and 
minor deviations as appropriate. Those corrective action plans may be reviewed/approved by 
the Co-PIs with overall approval by the IRB of record. All protocol deviations will be recorded 
in password protected project folders at HealthPartners Institute. 
10.6 Regulatory Files 
The regulatory files will contain all required regulatory documents, study-specific documents, 
and important communications.  
 
10.7 Reporting to Sponsor 
 
The lead investigators agree to submit accurate, complete, legible, and timely reports to the 
Sponsor, as required. These include, but are not limited to, reports of any changes that 
significantly affect the conduct or outcome of the trial or increase risk to study participants. 
Safety reporting will occur as previously described.  
 
10.8 Audits 
 
The Lead Investigator and authorized staff from the NIA CTN (the study sponsor); NIA’s 
contracted agents, monitors or auditors; and other agencies such as the HHS, the OHRP and 
the IRB of record may inspect research records for verification of data, compliance with federal 
guidelines on human participant research, and to assess participant safety. 
 
10.9 Study Documentation 
Each participating site will maintain appropriate study documentation (including medical and 
research records) for this trial, in compliance with ICH E6 R2 and regulatory and institutional 
requirements for the protection of confidentiality of participants.  
 
11. PARTICIPANT RIGHTS AND CONFIDENTIALITY  
11.1 Institutional Review Board (IRB) Review 
Prior to initiating the study, participating site investigators will obtain written IRB approval from 
the IRB to conduct the study at their respective sites, which will include approval of the study 
protocol. If changes to the study protocol become necessary, protocol amendments will be 
submitted in writing by the investigators for IRB approval prior to implementation. In addition, 
IRBs will approve recruitment materials, and any materials given to the participant, and any 
NCT05723523

changes made to these documents throughout study implementation. Unanticipated problems 
involving risk to study participants will be promptly reported to and reviewed by the IRB of 
record, according to its usual procedures. 
 
HealthPartners will serve as the primary effectiveness study site, and as such, the 
HealthPartners IRB will serve as the central IRB for this study. The replication site will cede 
to the HealthPartners IRB and enter into reliance/authorization agreements. The 
HealthPartners IRB will provide study oversight in accordance with 45 CFR 46. HealthPartners 
IRB will follow written procedures for reporting its findings and actions to appropriate officials 
at each participating institution. Of note, it is possible a partner site may meet Exception 
Criteria to the NIH sIRB (single IRB) Policy and may not utilize the IRB of Record. 
 
11.2 Informed Consent Forms 
As this study aims to help clinicians achieve accepted standards of care for cognitive 
impairment, we have received a waiver of consent for clinicians and patients in this study. 
Patients and clinicians will be consented to participate in any surveys or interviews.  The 
consent process will be IRB approved. We will request a waiver of documentation of consent 
as described in Section 15.2. 
11.3 Participant Confidentiality  
Participant confidentiality and privacy is strictly held in trust by the participating investigators, 
their staff, the safety and oversight monitor(s), and the sponsor and funding agency, and will 
be maintained in accordance with all applicable federal regulations and/or state laws and 
regulations. This confidentiality is extended to the data being collected as part of this study. 
Data that could be used to identify a specific study participant will be held in strict confidence 
within the research team. 
All research activities will be conducted in as private a setting as possible. 
Participant records will be held confidential using study codes for identifying participants, 
secure storage of any documents that have participant identifiers, and secure computing 
procedures for entering and transferring electronic data. At the end of the study, all records 
will continue to be kept in a secure location for as long a period as required by state or federal 
law (whichever is longer).  
11.4 Study Discontinuation  
The study may be discontinued at any time by the IRB, the NIA, the OHRP, the FDA, or other 
government agencies as part of their duties to ensure that research participants are protected.  
12. ETHICAL CONSIDERATIONS 
12.1 Statement of Compliance 
This study will be conducted in accordance with the current version of the protocol, in full 
conformity with the ethical principles outlined in the Declaration of Helsinki, the Protection of 
Human Subjects described in the International Council for Harmonisation (ICH) GCP 
Guidelines, applicable United States Code of Federal Regulations (CFR), the NIA Terms and 
Conditions of Award, and all other applicable state, local and federal regulatory requirements. 
NCT05723523

The Co-PIs will assure that no deviation from, or changes to, the protocol will take place 
without prior agreement from the Sponsor and documented approval from the IRB, except 
where necessary to eliminate an immediate hazard(s) to the trial participants. 
12.2 Investigator Assurances 
Each site must file (or have previously filed) a Federalwide Assurance (FWA) with the HHS 
Office for Human Research Protection (OHRP) setting forth the commitment of the 
organization to establish appropriate policies and procedures for the protection of human 
research subjects in alignment with 45 CFR 46, Subpart A, with documentation sent to NIA or 
its designee. Research covered by these regulations cannot proceed in any manner prior to 
NIA receipt of certification that the research has been reviewed and approved by the IRB 
provided for in the assurance (45 CFR 46.103).  
 
All investigators will comply with the requirements of 42 CFR Part 50, Subpart F to ensure 
that the design, conduct, and reporting of the research will not be biased by any conflicting 
financial interest. Everyone with decision-making responsibilities regarding the protocol will 
complete annual institutional financial disclosure requirements. 
 
12.3 Inclusion of Women and Minorities 
All patients aged 65 and older, inclusive, who are determined by the CI-CDS algorithms to be 
at high risk for CI and who meet all other protocol-defined eligibility criteria will be included in 
the study, regardless of sex or racial/ethnic group. Patients who have opted out of research 
will be excluded from analyses. All clinicians in randomized clinics will be included in the study, 
regardless of sex or racial/ethnic group. 
 
12.4 Prisoner Certification 
As per 45 CFR 46 Subpart C, there are additional protections pertaining to prisoners as study 
participants. A prisoner is defined as any individual involuntarily confined or detained in a 
penal institution. The term is intended to encompass individuals sentenced to such an 
institution under a criminal or civil statute, individuals detained in other facilities by virtue of 
statutes or commitment procedures which provide alternatives to criminal prosecution or 
incarceration in a penal institution, and individuals detained pending arraignment, trial, or 
sentencing. 
 
Due to the nature of the CI-CDS, a prisoner certificate is not applicable to this data collection. 
Patient participants who meet the OHRP definition of “prisoner” will be ineligible for 
participation in any surveys. 
 
13. COMMITTEES 
Committee 
Role 
Steering 
Committee 
• 
Set strategic direction of project 
• 
Monitor project process and plans 
• 
Advocate for project across wider organization 
• 
Prioritize and reprioritize project deliverables 
NCT05723523

AI Team 
• 
Development of MC-PLUS prediction model 
• 
Testing of prediction model 
• 
Validation of prediction model 
Technical Team 
• 
CDS algorithm development 
• 
Web/EPIC development 
Dementia Expert 
Panel 
• 
Contribute expertise in cognitive disorders and MiniCog testing 
• 
Contribute expertise in care processes improvements 
• 
Contribute expertise in machine learning 
Data Collection 
and Analysis 
• 
Analyze date from pilot and final intervention 
• 
Dissemination of results through publications and presentations 
 
14. PUBLICATION OF RESEARCH FINDINGS 
Publication of the results of this trial will be governed by the policies and procedures 
developed by the Co-PIs.  
15. SUPPLEMENTS / APPENDICES   
15.1 Procedures schedule 
In this pragmatic clinical trial, patients will receive care as part of their usual primary care. 
Clinic visits will be scheduled at a frequency determined by the patient with his or her primary 
care clinician.  As such, there is not a specified procedures schedule. However, the timeline 
for the R33 Phase of this trial is below: 
Phase 
R61 
R33 
  
Year 1 
Year 2 
Year 3 
Year 4 
Year 5 
Quarter 
1 
2 
3 
4 
1 
2 
3 
4 1 
2 
3 
4 
1 
2 
3 
4 1 
2 
3 
4 
Phase 1: Planning, Pilot, 
Validation 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Machine Learning Analysis & 
Refinement 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
CDS & EHR Tool 
Development 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Pilot and Revise CI-CDS 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Phase 2: Implementation 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
CDS revisions; provider 
baseline survey; clinic 
randomization & training 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Patient Accrual for Primary 
Effectiveness Site 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Observation Period for 
Primary Effectiveness Site 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Patient Accrual for 
Replication Site 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Observation Period for 
Replication Site 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Intervention fidelity 
monitoring; maintenance 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Phase 2: Analysis, 
Reporting, Dissemination 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
NCT05723523

Analysis preparation 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Main outcome, disparities, & 
secondary analysis 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Manuscript and conference 
dissemination 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
15.2 Informed Consent Document 
As noted above, we have requested and been granted a waiver of consent for use of the CI-
CDS in clinical care in both the R61 and R33 phases of this study. Reasons for requesting a 
waiver of consent include: 
a) All treatment options identified by the CDS intervention are U.S. Food and Drug 
Administration-approved and evidence-based, as specified in the regional and 
national clinical practice guidelines. No care is advocated that is not evidence-
based. The care conforms to current standards of care and does not represent 
any additional risk to patients beyond the routine risk that all patients assume 
whenever they have contact with the medical care system. The CDS system 
has been used in several previous randomized clinical trials with DSMB 
monitoring that revealed no safety concerns. 
b) At clinic training sessions and on printed/electronically displayed decision 
support tools, we emphasize that it is inappropriate for a clinician to follow 
suggested treatment options without further checking the clinical status of a 
given patient. A disclaimer on the CDS says that treatment options are based 
only on electronically available data and are not intended to be a substitute for 
clinical judgment. 
c) It would be impractical to consent patients (due to large numbers of patients) 
and impossible to answer the primary research questions (due to selection 
effects related to consent) if written informed consent of all patients were 
required. 
d) The CDS platform upon which the CI-CDS intervention is based is supported 
by the care system’s leadership, implemented with their collaboration and 
support, and embedded in the care process at the clinic and department level. 
Therefore, the CDS platform itself is part of routine care, and is accessible to 
all clinicians, with use of the CDS tool indicating implied consent. 
 
A waiver of documentation of consent is being requested for surveys of providers and dyad 
interviews.     
A general script for conversations with patients and care partners for interviews is as follows: 
First, let me to go over some details about the study with you.  If you have questions at any 
point as we go through the information, please feel free to interrupt me so that I can answer 
your questions. Once we are done, I will ask if you are still willing to participate in this interview. 
In this study, we want to learn more about the assessments and diagnosis you received for 
cognitive impairment and to see how prepared you feel for the disease management.  We 
expect up to 20 dyads (pairs of patients and care partners) to participate. 
NCT05723523

As a subject, you will participate in a telephone interview that will take about 20 minutes. We 
will schedule the interview for a time that works for you.  
There are no anticipated physical risks to you related to this study. Some questions may make 
you uncomfortable, but you can skip any questions that you do not want to answer. 
There is no cost for you to participate. If you complete the telephone interview, we will send 
you a $10 Amazon.com Gift Card.  
Your study records will be kept confidential. 
You do not have to be in this study. If you start the study, you may stop at any time. Your 
decision to participate will not affect the care provided by your clinic, your insurance coverage, 
or your relationship with your health care providers. 
15.2.2 Survey Invitation and Consent for Primary Care Clinicians 
An email invitation will be sent to clinicians to participate in the survey like this: 
Our primary care clinics are collaborating with HealthPartners Institute on a short survey to 
better understand the experience of clinicians in detecting and diagnosing cognitive 
impairment. 
You will soon receive an email from CIwizard@HealthPartners.com with a link to an online 
survey. Please take 10 minutes of your time to complete this confidential survey. If you choose 
to participate, you will receive a $100 Amazon.com Gift Card* after you complete the survey. 
Your privacy is important to HealthPartners, and your name will not be associated with your 
responses. The results will go to HealthPartners Institute’s central survey center and will only 
be reported to HealthPartners leadership in aggregate to ensure confidentiality. However, the 
center will be tracking who responds so they can provide follow-up reminder emails since it is 
important that we hear from as many providers as possible. 
Thank you in advance for your participation; your feedback is greatly appreciated. 
15.3 Summary Results of Primary Care Clinician Pilot Surveys 
A survey to identify clinician-reported barriers to diagnosing cognitive impairment and 
managing care was designed following best practices in survey methods. The survey was 
designed to be completed in 10 minutes or less and included 30 items aligned with the 
Capability, Opportunity, Motivation – Behavior (COM-B) and Technology Acceptance 
Model (TAM) theoretical frameworks. These surveys were successfully fielded at all 3 of the 
clinics recruited to participate in the pilot. Prior to initiating the pilot intervention (clinician 
training and providing access to the clinical decision support tool), 25 eligible primary care 
clinicians were identified. All were in family practice or internal medicine departments and 68% 
had an MD or DO (as opposed to nurse practitioner or physician assistant degrees). With 
permission from clinic leaders, all clinicians were emailed a survey invitation from the study 
principal investigators offering an optional opportunity to participate. In total, 14 clinicians 
completed the survey (response rate of 56%) and were given a $100 gift card as a thank you 
for their time. Responding clinicians were primarily female (54%), white (92%), 40 years of 
age or younger (62%), had practiced for 10 years or less (62%), and saw patients in the clinic 
at least 4 days a week (86%).  
NCT05723523

Barriers to assessing and diagnosing cognitive impairment were identified across clinician 
motivation to diagnose and their social and environmental opportunity to diagnose. There 
were no substantial ceiling effects in these items, with all having opportunity for improvement 
with intervention. All clinicians reported time in visits and usability of EHR as at least 
somewhat of a barrier to diagnosing. The majority of clinicians also identified lacking 
confidence in diagnosing, perceiving patients as resistant to assessments, lacking access to 
care partners in visits, and lacking standard workflows for assessment and reimbursement. 
The least commonly reported barriers were lack of financial reimbursement and concerns 
about the reliability of cognitive screenings conducted.  
Table 2. Clinician-report barriers to assessing and diagnosing cognitive impairment, N=14 
clinicians. 
 
% 
Motivation to Diagnose 
 
Beliefs about Capability to Diagnose/Confidence in… 
 
Diagnosing overall, without referral to specialist 
57% a little or not 
at all confident 
Understanding brief cognitive screening results 
29% a little or not 
at all confident 
Conducting comprehensive cognitive assessments 
29% a little or not 
at all confident 
Knowing labs or imagining needed 
29% a little or not 
at all confident 
Distinguishing types of dementia 
71% a little or not 
at all confident 
Having conversations with patients about new diagnoses 
50% a little or not 
at all confident 
Beliefs about Consequences of Diagnosing 
 
Patients will not benefit from diagnosis because no effective 
treatments 
21% somewhat or 
strongly agree 
Opportunity to Diagnose 
 
Social Influences on Diagnosing 
 
Patients are resistant to cognitive assessments 
57% somewhat or 
strongly agree 
Brief cognitive screenings conducted in my clinic are reliable 
21% somewhat or 
strongly disagree 
Lack of care partner availability at visits 
71% moderate or 
large barrier 
Environmental Influences on Diagnosing 
 
Standard workflows for abnormal screenings exist in my clinic  
57% somewhat or 
strongly disagree 
Lack of reimbursement for assessment and diagnosis 
14% moderate or 
large barrier 
Lack of time in visits for assessment and diagnosis 
100% moderate or 
large barrier 
Usability of EHR for Diagnosing 
 
EHR alerts make it easier to recognize cognitive impairment risk 
100% somewhat or 
strongly disagree 
EHR tools make it easier to assess and diagnose cognitive 
impairment 
64% somewhat or 
strongly disagree 
 
NCT05723523

Similarly, barriers to managing care for dementia were also identified across clinician 
motivation and social and environmental opportunity. Again, no ceiling effects existed with all 
items allowing opportunity to improve. All or most clinicians reported a lack of confidence in 
managing various parts of dementia care, lack of access to care partners, lack of time in visits 
and limited usability of EHR to manage care for those with dementia. Only 14% (n=2 clinicians) 
reported lack of reimbursement as a moderate or large barrier to providing dementia care 
management.  
Table 3. Clinician-report barriers to managing care for patients with dementia, N=14 primary 
care clinicians. 
 
% 
Motivation to Manage Care for Dementia 
 
Beliefs about Capability to Manage Care/Confidence in… 
 
Managing care overall, without referral to specialist 
71% a little or not 
at all confident 
Starting pharmacological treatment for dementia 
71% a little or not 
at all confident 
Addressing medication safety issues 
43% a little or not 
at all confident 
Starting treatment for related depression, insomnia or agitation 
29% a little or not 
at all confident 
Accessing patient education resources 
71% a little or not 
at all confident 
Accessing care partner education resources 
71% a little or not 
at all confident 
Addressing driving safety 
64% a little or not 
at all confident 
Addressing home safety 
57% a little or not 
at all confident 
Completing advanced directive, POLST or release of information 
50% a little or not 
at all confident 
Supporting estate planning, power of attorney, legal documentation 64% a little or not 
at all confident 
Opportunity to Manage Care for Dementia 
 
Environmental Influences on Managing Care 
 
Lack of care partner availability at visits 
64% moderate or 
large barrier 
Lack of reimbursement for managing care 
14% moderate or 
large barrier 
Lack of time in visits for managing care 
100% moderate or 
large barrier 
Usability of EHR for Managing Care 
 
EHR Tools make it easier to provide care and support 
86% somewhat or 
strongly disagree 
 
Open-ended responses from clinicians reiterated the findings above: “The biggest barrier is 
not having time. The patient oftentimes does not want to do the assessment or sees it as an 
insult. The time to discuss and reach shared understanding is not available and on top of that 
is the time to perform the screening.” Some offered suggestions in addition to clinical decision 
support tools to address barriers such as lack of confidence and time. One example was 
developing a “comprehensive care team approach with the ‘right person doing the right job’”. 
NCT05723523

Clinicians were also asked to answer a brief set of questions about the quality of the pilot 
survey. All clinicians thought the items and response options were understandable, the survey 
was about the right length, and was comprehensive with few additional topics identified.  
15.4 Clinical Decision Support Content for Diagnosis and Management of CI 
Example Alert Section 
 
 
Screening and Evaluation Module 
 
 
Quick Actions Section 
 
 
 
 
 
NCT05723523

Example Patient Education Materials 
 
 
Example Patient Education Materials 
 
 
Assessments Module showing the MiniCog 
 
 
NCT05723523

Assessment Module showing the Available Comprehensive Assessments 
 
 
Additional Evaluation Module 
 
 
Diagnostic Criteria Module 
 
 
 
 
 
 
 
 
 
NCT05723523

Care and Support Module 
 
 
 
 
NCT05723523

15.5 REPLICATION SITE PROCEDURES  
The replication site, OCHIN, will follow all the procedures outlined in this protocol except 
where indicated in this appendix.  
 
OCHIN STUDY TEAM ROSTER 
 
NCT05723523

Investigator 
Role 
Contact 
Information 
Main 
responsibility/Key 
Roles 
Dates on 
Project 
Constance 
Owens-Jasey, 
PhD 
 Lead 
Co-
Investigator 
OCHIN, Inc. 
owensc@ochin.org 
• 
Assure 
adherence to 
protocol at 
replication site 
• 
Oversee 
replication site 
activities 
9/1/2022-
Current 
Rachel Gold, 
PhD, MPH 
Co-I 
OCHIN, Inc. 
rachel.gold@kpchr.
org 
• 
Assist in CDS 
implementation at 
replication stie 
9/1/2022-
Current 
Maura 
Pisciotta, MS 
Project 
Manager 
OCHIN, Inc. 
pisciottam@ochin.o
rg 
• 
Support project 
activities at 
replication site 
• 
Recruit service 
areas for study 
participation 
9/1/2022-
2/28/2024 
Joanna 
Georgescu, 
PhD 
Primary 
Analyst 
OCHIN, Inc. 
georgescuj@ochin.
org 
• 
Data collection 
planning at 
replication site 
9/1/2022-
10/31/2024 
Mary 
Middendorf 
Epic 
Application 
Developer 
OCHIN, Inc. 
middendorfm@ochi
n.org 
• 
Lead EPIC 
development and 
implementation of 
CI-CDS tool at 
replication site 
9/1/2022-
Current 
Andrew 
Weresch 
Interoperabili
ty Developer 
OCHIN, Inc. 
werescha@ochin.o
rg 
• 
Lead FHIR 
integration of CI-
CDS tool at 
replication site 
9/1/2022-
Current 
Shelby 
Watkins, MPH, 
CHP 
Project 
Coordinator 
OCHIN, Inc.  
watkinss@ochin.or
g 
• 
Support project 
activities at 
replication site 
• 
Recruit service 
areas for study 
participation 
10/1/2023-
12/31/2023 
Jenny 
Hauschildt, 
MPH 
Research 
Associate 
II/Project 
Manager 
OCHIN, Inc. 
hauschildtj@ochin.
org 
• 
Support project 
activities at 
replication site 
• 
Recruit service 
areas for study 
participation 
2/1/2024-
Current 
Dan Budney 
Epic 
Applications 
Developer 
OCHIN, Inc. 
budnyd@ochin.org 
• 
EPIC 
development and 
implementation of 
CI-CDS tool at 
replication site 
2/1/2024-
Current 
NCT05723523

 
 
 
4. SELECTION AND ENROLLMENT OF PARTICIPANTS 
 
Primary care clinics  
 
The OCHIN Lead Co-Investigator will collaborate with the HealthPartners Co-PIs to identify 
eligible service areas (groups of related clinics) that will be randomized 1:1 to the CI-CDS or UC 
treatment group. OCHIN will enroll at least three service areas with at least 10 total constituent 
primary care clinics as CI-CDS replication sites; a similar number of service areas and primary 
care clinics will serve as usual care (UC) replication sites. Stratified randomization by SA size 
may be used to balance the number of constituent clinics in each treatment group. Replication 
site clinics must currently have access to cardiovascular (CV) Wizard to be eligible for 
randomization. Service areas were chosen as a unit of randomization to avoid intervention 
contamination within groups of these highly related clinics. 
 
CI-CDS and UC service areas will be approached for participation in the study. Service areas, 
and their constituent clinics, that agree to participate by allowing full or silent implementation of 
the CI-CDS system will be considered enrolled.  
 
Recruitment will be conducted by the OCHIN study team, with support as needed from the 
HealthPartners staff. Eligible service areas and their clinics will be sent an introductory recruitment 
email with relevant study-related information, such as outlining the goals of the study, the CI 
Wizard tool, and an invitation to participate. Study team staff will follow-up via email, telephone, 
or video conference platform to answer any questions, provide demos of CI Wizard CDS, and/or 
share additional information regarding participation with service areas that are interested, as 
appropriate. 
 
Once a service area agrees to participate, OCHIN will follow up with a Memorandum of 
Understanding (MOU) that outlines participation expectations, a timeline for study activities, and 
details regarding compensation. These MOUs will be signed by OCHIN and leadership at the 
participating service areas prior to starting study activities. Service areas will receive an impact 
payment of up to $3000 as compensation for the time spent engaging in study activities. This 
impact payment will be distributed by OCHIN and is provided to the service area and not to any 
individual within the service area or clinic.  
 
4.3 Study Enrollment Procedures 
 
The clinician survey will not be conducted at the replication site and thus any survey-related 
procedures are not applicable.   
 
5. STUDY INTERVENTIONS 
 
5.1 Interventions, Administration, and Duration  
 
At the replication site, the CI-CDS user interface will only be operative in intervention service 
areas that are approached and agree to fully implement the CI-CDS. The CI-CDS will run invisibly 
Matthew Jones Primary 
Analyst 
OCHIN, Inc. 
jonesm@ochin.org 
• 
Data collection 
planning at 
replication site 
 
NCT05723523

in the background in all enrolled service areas. Having the CI-CDS run invisibly at all study sites 
ensures that study and intervention eligibility are assessed identically for all patients with visits at 
all study clinics in all enrolled service areas. See Section 9.2 of this appendix for more information 
about randomization procedures. 
 
CI-CDS user interface functionality may vary slightly between service areas within the replication 
site, such as differences in quick order availability and region-specific patient resource materials. 
For example, if quick orders are not feasible in some service areas, a static recommendation to 
place an order may be implemented instead. 
 
The CI-CDS tool will function similarly at the replication site and the primary site, with OCHIN’s 
EHR securely exchanging data with the HealthPartners web service at every encounter of adult 
patients aged 65 or older. The OCHIN EHR data securely exchanged via the HealthPartners web 
service will initially be saved in an operational repository at HealthPartners. Limited EHR and 
Wizard use data will then be securely transferred and stored in a research repository at 
HealthPartners used for study purposes using a random study ID to reduce the risk of breach of 
confidentiality. See Section 10 of this appendix for more information on data that will be stored in 
this research repository. 
 
5.4 Adherence Assessment  
 
Print rates by rooming staff at replication site clinics will be measured and grouped by clinic and 
clinician but will not be reported monthly to clinic leadership.  
 
6. STUDY PROCEDURES 
 
6.1 Schedule of Evaluations 
 
Health care claims databases (e.g., paid amounts for emergency room visits or inpatient stays) 
are not available to be utilized at the replication site.  
 
6.2. Description of Evaluations 
 
6.2.5 Clinician surveys 
 
The clinician survey will not be conducted at the replication site and thus these procedures are 
not applicable.   
 
6.2.6 Chart audits  
 
Chart audits will not be conducted at the replication site and thus these procedures are not 
applicable.  
 
6.2.7 Patient / caregiver dyad interviews  
 
Interviews will not be conducted at the replication site and thus these procedures are not 
applicable.  
 
7. SAFETY ASSESSMENTS 
 
7.1 Specification of Safety Parameters 
NCT05723523

 
Clinicians at replication site clinics can use the Feedback Tab to let the OCHIN team know of any 
questions or potential errors in the CDS. The OCHIN team will triage any feedback received 
through the Feedback Tab and communicate to the HealthPartners team if appropriate.  
 
7.2. Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters 
 
Surveillance of information stored in the Wizard repository will be used to monitor safety events 
at the replication site. 
 
7.3 Clinician Feedback and Safety Events  
 
7.3.1 Clinician Feedback  
 
Use of the Feedback Tab at replication site clinics will generate an email that is sent to the OCHIN 
study team members. The OCHIN team will communicate any feedback received through the 
Feedback Tab to the HealthPartners Team, when appropriate. The study teams will then discuss 
this feedback and any necessary actions, and reply to the clinician to answer the question, discuss 
steps taken to address the issue, or gather additional information if needed to further trouble-
shoot.  
 
7.3.2 Safety events 
 
Due to limitations in availability of data for emergency department visits, hospitalizations, and 
deaths at the replication site, we are unable to monitor these safety events at this site. We will 
measure suicide attempts as a safety event at the replication site.  
 
9. STATISTICAL CONSIDERATIONS  
 
9.2 Sample Size and Randomization  
 
Operational considerations at the replication site make clinic-randomization impractical. To 
accommodate these considerations, randomization will be conducted by service area rather than 
clinic. Service areas eligible for randomization are those that currently have access to CV Wizard.  
 
9.2.1 Treatment Assignment Procedures 
 
All eligible service areas will be randomized 1:1 to CI-CDS intervention or Control to maximize 
the likelihood that CI-CDS and UC service areas are similar on observed and unobserved 
characteristics. Service areas may be stratified based on number of clinics to ensure a balanced 
number of clinics between intervention and control groups. Service areas in the intervention group 
will be approached to participate in the study, which includes turning on the CI-CDS user interface. 
If they do not agree, they will be invited to turn on the CI-CDS invisibly to collect data in the 
background. CI-CDS service areas that agree to either of these options will be considered 
enrolled. Service areas randomized to UC that agree to run the CI-CDS invisibly in the background 
will be considered enrolled.  
 
 
9.4 Outcomes 
 
9.4.2 Secondary Outcomes 
NCT05723523

 
Secondary Outcomes will not be assessed at the replication site and thus these procedures are 
not applicable.  
 
10. DATA COLLECTION AND QUALITY ASSURANCE 
 
The sole data source for the replication site is OCHIN EHR data. No claims-based data are 
available, nor will survey data be collected, at the replication site. The following EHR data will be 
collected from the CI-CDS tool and will be saved in an operational repository at HealthPartners. 
Individual and clinic identifying information, including patient name, date of birth, clinician 
name/ID, clinic name/ID, and department ID will then be removed from the data set and replaced 
by a random identifier prior to saving it into the research data repository.  
 
Clinic-level data 
Clinic name and/or ID, department ID, clinician name and/or ID 
Patient-level data 
Patient name (purged nightly), date of birth (purged nightly), age, 
gender, race, diagnosis (with dates), lab values (with dates), vitals 
(with dates), allergies (with dates), Rx (with dates), assessment 
results (with dates) 
CI Wizard use data  
Wizard print rates and click rates   
 
 
NCT05723523

16. REFERENCES 
1. 
Livingston, G., et al., Dementia prevention, intervention, and care. Lancet, 2017. 
390(10113): p. 2673-2734. 
2. 
Prince, M., et al., World Alzheimer Report 2015. The Global Impact of Dementia. An 
analysis of prevalence, incidence, cost and trends. 2015 Alzheimer’s Disease 
International: London  
3. 
Rocca, W.A., et al., Trends in the incidence and prevalence of Alzheimer's disease, 
dementia, and cognitive impairment in the United States. Alzheimers Dement, 2011. 7(1): 
p. 80-93. 
4. 
Bunn, F., et al., Psychosocial factors that shape patient and carer experiences of dementia 
diagnosis and treatment: a systematic review of qualitative studies. PLoS Med, 2012. 
9(10): p. e1001331. 
5. 
Cooper, C., T.B. Balamurali, and G. Livingston, A systematic review of the prevalence and 
covariates of anxiety in caregivers of people with dementia. Int Psychogeriatr, 2007. 19(2): 
p. 175-95. 
6. 
Goren, A., et al., Impact of caring for persons with Alzheimer's disease or dementia on 
caregivers' health outcomes: findings from a community based survey in Japan. BMC 
Geriatr, 2016. 16: p. 122. 
7. 
Brayne, C., C. Fox, and M. Boustani, Dementia screening in primary care: is it time? 
JAMA, 2007. 298(20): p. 2409-11. 
8. 
Bradford, A., et al., Missed and delayed diagnosis of dementia in primary care: prevalence 
and contributing factors. Alzheimer Dis Assoc Disord, 2009. 23(4): p. 306-14. 
9. 
Amjad, H., et al., Underdiagnosis of Dementia: an Observational Study of Patterns in 
Diagnosis and Awareness in US Older Adults. J Gen Intern Med, 2018. 33(7): p. 1131-
1138. 
10. 
Pinner, G. and W.P. Bouman, Attitudes of patients with mild dementia and their carers 
towards disclosure of the diagnosis. Int Psychogeriatr, 2003. 15(3): p. 279-88. 
11. 
Nielsen, C. and P. Batur, Running in place: The uncertain future of primary care internal 
medicine. Cleve Clin J Med, 2019. 86(8): p. 530-534. 
12. 
Hinton, L., et al., Practice constraints, behavioral problems, and dementia care: primary 
care physicians' perspectives. J Gen Intern Med, 2007. 22(11): p. 1487-92. 
13. 
Boustani, M., et al., Who refuses the diagnostic assessment for dementia in primary care? 
Int J Geriatr Psychiatry, 2006. 21(6): p. 556-63. 
14. 
Bernstein, A., et al., Primary Care Provider Attitudes and Practices Evaluating and 
Managing Patients with Neurocognitive Disorders. J Gen Intern Med, 2019. 34(9): p. 1691-
1692. 
15. 
McKhann, G.M., et al., The diagnosis of dementia due to Alzheimer's disease: 
recommendations from the National Institute on Aging-Alzheimer's Association 
workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement, 2011. 
7(3): p. 263-9. 
16. 
Lin, J.S., et al., Screening for cognitive impairment in older adults: A systematic review for 
the U.S. Preventive Services Task Force. Ann Intern Med, 2013. 159(9): p. 601-12. 
17. 
Galvin, J.E., C.H. Sadowsky, and A. Nincds, Practical guidelines for the recognition and 
diagnosis of dementia. J Am Board Fam Med, 2012. 25(3): p. 367-82. 
18. 
Mukadam, N., et al., A systematic review of interventions to detect dementia or cognitive 
impairment. Int J Geriatr Psychiatry, 2015. 30(1): p. 32-45. 
19. 
Folstein, M.F., S.E. Folstein, and P.R. McHugh, "Mini-mental state". A practical method 
for grading the cognitive state of patients for the clinician. J Psychiatr Res, 1975. 12(3): p. 
189-98. 
NCT05723523

20. 
Tsoi, K.K., et al., Cognitive Tests to Detect Dementia: A Systematic Review and Meta-
analysis. JAMA Intern Med, 2015. 175(9): p. 1450-8. 
21. 
Velayudhan, L., et al., Review of brief cognitive tests for patients with suspected dementia. 
Int Psychogeriatr, 2014. 26(8): p. 1247-62. 
22. 
Seitz, D.P., et al., Mini-Cog for the diagnosis of Alzheimer's disease dementia and other 
dementias within a primary care setting. Cochrane Database Syst Rev, 2018. 2: p. 
CD011415. 
23. 
Borson, S., et al., The Mini-Cog as a screen for dementia: validation in a population-based 
sample. J Am Geriatr Soc, 2003. 51(10): p. 1451-4. 
24. 
Borson, S., et al., Implementing routine cognitive screening of older adults in primary care: 
process and impact on physician behavior. J Gen Intern Med, 2007. 22(6): p. 811-7. 
25. 
Rosenbloom, M., et al., Screening Positive for Cognitive Impairment: Impact on 
Healthcare Utilization and Provider Action in Primary and Specialty Care Practices. J Gen 
Intern Med, 2018. 33(10): p. 1746-1751. 
26. 
Yokomizo, J.E., S.S. Simon, and C.M. Bottino, Cognitive screening for dementia in primary 
care: a systematic review. Int Psychogeriatr, 2014. 26(11): p. 1783-804. 
27. 
Bernstein, A., et al., Dementia assessment and management in primary care settings: a 
survey of current provider practices in the United States. BMC Health Serv Res, 2019. 
19(1): p. 919. 
28. 
Wagner, E.H., B.T. Austin, and M. Von Korff, Organizing care for patients with chronic 
illness. Milbank Mem Fund Q, 1996. 74(4): p. 511-544. 
29. 
Von Korff, M., et al., Collaborative management of chronic illness. Ann Intern Med, 1997. 
127(12): p. 1097-102. 
30. 
Wagner, E.H., Chronic disease management: what will it take to improve care for chronic 
illness? Eff Clin Pract, 1998. 1(1): p. 2-4. 
31. 
Yawn, B., et al., Time use during acute and chronic illness visits to a family physician. Fam 
Pract, 2003. 20(4): p. 474-7. 
32. 
Parchman, M.L. and S.K. Burge, Continuity and quality of care in type 2 diabetes: a 
Residency Research Network of South Texas study. Journal of Family Practice, 2002. 
51(7): p. 619-24. 
33. 
Greenfield, S., et al., Patients' participation in medical care: effects on blood sugar control 
and quality of life in diabetes. J Gen Intern Med, 1988. 3(5): p. 448-57. 
34. 
Tariq, S.H., et al., Comparison of the Saint Louis University mental status examination 
and the mini-mental state examination for detecting dementia and mild neurocognitive 
disorder--a pilot study. Am J Geriatr Psychiatry, 2006. 14(11): p. 900-10. 
35. 
Nasreddine, Z.S., et al., The Montreal Cognitive Assessment, MoCA: a brief screening 
tool for mild cognitive impairment. J Am Geriatr Soc, 2005. 53(4): p. 695-9. 
36. 
Han, L., et al., Tracking cognitive decline in Alzheimer's disease using the mini-mental 
state examination: a meta-analysis. Int Psychogeriatr, 2000. 12(2): p. 231-47. 
37. 
Barnes, D.E., et al., Development and Validation of eRADAR: A Tool Using EHR Data to 
Detect Unrecognized Dementia. J Am Geriatr Soc, 2019. 
38. 
Petersen, R.C., et al., Author response: Practice guideline update summary: Mild cognitive 
impairment: Report of the Guideline Development, Dissemination, and Implementation 
Subcommittee of the American Academy of Neurology. Neurology, 2018. 91(8): p. 373-
374. 
39. 
Duara, R., et al., Medial temporal lobe atrophy on MRI scans and the diagnosis of 
Alzheimer disease. Neurology, 2008. 71(24): p. 1986-92. 
40. 
Burton, E.J., et al., Medial temporal lobe atrophy on MRI differentiates Alzheimer's disease 
from dementia with Lewy bodies and vascular cognitive impairment: a prospective study 
with pathological verification of diagnosis. Brain, 2009. 132(Pt 1): p. 195-203. 
NCT05723523

41. 
Tanner, J.A., et al., A randomized controlled trial of a community-based dementia care 
coordination intervention: effects of MIND at Home on caregiver outcomes. Am J Geriatr 
Psychiatry, 2015. 23(4): p. 391-402. 
42. 
Selwood, A., et al., Systematic review of the effect of psychological interventions on family 
caregivers of people with dementia. J Affect Disord, 2007. 101(1-3): p. 75-89. 
43. 
Manthorpe, J. and J. Moriarty, Nothing ventured, nothing gained: New guidance on risk. 
Journal of Dementia Care, 2011. 19: p. 20-22. 
44. 
Lord, K., G. Livingston, and C. Cooper, A systematic review of barriers and facilitators to 
and interventions for proxy decision-making by family carers of people with dementia. Int 
Psychogeriatr, 2015. 27(8): p. 1301-12. 
45. 
Lord, K., et al., How people with dementia and their families decide about moving to a care 
home and support their needs: development of a decision aid, a qualitative study. BMC 
Geriatr, 2016. 16: p. 68. 
46. 
Manthorpe, J., et al., Dementia, dignity and quality of life: nursing practice and its 
dilemmas. Int J Older People Nurs, 2010. 5(3): p. 235-44. 
47. 
Aalten, P., et al., Neuropsychiatric syndromes in dementia. Results from the European 
Alzheimer Disease Consortium: part I. Dement Geriatr Cogn Disord, 2007. 24(6): p. 457-
63. 
48. 
Lyketsos, C.G., et al., Prevalence of neuropsychiatric symptoms in dementia and mild 
cognitive impairment: results from the cardiovascular health study. JAMA, 2002. 288(12): 
p. 1475-83. 
49. 
Steinberg, M., et al., The persistence of neuropsychiatric symptoms in dementia: the 
Cache County Study. Int J Geriatr Psychiatry, 2004. 19(1): p. 19-26. 
50. 
Ryu, S.H., et al., Persistence of and changes in neuropsychiatric symptoms in Alzheimer 
disease over 6 months: the LASER-AD study. Am J Geriatr Psychiatry, 2005. 13(11): p. 
976-83. 
51. 
Cohen-Mansfield, J., et al., The impact of psychotic symptoms on the persons with 
dementia experiencing them. Am J Geriatr Psychiatry, 2016. 24(3): p. 213-20. 
52. 
Cummings, J., et al., Agitation in cognitive disorders: International Psychogeriatric 
Association provisional consensus clinical and research definition. Int Psychogeriatr, 
2015. 27(1): p. 7-17. 
53. 
Cohen-Mansfield, J., Conceptualization of agitation: results based on the Cohen-
Mansfield Agitation Inventory and the Agitation Behavior Mapping Instrument. Int 
Psychogeriatr, 1996. 8 Suppl 3: p. 309-15; discussion 351-4. 
54. 
Cummings, J.L., et al., The Neuropsychiatric Inventory: comprehensive assessment of 
psychopathology in dementia. Neurology, 1994. 44(12): p. 2308-14. 
55. 
Morris, S., et al., Monetary costs of agitation in older adults with Alzheimer's disease in 
the UK: prospective cohort study. BMJ Open, 2015. 5(3): p. e007382. 
56. 
Cohen-Mansfield, J., et al., Can agitated behavior of nursing home residents with 
dementia be prevented with the use of standardized stimuli? J Am Geriatr Soc, 2010. 
58(8): p. 1459-64. 
57. 
Logsdon, R.G. and L. Teri, The Pleasant Events Schedule-AD: psychometric properties 
and relationship to depression and cognition in Alzheimer's disease patients. 
Gerontologist, 1997. 37(1): p. 40-5. 
58. 
Cohen-Mansfield, J., et al., Can persons with dementia be engaged with stimuli? Am J 
Geriatr Psychiatry, 2010. 18(4): p. 351-62. 
59. 
Ballard, C., et al., Brief psychosocial therapy for the treatment of agitation in Alzheimer 
disease (the CALM-AD trial). Am J Geriatr Psychiatry, 2009. 17(9): p. 726-33. 
60. 
Mahoney, R., et al., Anxiety and depression in family caregivers of people with Alzheimer 
disease: the LASER-AD study. Am J Geriatr Psychiatry, 2005. 13(9): p. 795-801. 
NCT05723523

61. 
Institute of Medicine, The computer-based patient record.  An essential technology for 
health care. 1991, Washington DC: National Academy Press. 190. 
62. 
Institute of Medicine, Health and data in the information age.  Use, disclosure, and privacy. 
1994, Washington DC: National Academy Press. 257. 
63. 
Meigs, J.B., et al., A controlled trial of web-based diabetes disease management: the MGH 
diabetes primary care improvement project. Diabetes Care, 2003. 26(3): p. 750-7. 
64. 
Cleveringa, F.G., et al., Computerized decision support systems in primary care for type 
2 diabetes patients only improve patients' outcomes when combined with feedback on 
performance and case management: a systematic review. Diabetes Technol Ther, 2013. 
15(2): p. 180-92. 
65. 
Montori, V.M., et al., The impact of planned care and a diabetes electronic management 
system on community-based diabetes care: the Mayo Health System Diabetes Translation 
Project. Diabetes Care, 2002. 25(11): p. 1952-7. 
66. 
Crosson, J.C., et al., Implementing an electronic medical record in a family medicine 
practice: communication, decision making, and conflict. Ann Fam Med, 2005. 3(4): p. 307-
11. 
67. 
Tierney, W.M., et al., Effects of computerized guidelines for managing heart disease in 
primary care. J Gen Intern Med, 2003. 18(12): p. 967-76. 
68. 
Murray, M.D., et al., Failure of computerized treatment suggestions to improve health 
outcomes of outpatients with uncomplicated hypertension: results of a randomized 
controlled trial. Pharmacotherapy, 2004. 24(3): p. 324-37. 
69. 
Tierney, W.M., et al., Can computer-generated evidence-based care suggestions 
enhance evidence-based management of asthma and chronic obstructive pulmonary 
disease? A randomized, controlled trial. Health Serv Res, 2005. 40(2): p. 477-98. 
70. 
Overhage, J.M., et al., A randomized trial of "corollary orders" to prevent errors of 
omission. J Am Med Inform Assoc, 1997. 4(5): p. 364-75. 
71. 
O'Connor, P.J., et al., Impact of electronic health record clinical decision support on 
diabetes care: a randomized trial. Ann Fam Med, 2011. 9(1): p. 12-21. 
72. 
Ziemer, D.C., et al., An intervention to overcome clinical inertia and improve diabetes 
mellitus control in a primary care setting: Improving Primary Care of African Americans 
with Diabetes (IPCAAD) 8. Arch Intern Med, 2006. 166(5): p. 507-13. 
73. 
Peterson, K.A., et al., Improving Diabetes Care in Practice: findings from the TRANSLATE 
trial. Diabetes Care, 2008. 31(12): p. 2238-43. 
74. 
Grant, R.W., et al., Practice-linked online personal health records for type 2 diabetes 
mellitus: a randomized controlled trial. Arch Intern Med, 2008. 168(16): p. 1776-82. 
75. 
O'Connor, P.J., et al., Customized feedback to patients and providers failed to improve 
safety or quality of diabetes care: a randomized trial. Diabetes Care, 2009. 32(7): p. 1158-
63. 
76. 
O'Connor, P.J., et al., Impact of an electronic medical record on diabetes quality of care. 
Ann Fam Med, 2005. 3(4): p. 300-6. 
77. 
Montori, V.M. and G.H. Guyatt, Progress in evidence-based medicine. JAMA, 2008. 
300(15): p. 1814-6. 
78. 
Agoritsas, T., et al., Decision aids that really promote shared decision making: the pace 
quickens. BMJ, 2015. 350: p. g7624. 
79. 
Hargraves, I., et al., Shared Decision Making: The Need For Patient-Clinician 
Conversation, Not Just Information. Health Aff (Millwood), 2016. 35(4): p. 627-9. 
80. 
Borrelli, B., et al., A new tool to assess treatment fidelity and evaluation of treatment fidelity 
across 10 years of health behavior research. J Consult Clin Psychol, 2005. 73(5): p. 852-
60. 
81. 
Kharbanda, E.O., et al., Clinical Decision Support for Recognition and Management of 
Hypertension: A Randomized Trial. Pediatrics, 2018. 141(2). 
NCT05723523

82. 
Stuckey, H.L., Three types of interviews: Qualitative research methods in social health. 
Journal of Social Health and Diabetes, 2013. 1(02). 
83. 
Greene, E.J., A SAS Macro for Covariate-Constrained Randomization of General Cluster-
Randomized and Unstratified Designs. J Stat Softw, 2017. 77(Cs1). 
84. 
PASS 2019 Power Analysis and Sample Size Software. 2019, NCSS, LLC: Kaysville, 
Utah. 
85. 
Fowler FJ, Improving Survery Questions. Applied Social Research Methods (Book 38). 
1995: SAGE Publications, Inc. 
86. 
Duan N, et al., Choosing between the sample-selection model and the multi-part model. 
Journal of Business and Economic Statistics, 1984: p. 3. 
87. 
Blough, D., C. Madden, and M. Hornbrook, Modeling risk using generalized linear models. 
J Health Econ, 1999. 18(2): p. 153-71. 
88. 
Manning, W.G. and J. Mullahy, Estimating log models: to transform or not to transform? J 
Health Econ, 2001. 20(4): p. 461-94. 
89. 
Park R.E., Estimation with heteroscedastic error terms. 1966. Econometrica(4): p. 888. 
 
NCT05723523