CV Wizard: Does a Prioritized, Point-of-Care Clinical Decision Support Tool Improve Guideline-Based CVD Risk Factor Control in Safety Net Clinics?
Phase: NA
Conditions: Heart Attack, Stroke, Cardiovascular Diseases
Sponsor: Kaiser Permanente

 
Study Protocol: CV Wizard: Does a Clinical Decision Support Tool Improve CVD Risk Factor 
Control in Safety Net Clinics?  
 
NCT#: NCT03001713  
 
Document Date: July 28, 2022 
 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 2 of 27 
 
1. Protocol Title 
 
CV Wizard: Does a Prioritized, Point-of-Care Clinical Decision Support Tool Improve Guideline-Based 
CVD Risk Factor Control in Safety Net Clinics? 
 
2. Objectives 
 
This project aims to reduce disparities in cardiovascular disease (CVD) risk factor control and in rates 
of heart attacks and strokes among the low-income, racially/ethnically diverse Americans who 
receive primary care at safety net community health centers (CHCs). To achieve this objective, we 
will implement a successful clinical decision support (CDS) system—CV WIZARD (currently used in 
CVD care at several large, integrated health care systems)—in at least 60 CHCs, with a staggered 
randomized implementation approach. We will modify CV WIZARD as needed and feasible to fit CHC 
preferences. We will determine if use of this CDS improves CVD care, reduces disparities in CVD care 
and outcomes, and increases patient engagement in CVD treatment choices, in CHCs.  
 
Specific Aims  
 
Aim 1. Conduct a clinic-randomized trial of the impact of an evidence-based point-of-care CDS 
system on (i) overall CVD risk scores, and (ii) control of individual CVD risk factors (blood 
pressure; HbA1c, lipid levels; smoking; body mass index), among high CVD risk adult CHC 
patients.  
 
Aim 2. Develop and hone need-based implementation support protocols to help Arm 1 CHCs 
implement the CV Wizard CDS system into their standard workflows; assess whether use of the 
protocols developed for Arm 1 CHCs accelerates implementation and adoption of the CDS 
system in the Arm 2 CHCs. Supplement Addendum: Compare CVD risk management in high-risk 
patients at in-person vs. virtual care (VC) encounters, in the periods pre- and post-COVID’s 
onset. 
 
Aim 3. Conduct a mixed methods process evaluation, guided by the Technology Acceptance 
Model (TAM), to identify and address patient, provider, and delivery system barriers to 
uptake/impact of this CDS in CHCs. Supplement Addendum: In a subset of study clinics, explore 
how CV Wizard’s individualized, prioritized CVD risk summary is used in VC to guide care 
decisions and engage patients in shared decision-making. 
 
3. Background 
 
Substantial progress in reducing cardiovascular disease (CVD) morbidity and mortality would 
be achieved if evidence-based guidelines for CVD risk factor control were implemented 
consistently in primary care settings. Electronic health record (EHR)-based clinical decision 
support (CDS) systems that identify uncontrolled CVD risk factors and provide individualized 
care recommendations improved rates of guideline-concordant CVD care in large, 
integrated healthcare settings, but little is known about how effective such CDS may be in 
safety net community health centers (CHCs). CHCs’ socioeconomically vulnerable patients 
have far worse CVD risk factor control and higher rates of major CVD events than the 
general population. Implementing CDS that leads to improved CVD risk factor control in 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 3 of 27 
 
CHCs could reduce national disparities in CVD outcomes, but CHCs rarely have the resources 
to develop sophisticated CDS, and very few currently have such systems for CVD care. The 
proposed study is designed to address this issue. Supplement Addendum: Many healthcare 
settings rapidly shifted to virtual care (VC; defined here as video or telephone clinical 
encounters) because of the COVID-19 pandemic. Little is known about whether or how the 
shift to VC is impacting CVD risk management in primary care. The potential detrimental 
impacts of this shift are particularly concerning in community health centers (CHCs), which 
serve vulnerable patients who face health disparities (e.g., higher rates of unmanaged CVD 
risk than in the general population). However, the impact of VC on disparities in CVD care 
quality and outcomes remains largely unexplored. In response, we adapted CV Wizard for 
use in the VC setting and will assess how the COVID-19-driven shift to VC impacts CVD risk 
management in CHC populations. 
 
NOTE:  The bibliography has been uploaded as a separate document in the eIRB. 
 
4. Study Design 
 
CV WIZARD is a clinical decision support (CDS) tool that provides point-of-care 
cardiovascular disease (CVD) care recommendations to the primary care team and the 
patient. CV Wizard identifies a patient’s uncontrolled CVD risk factors, prioritizes those 
factors based on potential CVD risk reduction for that patient, and generates specific 
guideline-based treatment recommendations for each uncontrolled risk factor. 
 
We will recruit at least 60 safety net community health centers (CHCs) and randomize them 
to implement CV Wizard in study year 2 (Arm 1), or 18 months later (Arm 2). Prior to full 
implementation of the CV Wizard in Arm 1, two CHC organizations will be engaged to pilot-
test CV Wizard. The pilot sites will test the accuracy and usability of the tool in clinic 
workflows. 
 
We will compare outcomes from Arm 1 (immediate implementation) and Arm 2 (delayed 
implementation), which will enable us to measure the intervention’s impact on CVD risk 
factor control in CHCs. Arm 1 CHCs will receive implementation support to address any 
barriers to adoption/sustained use of the CDS tool that are identified through study 
activities. We will apply these learnings to improve adoption rates in Arm 2.  
 
Through this design, we will (1) assess whether this CDS tool is effective in CHCs, (2) identify 
and address barriers to its effectiveness and adoption, and (3) fine-tune strategies to 
support implementation of CDS systems in CHCs. To meet these aims, we will conduct mixed 
methods analyses (described below). Our assessment includes interviews with CHC staff, 
providers, and patients to determine if use of CV WIZARD increases patient engagement in 
CVD treatment choices. 
 
5. Study Population 
 
a. Number of Subjects 
 
NOTE:  This study does not include any KP subjects.  

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 4 of 27 
 
 
Study Aim 1 
Clinics – We will randomize at least 60 CHCs from within the OCHIN collaborative to either 
immediate (Arm 1) or delayed (Arm 2) CV Wizard tool implementation. While CV Wizard will be 
used with patients in those clinics, they are not considered study subjects. 
 
Study Aim 2 
Clinic Staff 
• 
We will conduct semi-structured phone interviews with staff from 20 clinics (10 per 
study arm). We will interview at least one staff member from each of these clinics and 
may conduct follow-up “clarifying” interviews as needed with up to 5 additional staff 
per clinic. Up to 60 clinic staff members may be interviewed for this data collection 
component. 
• 
We will ask selected clinics (may include main trial and pilot clinics) to send a survey to 
their providers. While we do not know how many providers will respond to the survey, 
it is possible we will receive up to 500 responses.  
 
Study Aim 3 
Case Study Clinics – We will recruit four clinics from those taking part in the study (two 
per study arm; may include main trial and pilot clinics) for additional qualitative data 
collection. For these case study clinics, we will conduct semi-structured interviews with: 
• 
Patients whose most recent encounter involved CV Wizard. We will conduct up to 
60 patient interviews in total (up to 30 per study arm). 
• 
Staff (providers and non-providers) who use the CV Wizard tool with patients or are 
involved in implementation efforts. We will conduct up to 60 staff interviews in 
total (up to 30 per study arm). 
 
Supplement Clinics – We will recruit 10 clinics from those taking part in the study (from both 
study arms; may include main trial and pilot clinics) to collect qualitative data on how CV 
Wizard’s individualized, prioritized CVD risk summary is used in VC to guide care decisions and 
engage patients in shared decision-making.  
• 
We will interview 40 providers for whom CV Wizard was recommended at ≥3 VC encounters 
in the period post-COVID’s onset. 
• 
We will interview 30 patients who had a VC encounter at which CV Wizard was used in the 
post-COVID period from these same clinics. 
 
b. Inclusion and Exclusion Criteria 
 
CV Wizard – Clinical Decision Support Tool (Study Aim 1) 
NOTE: Individual patients are not being enrolled for this clinic-randomized study. The 
goal of the study is to examine the uptake and impact of implementing CV Wizard into 
regular care processes. In this clinic-randomized trial, the intervention is provided at the 
clinic level and targets clinic processes that are part of routine patient care. CV Wizard 
will automatically identify patients age 40-75 years with either (a) reversible risk of 
cardiovascular disease (CVD) >10%, (b) diabetes + ≥1 uncontrolled risk factor or high 
A1c, or (c) existing atherosclerotic cardiovascular disease (ASCVD) + ≥1 uncontrolled risk 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 5 of 27 
 
factor for CVD events. However, clinics can manually run the CV Wizard tool for any 
patient aged 18 and older.  
 
Interviews with Clinic Staff and Patients (Study Aims 2 & 3) 
Inclusion Criteria: 
• 
Aged 18 years and older 
• 
Experience with CV Wizard tool and/or telehealth visit 
Exclusion Criteria: 
• 
Non-English or Non-Spanish speaker  
 
Provider Survey (Study Aims 2 & 3) 
Inclusion Criteria: 
• 
Aged 18 years and older 
Exclusion Criteria: 
• 
Non-English speaker 
 
c. Vulnerable Populations 
 
CV Wizard – Clinical Decision Support Tool (Study Aim 1) 
While individual patients are not being recruited/enrolled in this study, the CV Wizard tool may 
be used with individual patients as part of their routine care. Below is a list of vulnerable 
population categories and whether they will be included/excluded with regard to use of the CV 
Wizard tool: 
• 
Children: Excluded 
• 
Pregnant Women: Excluded (when identified as having a current pregnancy diagnosis); 
may be incidentally included, but not in a focused or targeted manner 
• 
Neonates: Excluded 
• 
Prisoners: Excluded (this study does not include any clinics that serve the prison 
population) 
• 
Decisionally Impaired Adults: May be incidentally included, but not in a focused or 
targeted manner—just as part of the overall qualified sample 
 
NOTE: Children younger than 18 years will be excluded, as different clinical 
guidelines apply to children with CVD risk. The study will not preferentially recruit 
any vulnerable populations. However, the criteria listed above may result in the 
inclusion of pregnant women and/or decisionally impaired adults. It is important to 
systematically address CV risk in these populations, as these patients may be at risk 
for elevated CV risk and often have been underrepresented in previous research. 
 
Interviews with Clinic Staff/Patients and Provider Survey (Study Aims 2 & 3) 
Below is a list of vulnerable population categories and whether they will be included/excluded 
with regard to clinic staff/patient interviews and provider surveys: 
• 
Children: Excluded 
• 
Pregnant Women: May be incidentally included, but not in a focused or targeted 
manner—just as part of the overall qualified sample 
• 
Neonates: Excluded 
• 
Prisoners: Excluded 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 6 of 27 
 
• 
Decisionally Impaired Adults: May be incidentally included, but not in a focused or 
targeted manner—just as part of the overall qualified sample. 
 
d. Setting 
 
1. OCHIN, Inc. – OCHIN, Inc. is a non-profit, community-based health center-controlled 
network. Its members (>480 CHCs, in 19 states) share characteristics of other CHCs, so 
results will be generalizable to many CHCs. As the nation’s largest CHC network with a single 
EHR system, OCHIN pioneered the development of EHR tools for CHCs. OCHIN’s member 
CHCs share a single Epic© EHR, which is unduplicated, centrally maintained, and network-
wide. Data are standardized and quality-checked: thus, validated data are already linked 
between all study sites.  
 
 
 
Procedures Performed 
• 
Recruit member clinics to take part in the study 
• 
Randomize clinics to intervention arm 
• 
Complete necessary programming to enable CV Wizard tool to accept OCHIN data 
• 
Perform validity testing to ensure CV Wizard works correctly with OCHIN’s EHR data 
• 
Engage OCHIN’s Patient Engagement Panel to obtain patient feedback on CV Wizard 
• 
Develop implementation training materials  
• 
Provide implementation support to participating clinics 
• 
Assist in the collection of qualitative data (e.g., webinar recordings, issues log, 
provider survey administration, etc.) 
• 
Conduct quantitative data analysis 
• 
Ensure all required MOUs for participating clinic have been fully executed; 
also ensure that modified/amended MOUs have been fully executed before 
the modification is implemented 
 
IRB Review Requirements – OCHIN will rely on the Kaiser Permanente Northwest IRB for 
all IRB review.  
 
2. HealthPartners Institute (HPI) – HPI is one of the largest medical research and education 
centers in the Midwest. As part of HealthPartners integrated care system, the Institute uses 
research and education to accelerate improvements in quality, experience and affordability 
for members, patients, and the community. HPI owns the CV Wizard tool that will be used in 
this study.  
 
Study Procedures Performed 
• 
Work with OCHIN programmers to ensure CV Wizard can accept OCHIN data 
• 
Receive data from OCHIN through a secure file transfer to calculate the inputs 
necessary to populate the CV Wizard output.   
• 
Send back output data so CV Wizard recommendations can be displayed in OCHIN’s 
EHR for clinical use at participating clinics.   
• 
Send output data file to OCHIN via secure file transfer that can be used for analyses 
and monitoring of CV Wizard use in the participating study clinics. 
• 
Updating CV Wizard tool as needed/requested by OCHIN and study clinics 
 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 7 of 27 
 
IRB Review Requirements – HPI will rely on the Kaiser Permanente Northwest IRB for all 
IRB review. 
 
3. Center for Health Research—Northwest – The Center for Health Research–Northwest (CHR-
NW) is an academic-model organization that conducts independent research in a wide 
variety of areas including health services, public health, behavioral health, obesity and 
weight loss, mental health, maternal health, cost-effectiveness, cancer screening, genetics 
and genomics, and many others. In addition to conducting studies with our own databases 
and with KP’s EMRs, CHR-NW researchers participate in a variety of formal and informal 
research networks. We collaborate locally with scientists and physicians at Oregon Health & 
Science University, and nationally with other Kaiser Permanente health plans and research 
divisions. We also undertake research in partnership with the Portland, Oregon-based 
OCHIN network of community health centers, and through our participation in networks 
such as HCSRN, CHARN, CRN, PCORnet, and many others. These networks allow our 
investigators to share data across multiple health systems and engage in multi-site research 
projects that access millions of individual health records. 
 
Study Procedures Performed 
• 
Provide coordination across study sites 
• 
Manage overall study budget and subcontract awards 
• 
Obtain IRB approval for study; submit continuing reviews; submit study 
modifications as needed 
• 
Prepare and submit all required reports to funder (e.g., recruitment reports; yearly 
progress reports) 
• 
Oversee DSMB creation and coordinate meetings 
• 
Conduct qualitative data analysis 
• 
Support subcontract sites with study tasks/deliverables as needed 
 
IRB Review Requirements – CHR-NW will rely on the Kaiser Permanente Northwest IRB 
for all IRB review. 
 
e. Recruitment Methods 
 
Clinic Recruitment (Aim 1) 
OCHIN’s Research Associate/Project Manager will work with KPCHR and the OCHIN research 
team to recruit CHCs. OCHIN’s clinical leadership (including CEOs of its member CHC 
organizations) may also help with recruitment, as needed. Recruitment will be targeted to 
optimize diversity in clinic baseline characteristics.  
 
OCHIN utilizes a variety of recruitment methods when they invite CHCs to participate in research 
studies. Recruitment activities for this study may include: 
• 
Verbal conversations, in-person or by phone, with CHCs that the OCHIN research team 
has standing relationships with  
• 
Sending an introductory recruitment email to allow clinics to opt-in to the study, with 
study-related attachments:  
o Video summarizing the CV Wizard tool use in a clinical setting 
o Slide deck with background information about the tool 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 8 of 27 
 
o One-page summary document with a high-level description of the tool and the 
study 
• 
Presentation of recruitment materials at standing OCHIN webinars and meetings where 
members are in attendance. Any interested CHCs will be encouraged to contact study 
staff to discuss potential participation.  
 
Clinic Staff Recruitment (Aim 2) 
OCHIN and CHR’s research teams, working by email and phone with clinic contacts identified 
during the recruitment process, will identity and recruit those staff members from clinics who 
have experience using CV Wizard, for phone interviews. The team will perform similar liaison 
and recruitment activities for any necessary follow-up interviews. The study team will send 
interview guide questions in advance in an email attachment or in the body of an email / 
meeting invitation. 
 
Case Study & Supplement Site Recruitment (Aim 3) 
• 
Clinics – Once clinic-level recruitment from study aim 1 is complete, an OCHIN Research 
Associate/Project Manager will identify and engage clinics to be case study and/or 
supplement sites. A separate Memorandum of Understanding (MOU) outlining the 
participation requirements for this portion of the study will be executed for these sites. 
Each clinic will be asked to appoint a staff member to be the liaison between 
researchers and clinic staff.   
• 
Staff Interviews – The KPCHR qualitative research team, supported by OCHIN study 
staff, will work with the clinic liaison to identify and recruit clinic staff to observe and/or 
interview for qualitative data collection activities. Interviews may take place either in-
person (when site visits occur), or by telephone. The study team will send interview 
guide questions in advance in an email attachment or in the body of an email / meeting 
invitation.  
• 
Patient Interviews – Patients may be recruited by clinic staff or a member of the 
research team.  
o Recruitment by Clinic Staff: If CV Wizard is used during a patient encounter while 
research staff are onsite, clinic staff will inform the patient that a researcher would 
be interested in talking with them about their perceptions of the acceptability/utility 
of CV Wizard and related discussion with the provider. If CV Wizard is used during a 
patient encounter while research staff are not onsite, clinic staff may use a variety 
of methods to help recruit or identify patients for recruitment by the research team: 
(1) they may give the patient a flyer during an in-person visit; (2) they may send the 
flyer via post mail to a patient (in cases where the visit was virtual or the patient 
was identified after the visit); (3) they may indicate the patient is a good fit by (a) 
responding to a BPA alert in the EHR (a prompt that will fire when an eligible 
encounter is detected) or (b) using a smartphrase in the after visit summary (AVS) 
that indicates CV Wizard was used during the encounter; and/or (4) they may 
message the patient (e.g., via email or MyChart account) to share contact 
information for the research team (e.g., electronically sending the flyer, using the 
email message template provided by the research team). 
Recruitment by Research Staff: If CV Wizard is used during a patient encounter that 
is being observed as part of a site visit, research staff may directly invite the patient 
to participate in the interview. If CV Wizard is used during a patient encounter when 
the research team is not onsite (e.g., virtual visits; post-covid qualitative data 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 9 of 27 
 
collection), research staff will rely on clinic staff recommendations (e.g., direct 
referral, positive responses to the BPA alert) and/or other indicators of CV Wizard 
use (e.g., smartphrase entry) to identify potential patients to recruit for the 
interviews. Once identified, the OCHIN research team will use a variety of methods 
to follow up with patients (e.g., phone calls, email, text message, and/or post mail). 
The study team will send interview guide questions in advance in an email 
attachment or in the body of an email / meeting invitation. 
NOTES: 
 
The same recruitment flyer will be used for case study and supplement clinics.  
 
All recruitment materials will provide the patient with information on how to 
contact the CHR qualitative research team if the patient is interested in 
completing the interview by telephone. Patients calling the study phone number 
will be asked to leave a message, and research staff will return phone calls to 
complete recruitment and conduct the interview.  
 
The OCHIN research team will contact Spanish speaking patients via translated 
email and / or text message with information on how to contact the CHR 
qualitative research team. A bilingual qualitative study team member will 
contact the patient to conduct the interview. 
 
In cases where there is direct patient outreach, only OCHIN research team 
members will make initial contact with patients. If patients are interested in 
moving forward with the interview, contact information will be shared with KP 
CHR staff through OCHIN’s instance of REDCap (see Section 8. Privacy, 
Confidentiality, and Data Security: Storage of Recruitment Data). KP CHR staff 
will contact patients for interviews via study issued cell phone (call or two-way 
text messaging) and / or webinar platform.  
 
Provider Survey Recruitment (Aims 2 & 3) 
The OCHIN team will email clinic contacts to introduce the provider survey. Clinic contacts will 
be asked to distribute the survey to the providers in their clinics via email, using the message 
template provided. 
 
f. Consent Process 
 
Clinic Consent (Aim 1) 
Once a clinic agrees to participate, OCHIN’s Research Associate/Project Manager will follow up 
with a Memorandum of Understanding (MOU) that outlines the study timeline and participation 
expectations. This MOU is signed by OCHIN’s leadership (either the CEO or CFO) and leadership 
at the participating CHC. A separate MOU will be executed for the two (2) pilot sites that will be 
testing the tool prior to the go-live date for Arm 1 clinics. 
 
Clinic Staff/Provider Consent (Aim 2) 
• 
Clinic staff recruited for interviews will be provided with a study fact sheet (typically via 
email) that outlines the purpose of the research, study procedures, and contact 
information for the researchers and clinic liaison(s). As the risk to CHC staff is minimal, 
we will ask for their verbal consent for all qualitative data collection activities. We will 
also record interviews with staff permission; in these cases, we will document verbal 
consent as part of the audio recording.   

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 10 of 27 
 
• 
Clinic staff invited to complete the provider survey will be presented with text in the 
survey invitation that contains the necessary elements of consent. Because the risk to 
providers is minimal, we will infer consent if the survey is completed. 
 
Case Study & Supplement Site Consent (Aim 3) 
• 
Clinics – A separate MOU outlining the participation requirements for this portion of the 
study will be executed for case study and supplement sites.  
• 
Staff Interviews – Clinic staff recruited for in-person interviews (at case study sites) will 
be provided with a study fact sheet that outlines the purpose of the research, study 
procedures, and contact information for the researchers and clinic liaison(s). They will 
be informed that participation in the interview is voluntary. Clinic staff recruited for 
telephone interviews (at case study and supplement clinic sites) will be read 
introductory text containing the required elements of consent (see provider/MA 
interview guides). As the risk to CHC staff is minimal, we will ask for their verbal consent 
for all qualitative data collection activities. We will also record interviews with 
permission from staff. When permission is given to record interviews, verbal consent 
will be documented on the audio recording. While PHI will not be solicited, it is possible 
that staff may inadvertently include PHI in their response to a question.   
• 
Patient Interviews – Patients recruited for in-person interviews (at case study sites) will 
be given the study fact sheet and will be informed that participation in the interview is 
voluntary. Patients recruited for telephone interviews (at case study and supplement 
sites) will be read introductory text containing the required elements of consent (see 
patient interview guide). For Spanish speaking patients, the consent language will be 
translated, and a bilingual study team member (English / Spanish) will conduct the 
interview in Spanish. All consent language is included in the interview guide and will be 
translated by a study team member. Because risk is minimal, and some patients will be 
interviewed by telephone, we will ask the patient to provide verbal consent for 
participation. We will also record interviews with patient permission. When permission 
is given to record interviews, verbal consent will be documented on the audio recording. 
Given that the interview will revolve around the patient’s perceptions of the CV Wizard 
tool and use of the tool itself is capturing health information, it is possible that PHI will 
be included on the recording. 
• 
Observations – We will conduct observation of workflows, patient encounters, and 
clinic meetings relevant to CV Wizard during in-person case study site visits. Unlike the 
staff and patient interviews, these observations will not be audio recorded; instead, 
researchers will take field notes. When opportunities to observe patient encounters 
arise, the patient will be introduced to research staff and verbal consent will be 
obtained prior to the observation (note: the study fact sheet will not be presented 
during observation, as this has the potential to interfere with normal clinic workflow). 
While PHI may be observed by research staff, it will not be written down as part of the 
field notes. 
 
Justification for Waiver of Documentation of Consent: 
• 
Clinic Staff/Providers – The research involves minimal risk to clinic staff. All staff 
will be informed that participation in the interview is voluntary and may be 
discontinued at any time. In addition, the research does not include procedures 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 11 of 27 
 
for which written consent is normally required outside of the research context 
(e.g., quality improvement efforts). 
• 
Patients – The research involves minimal risk to patients. While all patients 
invited to participate in an interview are, by definition, at increased risk of CVD, 
the research does not involve more than minimal risk. Patients will be informed 
that participation in the interview is voluntary.  At any time, patients may 
choose to not answer specific questions or discontinue the interview, and all 
such requests will be immediately respected.  
 
Justification for Waiver of HIPAA Authorization: 
Patients – Interviews will include discussion of the patient’s health information. 
While interviewers will not intentionally solicit PHI, it is possible that a patient 
might share this information during the semi-structured interview. While we 
could not practicably carry out this portion of the research (i.e., obtaining 
patient perceptions of the CV Wizard tool) without the possibility of collecting 
PHI, the PHI that might be collected is not the focus of our data collection 
efforts. Given that we will not be analyzing PHI, we will request that any 
identifiers be removed during the transcription process. The transcriptionist will 
be instructed to delete any identifiable patient information in the transcript and 
replace with the type of information removed (e.g., [*Name*]) and to delete the 
recording once the transcription is complete. Only transcribed interviews will be 
shared with the research teams at OCHIN and/or HealthPartners.  
 
6. Study Procedures 
 
a. Study Aim 1 
 
Clinic Identification, Recruitment & Assignment 
Clinics will be included in the recruitment pool for this study if they (1) provide primary care to 
adults with high CVD risk annually, and (2) have been using OCHIN’s EHR for at least 18 months. 
Once all clinics have been recruited as outlined in Section 5.e., they will be assigned to either 
arm 1 or arm 2. Clinics were grouped based on the baseline number of encounters each service 
area had, percent of patients with hypertension, and percent of patients who use tobacco. Five 
randomly generated groupings were created and the grouping that created the most even 
distribution across both Arms was then selected to assign service areas and the clinics within 
each service area to an Arm. 
 
Programming CV Wizard to Accept OCHIN Data  
 
Working with HealthPartners’ programmers who previously disseminated CV Wizard to other 
large care systems, OCHIN’s programmers will (1) build tables for data extraction (e.g., 
medication, laboratory, diagnostic/problem list codes, vitals, smoking); (2) develop the 
interconnect routines needed to transmit EHR data to the CV Wizard web service, save response 
information in EHR flow sheets, and display the CDS results in the EHR; (3) create the alert to 
prompt CDS use in targeted patients; (4) develop “smart tools” (e.g., dot phrases) to facilitate 
documenting CV Wizard use and results in EHR encounter notes; (5) program the dynamic order 
sets that are tailored to facilitate clinical actions recommended for each patient; (6) create CDS 
use rate feedback reports; and (7) develop a process to manage provider feedback on clinical 
aspects of CV Wizard. 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 12 of 27 
 
 
Validity Testing 
Since 2007, CV Wizard’s algorithms have been continuously modified, and re-validated after 
each modification. OCHIN programmers will validate that CV Wizard works correctly with data 
sent from OCHIN’s EHR as follows: 
(1) OCHIN staff who routinely validate all changes made to the EHR will perform standard 
validation of data extracts for accuracy and completeness. This will include creating ‘test 
patients’ with variation in CVD risk factors and location of relevant data in the EHR, to 
assess how well CV Wizard performs with these patients. The validation team will also 
try to ‘break’ the CDS by testing how it performs with incomplete data. This will occur in 
a ‘copy’ of the Epic production environment used for testing EHR tools before they go 
into production.  
(2) We will conduct preliminary pilot work with 2-3 CHC clinician advisors to assess how CV 
Wizard performs in real (not test) patient data. We will ask them to identify 20-30 of 
their high CVD risk patients, run those patients’ data through CV Wizard’s algorithms, 
and review CV Wizard’s results for accuracy and clinical plausibility. 
(3) Next, CV Wizard will be activated in pilot clinics served by clinician advisors, to further 
identify potential errors; these sites will be excluded from subsequent study 
randomization.  
(4) Further quality checks will occur throughout the study, via CV Wizard’s feedback 
mechanism and built-in website monitoring routes designed to identify aberrant web 
service input or output. Any changes needed to the data sent to the CV Wizard website, 
as identified through these processes, will be reviewed by OCHIN / HealthPartners.  
 
Patient Engagement 
CV Wizard’s patient-facing component is designed to support patient engagement in CVD risk 
management. To improve this aspect of the tool, we will engage OCHIN’s Patient Engagement 
Panel (PEP). This panel is comprised of >20 patients from OCHIN member CHCs who meet 
regularly to contribute input on research conducted at OCHIN. In months 1-6, we will meet with 
the PEP monthly to engage them in a user-centered process on whether and how to redesign 
the CV Wizard ‘Patient View’. 
 
Data Collection 
All outcome variables collected for analyses align with RE-AIM (Reach, Effectiveness, Adoption, 
Implementation, Maintenance), a widely used framework for evaluating implementation 
success. Below are details about the specific outcomes and measures for each. Please refer to 
Table 6 further below for additional information. 
 
Primary Outcome Measures 
• 
Study Aim 1: Compare patients' change in reversible risk in the Arm 1 vs. Arm 2 CHCs (in 
months 18-35) 
• 
Study Aim 2: Assess whether the revised implementation support materials expedite 
CDS adoption 
• 
Study Aim 3: Use mixed methods to identify multi-level barriers / facilitators to 
adoption of the CDS tool 
 
Secondary Outcome Measures 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 13 of 27 
 
• 
Reach: Encounters affected – % clinic encounters where CV Wizard suggests running the 
full risk assessment tool, i.e., identified a target patient. 
• 
Effectiveness (impact): Patient outcomes – 10-year pooled ASCVD risk score – change 
in ASCVD risk score from baseline to last visit of study period (American College of 
Cardiology-ACC/ American Heart Association - AHA) 10-year pooled ASCVD risk score, 
40-75 year olds 
• 
Effectiveness (impact): Patient outcomes - last BP ≤140/90 – Control of individual CVD 
risk factors: last BP ≤140/90 
• 
Effectiveness (impact): Patient outcomes - last A1c≤8 – Control of individual CVD risk 
factors: last A1c≤8 
• 
Effectiveness (impact): Patient outcomes – last LDL<100 – Control of individual CVD risk 
factors: last LDL<100 
• 
Effectiveness (impact): Patient outcomes - not current smoker – Control of individual 
CVD risk factors: not current smoker 
• 
Effectiveness (impact): Patient outcomes - last BMI ≤25 – Control of individual CVD risk 
factors: last BMI≤25 
• 
Adoption: CDS uptake – % of encounters where care team member opts to run the CV 
Wizard risk assessment (explored as both view and print rate and just print rate) 
• 
Implementation: User perceptions – Perceived ease of use, usefulness, acceptability of 
CV Wizard; intent to use it. 
• 
Maintenance over time – All measures over 2.5 years of follow-up, Arm 1; 1.5 years, 
Arm 2. 
 
Table 6. Impact and uptake of the CV Wizard system: Quantitative Measures 
Outcomes, per RE-AIM 
Measurement 
Reach:  
Encounters affected 
% clinic encounters where CV Wizard suggests running the full risk assessment tool, 
i.e., identified a target patient 
Effectiveness (impact):  
Patient outcomes 
 
(i) change in ASCVD risk score (reversible risk; ACC/AHA 10-year pooled ASCVD risk 
score, 40-75 year olds; Framingham 30-year CVD risk score, 20-39 year olds).  
(ii) Control of individual CVD risk factors: last BP ≤140/90; last A1c≤8; last LDL<100; 
not current smoker; last BMI≤25);  
Adoption: CDS uptake 
% of encounters where care team member opts to run the CV Wizard risk 
assessment 
Implementation: User 
perceptions  
Perceived ease of use, usefulness, acceptability of CV Wizard; intent to use it; see 
3.3.f.ii  
Maintenance over time  
All measures over 2.5 years of follow-up, Arm 1; 1.5 years, Arm 2 
Potential covariates 
Measurement 
Patient demographic 
characteristics 
Age; gender; race / ethnicity; primary language; poverty level; insurance status at 
visit or insurance type through study period; # visits to that site / provider in last 
year; whether visit was with patient’s primary care provider; provider type. Others 
TBD based on input from clinician advisors. 
Other patient comorbid 
conditions 
Renal function: diagnosed with end stage renal disease or chronic kidney disease; 
Charlson Comorbidity Score (Modified): Indicator of serious comorbid conditions 
that may shorten life expectancy, modified to exclude CV components; depression 
296.xx, 311.xx diagnosis. Other relevant comorbidities TBD. 
Visit type 
New patient or established patient 
Provider type 
Provider type (PCP or other); # in patient panel 
Other clinic factors 
See Table 5; data from baseline survey 
CVD care / process 
Measurement 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 14 of 27 
 
Recommended care 
that is ordered. 
Whether care suggestions were acted on within 7 days of the encounter; e.g., if CV 
Wizard suggests starting a statin, we will ask whether a prescription was issued.  
 
b. Study Aim 2 & 3 
 
Develop Training Materials  
Research team members will work with OCHIN’s training and QI teams, with CHC clinician input, 
to develop an initial set of resources for implementing CV Wizard in CHC workflows. These 
materials will be augmented over time, as described below, but at first will include information 
on how CV Wizard works, how it might be integrated into CHC workflows, and how care team 
members can use CV Wizard to present/discuss CVD risks with patients in a manner that 
minimizes patient discomfort, reduces potential adverse effects, and maximizes patient 
engagement. 
 
Implementation Support 
CV Wizard’s impact will be affected by the CHCs’ ability to implement it and achieve ongoing use 
at targeted visits. We will provide iterative, adaptive, pragmatic approach to providing the study 
CHCs with implementation support, as follows: 
(1) When Arm 1 CHCs implement CV Wizard, they will receive the basic implementation 
support that OCHIN’s QI team provides with any new EHR functions (e.g., written 
materials, a live monthly webinar describing CV Wizard and suggested workflows for its 
use). We will also provide monthly CV Wizard use-rate reports to the Arm 1 CHC 
providers and clinic managers. CDS use rates are based on proportion of targeted visits 
where the CDS is viewed and / or printed.  
(2) For the next 18 months, we will review monthly use rates of CV Wizard in the Arm 1 
CHCs. The OCHIN practice facilitator will contact operational leaders at the CHCs with 
low adoption rates to ask about barriers to uptake, and what might help address these 
barriers; this aligns with the QI support that OCHIN regularly provides its member CHCs. 
We will work iteratively with OCHIN’s Implementation and Training teams to develop 
and provide additional support as needed to address identified barriers to adoption and 
sustained use at the desired level (80% of targeted visits). The specifics of this ongoing 
support will be determined based on CHC need. However, based on our team’s 
expertise/experience and the implementation science literature, we anticipate that 
some CHCs may need additional training on integrating CDS tools with clinic workflows; 
harnessing use rate data to guide subsequent efforts; the evidence-based guidelines 
underlying the CDS; and how to use CV Wizard to engage patients. Such trainings will be 
provided to clinic leaders, providers, managers, QI coordinators and administrators at 
the study CHCs via written materials and monthly webinars. If indicated, we will also 
provide academic detailing relating to the CDS, and customized implementation plans. 
These resources will become part of our revised implementation materials. We will also 
conduct monthly interactive webinars to answer questions, identify additional barriers 
to uptake/needed training, and enable peer-to-peer learning in the Arm 1 CHCs. If there 
are CHCs for whom these approaches do not enhance uptake, we will confer with CHC 
clinicians, OCHIN’s QI team, and our team’s implementation scientists to identify and 
implement alternative strategies. 
(3) Arm 2 clinics will receive the revised implementation materials when CV Wizard is 
activated in their EHR. We will conduct a similar process to attempt to further improve 
these materials. We will collect quantitative data on which staff members from each 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 15 of 27 
 
study CHC take part in each training activity, and qualitative data on perceptions of the 
training strategies. 
 
Data Collection 
The mixed methods process evaluation will explore “what works, for whom and in what 
circumstances” related to CDS uptake, guided by our logic model and the Technology 
Acceptance Model (TAM). Our 3-tiered approach to data collection follows recent guidance on 
process evaluation of complex interventions which advocates complementing collecting key 
variables from all sites with in-depth data from purposively selected samples. This balances 
collecting data detailed enough to enable transferability (degree to which results can be applied 
to other contexts) and pragmatism (what is possible).  
 
(1) All Clinics –We will collect the following information as it pertains to all clinics 
participating in the study: (1) baseline clinic data (e.g., location, patient characteristics, 
clinic characteristics, clinic size, etc.); (2) all ‘trouble-tickets’ relevant to CV Wizard 
submitted to OCHIN via its member help request system; (3) all user feedback submitted 
via CV Wizard’s feedback mechanism; (4) all exchanges from the implementation 
support webinars; and (5) feedback from the practice facilitator and others involved 
with implementation support.  
 
We will also collect attendee evaluations of all webinars. This pragmatic, adaptive 
approach will let us explore how to help diverse CHCs implement CDS, fine-tune 
materials and strategies for doing so, identify key barriers to adoption of CDS, and 
inform developing a guide for CHCs implementing CV Wizard. 
 
SUPPLEMENT ADDENDUM 
Process outcomes 
Description 
Encounter patterns 
(i) Total clinical encounters; % in-person, by video, or by phone 
(ii) % of patients whose encounters were all in-person, all VC, or a mix of 
both 
(iii) % encounters where CV Wizard alerts that there are CDS suggestions, 
i.e., identified a target patient 
Missed / delayed 
encounters 
(i) % of scheduled appointments that are missed / cancelled, by encounter 
type  
(ii) Appointment wait times (time from scheduling call to appointment), by 
encounter type 
Tool use 
% of encounters where care team member ‘used’ (viewed) suggested CV 
Wizard CDS, by encounter type (among encounters occurring after CV 
Wizard was activated at the clinic)  
Primary outcomes 
Description 
CVD risk data up to 
date  
(i) % patients with up-to-date BP documented in EHR by 6 months post-
index visit, and (ii) % patients with up-to-date A1c data documented in 
EHR by 6 months post-index visit, by encounter type  
   Note: These outcomes will be adjusted for # of encounters in follow-up 
period. Analyses of A1c updates will account for whether prior A1c 
measure indicated lack of control and need for more frequent monitoring.  
   Note: We will consider using coding of a hypertension diagnosis at index 
and follow-up encounters as a proxy for updated BP.  
Effectiveness (all 
eligible patients) 
Needed care provided within 7 days of encounter, by encounter type; e.g., 
If CV Wizard suggested (or would have suggested, if used) starting a statin, 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 16 of 27 
 
was prescription issued: YN; If an A1c test, was the test ordered: YN; If a 
referral to a nutritionist, was the referral made: YN; If medications to help 
quit smoking, was prescription issued; etc. 
Effectiveness (subset 
of patients with an 
index encounter and 
≥1 encounters ≥6 
months post-index) 
Change in CVD risk over time, measured as: 
(i) Decrease in total reversible CVD risk score (ACC/AHA 10-year pooled 
ASCVD risk score64-67) 
(ii) Change from BP at index visit to increased, decreased, or no change, by 
end of follow-up period. 
(iii) Among patients with uncontrolled A1c (>8%) at index encounter, % in 
control by end of follow-up period; among patients with controlled A1c 
(≤8%) at index, % who had a ‘relapse’ by end of follow-up. 
Potential covariates 
Description 
Patient 
characteristics 
Age; gender; race / ethnicity; poverty; insurance status at index; # 
encounters to site / provider, prior year 
Comorbid conditions 
Severe mental illness or diabetes diagnosis at index visit 
Index characteristics 
New patient to the clinic: YN 
Post-index 
encounters 
# in-person / VC encounters between index visit and end of follow-up 
period; was the encounter with the PCP: YN; visit’s primary purpose; 
whether visit was for an urgent need: YN; patient new vs. established 
Provider factors 
Degree (MD, RN, PA, etc.); Prescribing privileges: YN; # in patient panel 
Independent vars. 
Description 
Encounter type 
In-person vs. VC telephone vs. VC video 
CV Wizard use 
CV Wizard ‘used’ (viewed / printed) when it alerted that CDS suggestions 
applied to a given patient: YN 
 
(2) Selected Clinics 
• 
We will conduct semi-structured phone interviews with care team members 
involved with implementing/using CV Wizard, about 10 months post-
implementation. Follow-up interviews will be conducted as needed, to explore 
perceptions, acceptance, and use of CV Wizard and the implementation support 
provided. Sites will be purposively sampled for diversity in CV Wizard use rates. 
• 
We will administer a provider survey that asks about use of the tool and solicits 
feedback. The survey will be programmed in REDCap and sent to clinic contacts via 
email. The clinic contact will be asked to email the providers at their clinics directly 
to request survey completion. Providers are informed (via the consent language that 
accompanies the survey) that (1) all responses/results are anonymous, and (2) we 
will aggregate non-identifiable response data to provide feedback to their clinic’s 
study champion. NOTE: This survey may be administered to some clinics more than 
once over the course of the study period.    
 
(3) Case Study/Supplement Clinics 
• 
For case study sites, we will perform an in-depth ethnographic case study, which 
may include observation/personal interaction to explore the dynamics underlying 
implementation outcomes. We will follow the Arm 1 case study clinics for 2.5 years, 
the Arm 2 case study clinics for 1.5 years. Methods may include naturalistic 
observation (of workflows, patient encounters, clinic meetings), key informant and 
in-depth interviews, and collecting relevant artifacts (process maps, 
communications, etc.).  
• 
For supplement sites, we will perform interviews with CHC staff and patients. 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 17 of 27 
 
Interviews will occur ≥3 months after CV Wizard is activated at a given clinic, and we 
will follow these clinics for one year.  
 
7.  Data Analysis 
 
a. Analysis Plan 
 
Overview 
In this service area pragmatic clinical trial, we will implement CV Wizard via a staggered 
process. We will compare patients’ CVD outcomes in the Arm 1 vs. Arm 2 CHCs, in 
months 18-35 (Aim 1); assess whether the revised implementation support materials 
expedite CDS adoption (Aim 2); and use mixed methods to identify multi-level 
barriers/facilitators to adoption of the CDS tool (Aim 3). 
 
Quantitative Data 
To be included in Aim 1 analyses a patient must have at least one visit in the 6 months after the 
Arm 1 implementation date at an Arm 1 or Arm 2 CHC and at least one additional visit at a study 
clinic in the 12-month post-index visit period. The first visit is referred to as the index visit. We 
will describe those who had inadequate follow-up data (did not have at least one additional visit 
in the post-index period). For each study subject, CVD risk score / risk factor management status 
will be calculated at index visit and each encounter during follow-up. While the tool can be run 
for patients of any age, we will restrict our analyses to patients 40-75 years of age.  
 
We will test for between-group differences in baseline characteristics such as age, sex, race and 
ethnicity, insurance type, visit count, and health status using chi-square, t-test, and 
nonparametric Wilcoxon rank-sum tests as appropriate. We will also test for association 
between baseline characteristics and our outcomes of interest. We will examine the number of 
patients seen in each service area and clinic and each Arm and compare the characteristics of 
Arm 1 patients for whom the tool was used to those in Arm 1 for whom the tool was not used 
and to Arm 2 patients. Because the two arms were selected based on clinic size, percent of 
patients with a diagnosis for hypertension and the percent of patients that use tobacco, we 
anticipate there will be differences between the two groups resulting in the need to adjust for 
these differences. We will also examine characteristics at the service area and clinic-level in Arm 
1 for whom the tool was used compared to Arm 1 service areas / clinics for whom the tool was 
not used and Arm 2 service areas / clinics.  
 
Unadjusted trends in the monthly use and view rates for each of the study CHCs will be explored 
with use rates defined as: the number of encounters where CV Wizard was printed / number of 
targeted encounters with high-CVD risk patients per month and view rates defined as the 
number of encounters where CV Wizard was viewed / number of targeted encounters with high-
CV risk patients per month.  
 
We will also examine overall rates of each of the impact outcomes (BMI in normal range, 
controlled blood pressure, controlled LDL, controlled HbA1c, and non-smoking status) pre/post 
tool use to assess any changes in patterns for patients in Arm 1 for whom the tool was used 
compared to those in Arm 1 for whom the tool was not used as well as to Arm 2 patients. The 
date the tool was first used for each patient will be assigned as the start of the post-period. 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 18 of 27 
 
 
Extent of adoption of a CDS tool like CV Wizard directly impacts its population-level impact. To 
differentiate between the tool’s population level impact versus impact when used, we will 
conduct intent to treat (ITT) analyses to compare all targeted patients at intervention versus 
control clinics. We will also conduct effect of treatment on the treated (ETOT; also called per 
protocol) analyses limited to intervention CHC patients for whom the CV Wizard tool was used 
(results viewed / printed out) at the index visit, compared to intervention clinic patients for 
whom it was not used, and separately, to control clinic patients. The ITT analyses test the 
hypotheses that targeted patients in intervention clinics will have significantly lower CVD risk 
scores and higher rates of CVD risk factor control, at visits in the year post-index visit period, 
compared to similar patients in control clinics. The ETOT analyses test the hypotheses that 
targeted patients in intervention clinics for whom the CV Wizard CDS tool results are viewed and 
/ or printed will have significantly lower CVD risk scores and higher rates of CVD risk factor 
control, at visits in the year post-index visit period, compared to similar patients in control 
clinics, and compared to patients in intervention clinics for whom the tool alerted but was 
ignored at the index visit. 
 
Analyses will be stratified by first-time and repeat patients to see if there is a difference in tool 
use as well as examine whether the time between visits effects tool use among repeat patients. 
 
Intent to Treat Analysis 
Differences in change in outcomes will assessed using multi-level mixed models. Intra-class 
correlation coefficients will be obtained at the patient, clinic, and CHC organization levels for 
each outcome showed that much organization-level variance could be attributed to the clinic 
level. Thus, clustering was accounted for by random effects for clinic and patient, and a fixed 
effect for organization. Analyses will adjust for fixed effects: distribution of eligible patients by 
age, race / ethnicity, gender, rural / urban status, and federal poverty level (FPL) at index visit; 
number of ambulatory visits during the follow-up period; and time from index to last visit. These 
variables were selected a priori and will be confirmed by descriptive analyses, to determine 
whether there are significant differences between baseline intervention and control 
organization and patient characteristics. After checking distributions, negative binomial mixed-
models or poisson mixed-model will be used for reversible risk analyses, and linear-mixed 
models for all other outcomes. 
 
Patients diagnosed with diabetes during the follow-up period will be excluded from reversible 
risk analyses because diabetes might be diagnosed more often with CDSS use. Presence of 
diabetes raises reversible risk estimations even if CVD risk factor control was addressed; this 
exclusion will remove the possibility of confounding based on likelihood on new diabetes 
diagnosis. 
 
Effect of Treatment on the Treated Analysis 
Per-protocol analyses require adjusting for loss to follow-up and off-protocol therapies or 
treatments, but if group differences vary too greatly, model misspecification can yield biased 
estimates. Propensity score methods are an alternative. We will match intervention organization 
patients in three categories of tool use (never used during follow-up, used once, used more than 
once) to control organization patients. Counts of tool use will include use at index visit or 
subsequent study period visits, excluding the last visit. Matching variables will include age, FPL, 
outcome of interest at baseline, race / ethnicity, sex, count of ambulatory care visits post-index 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 19 of 27 
 
visit, time from index visit to last study period visit, and clinic rural / urban status. Propensity 
scores will be estimated using nearest-neighbor matching with replacement.  
 
As analyses in other settings indicated CV Wizard’s potential for greater impact among 
patients with higher baseline risk, analyses of change in reversible risk will be stratified 
by baseline risk <10% or ≥10%. The assumption being that patients with lower baseline 
risk are less likely to improve. Additionally, we will run analyses by quartile of baseline 
reversible risk among patients for whom the tool was ever used during follow-up, 
matched to controls as described above. Analyses of change in reversible risk will include 
all patients meeting study inclusion / exclusion criteria; for other outcomes, analyses 
were restricted to patients with uncontrolled baseline risk. Linear mixed models will be 
used for all outcomes and checked for normality of residuals. 
 
 
Secondary Analyses 
Secondary analyses will examine use of the CV Wizard CDSS (main outcome). We will assess 
which patient-, encounter-, provider-, and clinic-level characteristics are associated with use of 
the CDS at eligible encounters. Additional analyses will explore the impact of CDS system use on 
prescribing pattens targeting CVD risk as well as the influence of system prioritization 
recommendations on risk factor management. We will also analyze results from a provider 
survey conducted to assess reasons for CV Wizard use and non-use.  
 
Quantitative data will be derived from the CDS data repository and extracted EHR data.  
Provider survey data was collected from 44 study clinic clinicians conducted midway through the 
study period. 
 
For quantitative analyses, descriptive statistics will show patient and encounter characteristics 
included in analyses. After exploring the distribution of the outcome variables to identify the 
most appropriate model, multilevel models with random effects for provider and clinic will be 
used to assess (1) multi-level characteristics associated with CV Wizard use and (2) the 
association between system use and system prioritization on CVD risk factor management. 
 
Supplement Addendum 
 
(1) Descriptive Analyses (Aim 2a) 
• 
We will conduct a descriptive analysis on the following process outcomes: (1) Encounter 
patterns; (2) Rates of missed or cancelled appointments at in-person vs. VC (telephone, 
video) encounters; (3) Appointment wait times by encounter type, and (4) CV Wizard 
use (in encounters where there are CV Wizard CDS suggestions by encounter type.  
• 
We will use standard statistical tests (e.g., chi-square, t-tests) to describe measures in 
the periods pre- vs. post-COVID’s onset. 
(2) Regression Analyses (Aim 2b) 
• 
Encounter is the unit of analysis.  
• 
We will assess associations between encounter type and: (1) Action taken on needed 
care plan components; (2) Up-to-date documentation of BP and A1c by 6 months post-
index visit (accounting for whether BP data were taken in person or reported by the 
patient); and (3) Change in CVD risk factors by end of follow-up period. To do so, we will 
conduct adjusted regression analyses using longitudinal electronic health record data.  

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 20 of 27 
 
• 
To assess action taken on needed care, data for each factor will be extracted at each 
visit in the study period. We will use adjusted logistic regression models, with a 
dichotomous outcome of action taken or not taken.  
• 
To assess impact on individual CVD risk factors, data for each factor will be extracted at 
index visit and all visits in the follow-up period (coded as controlled vs. not).  
• 
Generalized linear mixed models (GLMM) with time nested within patients will be used 
to determine if risk factor control differs across encounter types, to estimate the effect 
of encounter type on change over time in CVD risk score and BP and A1c control. All 
encounters for each eligible patient will be included. CVD risk is influenced by factors 
amenable to clinical intervention (e.g., BP) and others that are irreversible (age); absent 
intervention, overall risk should increase with age. These analyses will be adjusted for or 
stratified by potential confounders at the patient, provider, and encounter level. This 
will include accounting for whether post-index visit encounters were in-person or VC, 
and the reason for the encounter, as in-person encounters in the post-COVID period 
may be for different reasons than VC encounters.  
• 
To assess whether CV Wizard moderated the relationship between encounter type and 
CVD outcomes, we will also conduct these analyses using only encounters where CV 
Wizard alerted the user that there are CDS suggestions. We will compare outcomes 
between encounters where CV Wizard was used (viewed or printed out) vs. not used, by 
including its use as a variable in the models described above.  
Data on race / ethnicity and gender will be obtained for all study patients; analyses will 
adjust or stratify for them as indicated.  
 
Statistical Power 
To conservatively calculate power for our main outcome (CVD risk score) we estimated 60 clinics 
with >35 high-CVD risk patients, randomized to two arms; then, using means and standard 
deviations for this measure from previous studies, we varied effect size associated with the 
group x time interaction and intraclass correlation. We examined a 1.5%, 2%, 3% greater 
absolute reduction in risk score over time in intervention vs. control CHCs. Aggressive 
management can reduce 10-year CVD risk by an absolute 4-5% in high-risk patients in a short 
time. We do not expect so large an effect and have power to detect smaller effects. Supplement 
Addendum (Aim 2b). This is a 2x2 repeated measures design with two groups of subjects 
measured at two time points. The primary goal is to compare change across time in group 1 (in-
person index encounters) to change across time in group 2 (VC index encounters). A 
conservative estimate (based on preliminary results from the parent study) of 6000 patients in 
group 1 and 6000 in group 2 achieve 100% power to detect a difference in mean changes in CVD 
risk score of 5.0, with a standard deviation of 9.1 at the first time point, a standard deviation of 
9.1 at the second time point, and a correlation between measurement pairs of 0.200. The 
significance level (alpha) is 0.050 using a two-sided, two-sample t-test. 
 
Qualitative Data 
Using a realist approach our process evaluation will explore “what works, for whom and 
in what circumstances” related to uptake/impact of the CDS system. It will be guided by 
our logic model and the TAM’s conceptualization of the relationship between user 
perception and CDS acceptance/use. We will identify mechanisms of change 
(barriers/facilitators; impact of implementation strategy) at the clinician and health care 
system levels and assess how contextual factors impact outcomes. Supplement 
Addendum: The parent study involves assessing how CV Wizard is used at in-person 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 21 of 27 
 
visits. We will account for the shift to VC by adding qualitative data on its use in VC. As 
in the parent study, process evaluation will be informed by the Technology Acceptance 
Model to understand (1) how CVD care takes place in VC and (2) the role of CDS in VC. 
 
We will triangulate process data from all sources for a deep understanding of CV 
Wizard’s acceptance/use in the CHCs. We will look at areas of consistency/inconsistency 
in these data (where different data sources show the same thing, or do not). A 
grounded theory approach coupled with an immersion-crystallization process will be 
used to identify themes and patterns in the qualitative data. We will emphasize factors 
influencing use of CV Wizard and its impact on care decisions. Data collection and 
analyses will be parallel and iterative, letting us identify salient constructs and 
knowledge gaps while implementation is ongoing. The study team will meet regularly to 
discuss and integrate qualitative and quantitative process evaluation and outcome data 
to help identify factors that affect success. The integration of mixed methods data will 
provide a more complete, nuanced understanding of the impact and use of the CV 
Wizard system than either method alone, and permit examination of the reliability and 
validity of the data sources. Supplement Addendum: Supplement data will be added to 
an existing NVIVO database to allow integration of findings. It will also be flagged to 
facilitate sub-analyses specific to this supplement’s goals. To incorporate data from this 
study while allowing new themes to be considered, we will use an immersion-
crystallization process which entails multiple iterations of data immersion, reflection, 
and code development and application, to identify new data themes or patterns related 
to the use of CV Wizard, and its impact on CVD risk management, in the VC context. 
 
b. Sharing of Results with Subjects 
 
If the CV Wizard decision support tool is used during a patient encounter, the provider has the 
option to print the provider-facing report. If the provider wishes to discuss the results with the 
patient, he/she may print the patient-facing report and give a copy to the patient or post to 
clinic’s patient portal (if available). To ensure patient confidentiality (i.e., the correct patient 
receives the printout), the patient’s name is printed on the top of the report.  
 
c. Data and Specimen Banking 
 
Not applicable. 
 
8. Privacy, Confidentiality, and Data Security 
 
Quantitative Data Management, Sharing and Protection 
OCHIN uses safeguards similar to those used at CHR to ensure confidentiality when handling data. 
All patient-related study data will be de-identified and unique patient identification codes will be 
used, and all data sources will be linked through a secure relational database at OCHIN. Data sent to 
CV Wizard, and then returned to OCHIN for analysis, will be de-identified and protected as described 
in the Security Measures section below. Only aggregate data will be shared with CHR through a 
secure data transfer web site, as necessary. The CV Wizard CDS system is a web service that uses 
input EHR data to identify eligible patients, compute CV risks, prioritize CV risk reduction, and 
provide treatment recommendations based on a complex set of evidence-based algorithms. A web-

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 22 of 27 
 
based display of the results is then provided to the primary care providers.  
 
Qualitative Data Management, Sharing and Protection  
Qualitative data will be securely stored at both CHR and OCHIN. All data work will be done on 
password-protected workstations, in a secure environment (HIPAA-compliant). At CHR, only study 
team members have access to the saved data. At OCHIN, access to study data is limited to the 
research department.  
 
Data will be catalogued within 1 week of collection and will be entered in QSR NVivo. Data sharing 
agreements/protections will be set as necessary, and electronic data transfers will use a secure 
website or encrypted email. Hard copies of fieldnotes will be stored in a locked office at KPCHR that 
only research staff can access. 
 
Audio recordings of staff/patient interviews will be stored on a secure file server that is accessible 
via password protected computers. Audio recordings will be transcribed by a KP/CHR-approved 
vendor. Audio files and transcripts will be transferred between CHR and the transcriptionist using a 
secure file transfer website. All audio files will be destroyed at the end of the study. 
 
When research staff travel to/from the case study site clinics, they will follow the CHR procedure for 
physically moving data. The data being transferred may include encrypted audio recordings and 
notebooks containing fieldnotes (e.g., observation date/time, provider/staff name, patient 
encounter activity, etc.). Because clinic visits may take place at various locations and times 
throughout the day, there may be some instances (e.g., early morning or late afternoon/evening) 
that necessitate study staff transporting study materials to their home. In these cases, staff will: 
(1) Place all data in a secured pouch with the appropriate labeling outlined in the CHR procedure 
for physically moving data. 
(2) Transport data in the trunk of their vehicle and commute directly home. 
(3) Bring the data into their home and place in a secure location. 
(4) Transport data directly to KPNW facility. 
 
Storage of Recruitment Data 
As outlined in Section 5.e (Case Study & Supplement Site Recruitment), we may identify patients for 
qualitative interviews using a BPA alert in the EHR (where providers flag patients), by pulling data on 
smartphrase use in the after-visit summary (AVS) that indicates CV Wizard was used during an 
encounter, or other indicators of tool use (e.g., CVW output was viewed or printed). In these cases, 
patient data will need to be accessed and stored for recruitment and tracking purposes. We will 
store the following data in OCHIN’s instance of REDCap: patient name, encounter date, referring 
provider, clinic name, mailing address, phone number, email address, call attempts, interview 
completion, gift card distribution, and notes entered by the study team related to recruitment 
efforts (e.g., outreach type, date of outreach, etc.). OCHIN and KP CHR staff will have access to 
participant data in REDCap to make initial contact with participants. KP CHR staff participating in 
data collection will be given access to only the minimum necessary information. Access will be 
controlled in the Data Viewing Rights section of RedCap. This data will not be transferred or stored 
in any other location outside of REDCap. 
 
Additional Information about Sensitive Data 
We plan to pull data related to mental health diagnoses (depression, anxiety, and severe mental 
illness) to determine whether it explains variance in CV Wizard tool use. This data will be pulled 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 23 of 27 
 
from the EHR (i.e., this information will not be collected directly from patients that are participating 
in the qualitative interviews). We will follow the same safeguards for this sensitive data as we do for 
all data collected in this study (see sections on data management, sharing, and protection; 
confidentiality; and security). This data will be collected (as applicable) for all clinics participating in 
the study. Given that individual patients are not enrolled for this aim of the study, there is no 
consent form or PRA to which this information should be added. NOTE: Each OCHIN member clinic 
has business use agreements with OCHIN to handle and manage PHI from their clinical data, and 
each CHC agrees that their electronic health record data may be used in research as part of their 
membership agreement. 
 
Confidentiality Measures  
All CHR investigators and project staff, all HealthPartners data managers, and all OCHIN staff who 
work on research projects and handle human subjects data/have human subjects contact, sign 
confidentiality pledges and receive IRB and HIPAA training/certification. 
 
Security Measures  
Multiple measures are in place to ensure the security of PHI. Data transfer to and from the EHR (at 
OCHIN), the web service (at Health Partners), and the web display (accessed at OCHIN member 
CHCs) uses a Simple Object Access Protocol (SOAP) with Secure Sockets Layer (SSL) encryption over 
a Hypertext Transfer Protocol Secure (HTTPS) computer network. There is a double firewall in the 
Web service so that once the data flow through the initial Web service firewall, the data cross 
another firewall into a new secure pathway that once again employs SOAP, SSL, and HTTPS to 
process the data and provide recommendations. This includes sending the data through a batch 
server for more efficient processing, but all within the double-firewall Web service. Limited clinical 
data for all adults are initially extracted from the EHR to determine CDS eligibility.  
 
We estimate that 15-25% of study clinic patients may meet the eligibility criteria at which point 
more clinical information is extracted and full processing through the CV Wizard clinical algorithms 
occurs and is displayed back to the providers. If/when the CV Wizard tool is used, to ensure patient 
confidentiality, the name of the patient is printed on the CDS sheet given to the patient and their 
care provider at the clinic visit. This is needed to avoid mixing up printed pages and giving protected 
health information (PHI) to the wrong patient. It is also desirable to have each patient’s name on 
this sheet of paper to assure the patient that the information on the paper is related to their own 
health state and not someone else’s. The technical aspects of this system are also relevant; the 
printout is controlled from the CDS Web site, so the Web site must have the patient’s name and 
other PHI.  
 
With the analytic databases, measures will be taken to protect PCPs and patients from the risk of 
breach of confidentiality: A unique study ID code unrelated to the EHR record number or other 
study subject-specific information will be assigned to each patient and provider study subject and 
used to link data from various sources needed for analysis. A crosswalk table linking this code 
number to a provider PCP or patient name or medical record number will be destroyed within 12 
months of completion of the linked databases needed for study analyses. To minimize the risk that a 
PCP will act wrongly on the basis of information provided through CDS developed for this study, 
communication to providers will have a written explanation that the CDS is a suggestion, not a 
mandate, and that the action should only be taken if judged to be clinically appropriate by the 
treating provider on the basis of the patient’s current clinical status and preferences.  
 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 24 of 27 
 
9. Provisions to Monitor the Data to Ensure the Safety of Subjects 
 
This study involves data from OCHIN, and through OCHIN, patient data from enrolled CHCs. The web 
service for CV Wizard© is hosted by Health Partners. All EHR data from OCHIN clinics are stored at 
OCHIN. Each clinic has business use agreements with OCHIN to handle and manage PHI from their 
clinical data. EHR data will be linked using OCHIN’s unique patient identification codes, and data 
sources will be linked into a secure relational database at OCHIN. When data linkages have been 
completed, the data will be fully de-identified. Data analysis will be conducted at OCHIN. De-
identified, aggregate data will be shared with study team members from CHR and HealthPartners.  
 
Yearly datasets will be created with data from CV Wizard’s web server, imported into SAS, and 
securely maintained at OCHIN. Data will be extracted for all target patients seen in OCHIN study 
CHCs and stored (limited data set) in a Web-based secure repository. Data from subsequent 
encounters is stored and linked to the same patient. If necessary, to share PHI, we will use a secure 
data transfer website with access limited to appropriate members of the research team. IRB and 
HIPAA approval will be obtained for all study steps. CHR and OCHIN have expertise in high quality 
data management and project operations and multisite research collaborations. Data structures are 
typically designed to separate personal identifiers from other critical data, further enhancing 
protections. CHR, HealthPartners, and OCHIN standards meet or exceed requirements for patient 
data safety established in the federal HIPAA guidelines. Data structures will derive from the data 
confidentiality, security, and privacy standards that CHR/OCHIN/HealthPartners have in place to 
meet or exceed all current HIPAA requirements. These standards exceed Level 1 requirements, and 
meet the majority of Level 2 requirements, as specified by DHHS in its Automated Information 
Systems Security Handbook. All analyses will be carried out using SAS® version 9 or later. Quality 
control will begin with real-time, inter-field checks in the data at OCHIN. Additional back-end checks 
(for missing data and logical inconsistencies) will be conducted to ensure the highest standards of 
data reliability. We will examine the distribution and measurement properties of variables before 
making final decisions about analyses. 
 
In addition, we will compose a data safety and monitoring board (DSMB) of three members with 
expertise in clinical trials methodology and the clinical domains addressed in the proposed research. 
Members will include one clinician with expertise in CV disease and treatment, one expert in 
information technology, and one statistical expert. The PI will participate in the DSMB meetings in a 
limited way, as recommended in NIH policy. The DSMB will provide input and guidance on the study 
evaluation and intervention protocols, including quality assurance and safety issues related to the 
protocols and intervention strategy, as well as data-handling activities. The DSMB will provide 
periodic input via email, conference calls, and annual meetings. The DSMB will convene for one 
meeting in project year 2 to review the study protocol and adopt a formal charter. They will then 
convene twice each year during project years 3-4 (the active intervention phase) and one final time 
in project year 5 to review study results. A special focus of interest will be the safety of patients 
exposed to the study intervention. The intervention provides point-of-care CDS related to 
management of elevated CV risk and suggests evidence-based treatment options based on national 
guidelines, and further vetted by clinical leaders at OCHIN and HealthPartners. CDS 
recommendations provided as part of the intervention are designed to support clinicians’ decision-
making, not to override clinical judgment. Information related to adverse events will be collected by 
OCHIN and/or HealthPartners and sent in writing to KPCHR for reporting to the IRB.  
 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 25 of 27 
 
10. Risks and Benefits 
 
a.  Risks to Subjects 
 
Breach of Privacy/Loss of Confidentiality 
One risk to clinic staff and patients is breach of privacy and subsequent loss of 
confidentiality, but this is extremely unlikely. All CHR, HealthPartners, and OCHIN staff 
are highly trained and appropriately certified in data security and confidentiality, and 
the current protection measures represent the cutting edge of electronic protection. All 
patient data used in analyses will be de-identified by OCHIN staff, who work with 
patient data from their member clinics on a regular basis and are appropriately trained 
in data security. Thus, we feel any risk due to loss of confidentiality is remote.  
 
CV Wizard Recommendations 
Another potential risk is the possibility that the CV Wizard CDS tool may provide advice 
(according to national evidence-based CVD guidelines) that is inappropriate for a given 
individual patient and, if applied without further checking the clinical status of the 
patient, could lead to erroneous therapy or adverse events. However, the 
recommendations generated by CV Wizard are evidence-based and operationalize 
current national and regional standards of care and, therefore, the risk of untoward 
consequences of such clinical actions is considered minimal. Moreover, this potential 
risk is routinely present in every clinical encounter in the healthcare system. All 
treatment recommendations will be evidence-based, but acting on them will ultimately 
be up to the provider’s judgement; thus, we do not anticipate any clinical harm to 
patients based on patients/their providers seeing the CV Wizard recommendations. We 
have described below the methods used to minimize this risk. In addition, our 
qualitative data collection will include a focus on potential harms to patients – such as 
anxiety – that may be incurred by their exposure to the CV Wizard CVD risk assessment. 
 
b. Potential Benefits to Subjects 
 
Overall Benefits  
This study is expected to yield several key benefits. It will determine whether and how 
clinical decision support (CDS) tools that address multiple aspects of guideline-based 
CVD prevention and care, have provider- and patient-facing elements (enabling patient 
engagement), and involve some workflow changes, can and will be adopted by, and will 
successfully increase rates of guideline-concordant care in CHCs. The use of 
sophisticated CDS in populations with persistent disparities is a long overdue, critical 
step towards addressing CVD risk/outcome disparities in socioeconomically vulnerable 
(i.e., CHC) populations in the US. This study will determine how persistent guideline-to-
practice gaps impacting high-risk/prevalence patients can be addressed using targeted, 
innovative, multi-level, team-based decision support tools. Thus, key benefits include: 
(1) Collecting, documenting and presenting data on prioritized CVD risk to care teams 
and patients may result in patients receiving care that ultimately improves their health. 
This potential benefit will incur no additional clinic visits or costs to patients. (2) The 
intervention will increase care teams’ knowledge about CVD risks that may influence 
their patients’ health. (3) This intervention may bring improvements in EHR functionality 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 26 of 27 
 
to CHCs that would otherwise not receive them. (4) Because this study develops and 
tests different ways of presenting CVD risk data in clinic EMRs, it will provide needed 
information on how prioritized CVD risk data can be used to improve care and services 
in community health center populations nationwide.  
 
Findings will support further improvements in the US healthcare system to mitigate 
health disparities and will inform future efforts to bring cutting-edge CDS to CHCs. If the 
intervention significantly improves primary care with respect to identification or 
management of elevated CV risk factors in adults, the risk of CVD events and/or 
mortality related to elevated CV risk may be reduced later in life for large numbers of 
patients. If the intervention fails to improve identification or management of CV risk 
factors, that knowledge will also be important because it will direct the attention of 
investigators to other potentially more fruitful lines of investigation. Thus, regardless of 
specific findings, the results of this trial will provide important new knowledge that may 
ultimately contribute to improved care for adults with elevated CV risk. 
 
Patients and Clinic Staff  
Patients at the study CHCs will have no defined personal benefit from this project. However, the 
CVD reduction recommendations may help assist in better patient-provider communication 
around CVD risks and lead to improved treatment outcomes for high-risk patients.  
 
Clinic staff at the study CHCs will have no defined benefits from participating in this 
project. However, the intervention is designed to optimize identification and 
management of adult subjects with elevated CV risk. Some providers exposed to this 
potentially useful CDS may use it to improve their clinical care during the study or after. 
 
11. Costs to Participants 
 
Patients will not incur any costs if CV Wizard is used during their office visit. Clinic staff and 
patients that participate in the qualitative interviews will not incur any costs.  
 
12. Compensation to Participants 
 
Case study site clinics will receive annual impact fees to compensate for staff time spent on 
qualitative data collection efforts. Clinics will receive $1000 for each year they participate as 
a case study site. Supplement study clinics that participate in qualitative activities will 
receive an impact fee of $1610 for their year of participation. 
 
Patients who participate in Aim 2 and 3 qualitative interviews (either by phone or in-person) 
will each receive a $25 gift card. Staff who participate in Aim 2 and 3 qualitative interviews 
(either by phone or in-person) may receive a $25 gift card if their workplace allows them to 
accept research incentives. Each MOU will outline the approved form of compensation, if 
any. 
 
Supplement Addendum: Patients who take part in the supplement qualitative interviews 
will each receive a $30 gift card. Clinic staff that participate in supplement interviews will 
not be individually compensated. Instead, the clinic will be compensated with a greater 

CV WIZARD PROTOCOL – 07/28/2022 
 
Page 27 of 27 
 
clinic impact fee. NOTE: This approach has been taken for these clinics, as many staff are 
unable to accept research incentives. 
 
Gift cards for telephone/remote interviews will either be post mailed or emailed (depending 
on gift card type). Any personal information collected for the purpose of sending a gift card 
will be stored electronically on the study file service at CHR, accessible only to study staff 
members.  
 
13. Resources Available  
 
No special resources or expertise are required to conduct this study. 
 
14. Drugs or Devices  
 
Not applicable.  
 
15. Multi-Site Coordination  
 
CHR will act as the coordinating center for this study. We will ensure that:  
• 
All sites have the most current version of the protocol, consent document, and HIPAA 
authorization. 
• 
All required approvals have been obtained at each participating site (including approval 
by the site’s IRB of record if required). 
• 
All protocol modifications have been approved and communicated to sites (including 
approval by the site’s IRB of record, if required) before the modification is implemented. 
• 
All engaged participating sites will safeguard data as required by local information 
security policies. 
• 
All local site investigators conduct the study appropriately. 
• 
All non-compliance with the study protocol or applicable requirements will be reported 
in accordance with local policy. 
• 
Communication of problems, interim results, and study closure. 
 
16. Community-Based Participatory Research  
 
OCHIN’s leadership strongly supports activating CV Wizard in its member CHCs, which aligns with 
OCHIN’s ongoing efforts to improve care and outcomes in CHCs via EHR-based strategies. In all study 
steps, the study team will engage OCHIN’s operational leadership (including a CHC clinician and 
informaticist). We will also engage OCHIN CHC clinicians via existing communication structures. 
OCHIN has a long history of engaging stakeholders in all system-wide efforts; CHC clinicians serve on 
standing committees that direct all changes made to OCHIN’s EHR. These committees include: 
OCHIN’s Executive Leadership Team, the Practice-based Research Network (PBRN) formed in 2006, 
the Clinical Operations Group, and the Clinical Review Advisory Committee. For this project, we will 
engage OCHIN leaders and clinicians by working directly with these groups to obtain their input and 
direction at key junctures.