An Assessment of Open Access Audio of the Clinical Encounter on Veterans and Their Care
Phase: NA
Conditions: Hypertension, Diabetes
Sponsor: VA Office of Research and Development

 
 
 
 
 
[IRBNET Number – 1573648-1] 
An Assessment of Open Access Audio of the Clinical Encounter on Veterans and 
their Care 
Funding Agency: Health Services Research & Development 
Principal Investigator: Saul J. Weiner, MD 
August 30 2022  
 
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 2 of 33 
 
 
Abstract 
Background: The medical encounter can be overwhelming in term of the amount of information 
discussed, its technical nature, and the anxiety it can generate. Easy access to a secure audio 
recording from any internet enabled device is an available low cost technology that allows patients 
to “revisit the visit” either alone or sharing with caretakers and family. It has been introduced and 
tested outside the VA with evidence that it increases patient recall and understanding and may even 
improve physician performance. Little is known, however, about whether and to what extent 
these effects lead to better outcomes, such as improved treatment plan adherence and chronic 
disease self-management. 
Objectives: The study aims to assess (1) the impact of an open access audio (OAA) 
program on two behaviors (patient activation, treatment plan adherence), and two 
chronic condition measures (glycosylated hemoglobin, blood pressure); (2) the impact of 
open access audio on provider communication and on their attention to patient 
contextual factors (i.e. individual veteran’s needs and circumstances relevant to planning 
effective care); and (3) patient, provider, and leadership perceptions of the extent to 
which the program is safe, not burdensome, and worthwhile at both the start and at two 
years into the program. A secondary analysis will descriptively measure the effect size of 
OAA on ED visits and hospital admissions. 
Design: The setting will be primary care and diabetes clinics, at two facilities for 
generalizability. To achieve aims 1 and 2, we plan a randomized controlled three arm 
design: (1) the encounter is recorded, with provider and patient aware, and uploaded to 
a server the Veteran, provider, and research team can access post visit; (2) the 
encounter is recorded, with both parties aware, and uploaded to a server only the 
research team can access; and (3) the encounter is recorded, with only the patient 
aware, and uploaded to a server only the research team can access. Resource 
utilization and disease measures indicated in aims 1 and 2 will be collected in all arms.  
Analysis: Comparisons of Arms “1” and “2” enable assessing the impact of the 
availability of audio on patients (for aim 1).  Comparisons of Arm “2” with “3” enables 
isolating the effect of providers knowing they are being recorded (for aim 2). Data for aim 
3 will come from survey tools, focus groups and semi-structured leadership interviews to 
elicit perceptions of project safety, burden, and value.       
Impact: This study will assess a new resource for enhancing Veterans capacity to understand 
their care plan and share information from their visit with caregivers. The study design is 
intended to yield information to guide decision makers about the value of bringing “open access 
audio” (OAA) to VHA. 
 
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 3 of 33 
 
List of Abbreviations 
Provide a list of all abbreviations used in the protocol and their associated meanings. 
4C:  
 
Content Coding for Contextualization of Care 
CDW:  
Corporate Data Warehouse 
CIRB:  
Central Institutional Review Board 
COIN:   
Center of Innovation 
Co-I: 
 
Co-Investigator 
ED: 
 
Emergency Department 
HSR&D:  
Health Services Research and Development 
HgB:  
Hemoglobin 
HIPAA:  
Health Insurance Portability & Accountability Act 
OAA:   
Open Access Audio 
PDC:   
Proportion of days covered (refill rate) 
PHI: 
 
Protected Health Information 
PI: 
 
Principal Investigator 
RVA:   
Return Visit Adherence 
 
VHA:  
Veterans Health Administration 
 
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 4 of 33 
 
 
 
Contents 
Protocol Title: ............................................................................................................................. 5 
1.0 
Study Personnel .............................................................................................................. 5 
2.0 
Introduction ..................................................................................................................... 6 
3.0 
Objectives ....................................................................................................................... 8 
4.0 
Resources and Personnel ............................................................................................... 9 
5.0 
Study Procedures ...........................................................................................................11 
5.1 Study Design ...................................................................................................................11 
5.2 Recruitment Methods.......................................................................................................21 
5.3 Informed Consent Procedures ....................................................................................... 231 
5.4 Inclusion/Exclusion Criteria ............................................................................................ 232 
5.5 Study Evaluations .......................................................................................................... 242 
5.6 Data Analysis ..................................................................................................................24 
5.7 Withdrawal of Subjects .................................................................................................. 286 
6.0 
Reporting ........................................................................................................................28 
7.0 
Privacy and Confidentiality ........................................................................................... 297 
8.0 
Communication Plan .................................................................................................... 308 
9.0 
References ................................................................................................................. 319 
 
 
 
 
 
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 5 of 33 
 
Protocol Title:   
1.0 
Study Personnel 
• Provide 
name, 
contact 
information, 
facility/organization, 
and 
affiliations/employee status for the following: 
o Principal Investigators/Study Chairs 
o Co-Investigators: 
o Collaborators: 
Principal Investigator:  
Saul J. Weiner, MD; Jesse Brown VA Medical Center; Staff Physician and Deputy Director, 
Deputy Director, VA Center of Innovation for Complex Chronic Healthcare; Professor of 
Medicine, 
Pediatrics 
and 
Medical 
Education, 
University 
of 
Illinois 
at 
Chicago. 
Saul.weiner@va.gov  
 
Co-Investigators: 
 
Frances Weaver, PhD, Site PI: Edward Hines, Jr. VA Hospital, VA Center of Innovation for 
Complex 
Chronic 
Healthcare; 
HSR&D 
Senior 
Research 
Career 
Scientist. 
Frances.Weaver@va.gov  
 
Alan Schwartz, PhD; University of Illinois at Chicago; Professor, Department of Medical 
Education; alansz@uic.edu 
 
Corinna D Falck-Ytter, MD, Site PI; Louis Stokes Cleveland VA Medical Center; Medical 
Director of Primary Care; Corinna.Falck-Ytter@va.gov 
 
Sherry Ball, PhD; Louis Stokes Cleveland VA Medical Center; Implementation Specialist 
Sherry.Ball@va.gov  
 
Paul Barr, PhD, Site PI; Dartmouth Institute, Geisel School of Medicine at Dartmouth; 
Paul.J.Barr@dartmouth.edu  
 
William Haslett, PhD; Biomedical Data Science Department, Geisel School of Medicine at 
Dartmouth College; will.haslett@gmail.com  
 
• Indicate the number of potential participating sites (both VA and non-VA) and if 
there is any graduated start-up plan for the sites 
 
Two sites are to participate in data collection:  
 
Jesse Brown VA Medical Center 
 
Louis Stokes Cleveland VA Medical Center 

Version#3 8/30/2022 
VA IRB Protocol  
Page 6 of 33 
 
Two sites include staff who will facilitate analysis of the data and have access to it 
 
 
Edward Hines Jr. VA Hospital 
 
     Biomedical Data Science department at the Geisel School of Medicine at 
     Dartmouth College 
2.0 
Introduction 
• Provide scientific background and rationale for study.   
Technically complex information is often shared with patients during a medical 
encounter. During these visits patients may be emotionally distraught, in physical pain, confused 
or with a level of low health literacy. A range of studies indicate patients forget as much as 80% 
of the information immediately following the visit.1-4 Recent innovations to address recall 
problems include after-visit summaries (AVS) and providing direct access to their medical 
record. These strategies are less helpful when patients lack high levels of medical literacy.5-7 
They also don’t capture many nuances of the visit, such as what reassurances the provider may 
have provided, and any questions the patient asked that were, in turn, addressed.  
 
 
Enabling patients to access audio recordings of clinical encounters (“Open Access 
Audio,” or OAA) offers an additional resource with the potential to improve recall and 
understanding of clinical instructions and, as a result, potentially improve health outcomes. 
Audio recordings, in contrast to AVS, provide a verbatim record of all that was discussed. Early 
evidence is that they are valued:  Patients listen to and share them with family and/or 
caretakers. In a scoping review of 33 studies, on average 72% of patients listened to the audio 
and 60% shared it with others, e.g. a family member or caregiver.8  Several studies show that 
audios provided by practices result in higher levels of engagement and recall, and that recall is 
associated with increased adherence.8-10 Patients have also reported that they are less likely to 
second guess decisions made during the visit if they can listen to them later, which may account 
for fewer follow-up calls to practices after visits.11 Finally, many believe their providers 
communicate more effectively and are more attentive when they know they are being audio 
recorded.11,12 We saw evidence of this advantageous “Hawthorne-like” effect in research we 
conducted comparing how attentive providers are to patient life context in care planning, 
comparing when they are being openly assessed by a standardized patient in a lab (audio out in 
the open), to an assessment with exactly the same script portrayed by the same actor as an 
incognito unannounced standardized patient embedded in their practice (audio concealed). In 
the former situation they were significantly more likely to identify and address contextual factors 
essential to care planning (e.g. that a patient needs a less costly medication, or needs 
transportation) than in the latter.13 We also found that such attentiveness predicts improved 
health care outcomes when measured using a blinded scoring system based on audio 
recordings of real patient encounters (e.g. HgB A1c more likely to come down when providers 
identifies cost issues and switches patient to more affordable insulin).14 Hence, open audio 
recording may be beneficial to patients both directly, i.e. as a resource they can access to better 

Version#3 8/30/2022 
VA IRB Protocol  
Page 7 of 33 
 
understand their care, and indirectly, by motivating their provider’s behavior in ways that are 
beneficial to them.  
• Include summary of gaps in current knowledge, relevant data, and how 
the study will add to existing knowledge.   
 
 
Although the evidence on OAA is promising, it’s measurable effects have not been 
substantially examined. Specifically, how might it impact service utilization and even disease 
specific measures?  We propose the following model, in Figure 1 for how it might drive these 
effects, and postulate a set of desired outcomes.   
 
 
The model postulates that audio recording visits for patients both directly improves patient recall 
and understanding (documented outside the VA), and that it motivates providers to do a better 
job communicating with their patients and getting to know their individual needs and 
circumstances (hypothesized) since the patient may listen to and share the audio with 
caregivers.  In turn, patients with better understanding and recall are more likely to be activated, 
which leads to greater treatment adherence (documented). Finally, activated patients who 
adhere to their treatment plan should miss fewer appointments, and have better control of their 
chronic conditions. It is also possible that they become less likely to end up in the ED or 
hospitalized. The evidence for these linkages come from other studies: Patient activation, 
defined as having the knowledge, skills, and confidence for self-management, can, in fact, 
predict increased treatment adherence, and both have been associated with the desired 
reductions in resource utilization and improved self-management of chronic conditions.15-17   
 
Of note: Setting up an audio recording program in the patient care setting requires considerable 
expertise.  The  research team that will conducting this study has over a decade of experience 
carrying out HSR&D funded studies in which encounters are audio recorded by patients, 
recording over 1000 visits a year in the prior three years.12,14,18 

Version#3 8/30/2022 
VA IRB Protocol  
Page 8 of 33 
 
3.0 
Objectives 
 
• Describe the study’s purpose, specific aims, or objectives.  
• State the hypotheses to be tested. 
      
In this study, we will test our model for the purpose of delineating the extent to which OAA can 
bring value to patient care in VHA. We plan to control for demographic factors, namely: age, 
sex, race, health literacy, and internet access. The project will be conducted in primary care and 
one specialty clinic: diabetes. We select diabetes because of its high prevalence in Veterans, 
the presence of a specific marker of disease control (Hgb A1C), and that disease management 
now relies on a growing number of medications and dietary interventions that can be complex 
and overwhelming for patients. OAA, hence, may offer substantial benefits to patients and their 
caregivers and, if it does, the effect should be captured by falling Hgb A1c levels among those 
with baseline poor disease control. As noted, we also plan to study the impact of OAA on 
uncontrolled BP, but that data can be collected entirely in primary care given the very high 
prevalence rate.  
This study has three aims, with the first two designed to test measurable hypotheses. The third 
is qualitative: 
 
Aim 1: Assess the impact of an open access audio program on two behaviors (patient 
activation, treatment plan adherence), and two chronic condition measures (glycosylated 
hemoglobin, blood pressure). Secondary analysis will descriptively measure effect size on ED 
visits and hospital admissions. 
 
Hypotheses:  
1A: OAA increases patient activation. 
1B: OAA increases care plan adherence, including fewer no-shows and more on-time 
refills. 
1C: OAA reduces Hgb A1c and BP 
 
Aim 2: Assess the impact of open access audio on provider communication behavior and on 
their attention to patient contextual factors (i.e. individual Veterans needs and circumstances 
relevant to planning effective care). 
 
Hypotheses:  
2A: Providers obtain higher scores on measures of communication behavior when they 
know they are being recorded.  
2B: Providers obtain higher scores on measures of attention to patient contextual factors 
essential to care planning when they know they are being recorded. 
   
Aim 3: Assess patient, provider, and leadership perceptions of the extent to which the program 
is safe, not burdensome, and worthwhile at both the start and at two years into the program.   
 
 
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 9 of 33 
 
• Indicate the relevance to Veterans and the VA 
The VA has worked to empower Veterans, who disproportionately live with complex 
chronic health problems, with access to information related to their care through the patient 
portal (MyHealtheVet).19,20 Giving them access to audio recordings of their visits is a next 
natural step. In contrast to the medical record which is written by providers for providers, the 
audio captures what the provider (or other provider) most wants their patient to understand. At a 
practical level it is a concrete solution to the ephemeral nature of the interaction, where much of 
importance is discussed, then forgotten and never shared accurately with loved ones and 
caretakers who need to understand the patient’s care. It may also be an incentive to the 
provider to be more attentive and communicative. 
 
The VHA Office of Connected Health has indicated a high level of interest in the study 
(and provided a letter of support), as the findings may provide a foundation of evidence for 
whether and how to include open access audio in their next generation patient portal.  
4.0 
Resources and Personnel 
• Include a list of personnel, their location, role in the study and their VA 
affiliation status  
• Provide a brief description of each individual’s role in the study.  Be sure 
to indicate who will have access to protected health information and who 
will be involved in recruiting subjects; obtaining informed consent; 
administering survey/interview procedures; and performing data analysis. 
• If applicable provide information on any services that will be performed by 
contractors including what is being contracted out and with whom. 
• If applicable provide information on any Memoranda of Understandings 
(MOUs), Data Use Agreements (DUAs), and/or CRADAs that are being 
entered into including with whom and for what reason. 
Data collection will occur at two sites: The Jesse Brown VA Medical Center in 
Chicago, and the Louis Stokes Cleveland VA Medical Center.  Data analysis will be 
conducted by staff at the Center for Innovation for Complex Chronic Health Care 
(CINCCH) based at Edward Hines Jr. VA Hospital.  
In addition, we will be collaborating with colleagues in the Dartmouth Department of 
Biomedical Data Science who have developed and employed HealthPAL (“Personal 
Audio Library”), previously called Open Recording Automated Logging System (ORALS) 
technology, developed specifically for OAA.21 HealthPAL, designed for sharing audios 
with patients, enables patients to selectively share their audio with others, such as family 
or caretakers, by sending them a link to establish a password.  This feature additionally 
enables the research team to track how patients are using audios including whether they 
are listening to them, how often, whether they share them and whether those individuals 
access the audios as well. A Data Use Agreement (DUA) is currently being established 
with Dartmouth audio recording will not start until a DUA is in place.  
Details are as follows:  

Version#3 8/30/2022 
VA IRB Protocol  
Page 10 of 33 
 
      Saul J. Weiner, MD will oversee the project as principal investigator, and serve as site-
investigator for JBVA. He is 5/8th VA.  He will have access to PHI and will participate in data 
analysis 
      Corinna D. Falck-Ytter, MD - Dr. Falk-Ytter will assist with setting up the project at the 
Cleveland, facilitate submission of the site IRB, and supervise the site RA. She is 8/8th VA. 
She will have access to PHI.  
Frances Weaver, PhD will provide consultation strategies to optimize audio collection and 
on dissemination and implementation strategies for future work. She is based at Hines VA and 
is 8/8ths VA.  
Gunjan Sharma, PhD Dr. Sharma has served as project manager of all previous VA 
studies utilizing audio collected data. She manages budget and HR, assists with any IRB 
activities, and participates in 4C coding. She will work 100% Jesse Brown VA via an IPA.  She 
will have access to PHI, will be involved in recruiting subjects and obtaining informed consent.  
      Sherry Ball, PhD Based at the VHA Collaborative Evaluation Center at the Cleveland VA. 
She will lead qualitative work including implementation and assessment of adoption measures 
to achieve aim 3. She is 8/8th’s VA. She will have access to PHI.   
Alan Schwartz, PhD Dr. Schwartz, professor in the Dept of Medical Education has served 
as methodologist and analyst on previous studies utilizing audio collected data. He will 
oversee quantitative data management and conduct all quantitative analyses. He is 1 FTE at 
UIC, and funded via an IPA with a WOC at Jesse Brown VA.  He will work with de-identified 
data.  
Amy Binns-Calvey Amy has played a central role in the development of the 4C coding 
system with over 8 years of experience coding nearly 3000 encounters using the 4C method. 
She will lead 4C coding of the audios for Aim 2. She will be 60% UIC and 40% Hines VA on an 
IPA. She will have access to PHI, will be involved in recruiting subjects and obtaining informed 
consent.  
Ben Kass He will assist with coordinating Jesse Brown VA site activities (100% on an IPA 
with UIC), including consenting subject, and responding to any technical issues related to 
accessing the audio. He will also assist Dr. Sharma with 4C coding--for which he is trained--for 
Aim 2. He will have access to PHI, will be involved in recruiting subjects and obtaining 
informed consent.  
Valencia Burton will provide operational assistance to project staff, e.g., purchasing, 
regulatory compliance.  She is 100% VA based out of Jesse Brown VAMC. 
RA Cleveland (1.0 FTE on IPA, Yrs 1-3). RA will assist with coordinating Louis Stokes VA 
site activities, including consenting subjects, and responding to any technical issues related to 
accessing the audio.  
Paul Barr, PhD, MSc (Co-I, 0.1 FTE IPA, Yrs 1-3). Dr. Barr is assistant professor at the 
Dartmouth Institute, Geisel School of Medicine at Dartmouth, where he leads the HealthPAL. 
He will provide continuous consultation on the design and findings given his extensive 
expertise studying and writing about open audio outside VHA. Although Dr. Haslett will have 
primary responsibility for PHI (see below), Dr. Barr will have access as well as a back-up 
if/when needed..  
William Haslett, PhD (Co-I, 0.1 FTE IPA, Yrs 1-3). Dr Haslett, is research scientist in the 
Biomedical Data Science department at the Geisel School of Medicine at Dartmouth College. 
He will provide all technical support and guidance regarding HealthPAL. He will have access 
to PHI, in the form of email addresses of all subject participants. 
Brian Bartle (.05 FTE in Yrs 1 and 3). Brian is a data analyst at CINCCH, 100% VA and 
based at Hines VA. He will extract the patient outcomes and demographic data from the 
Central Data Warehouse. He will have access to PHI briefly as he runs search in the 
Corporate Data Warehouse.  

Version#3 8/30/2022 
VA IRB Protocol  
Page 11 of 33 
 
5.0 
Study Procedures 
5.1 Study Design 
• Describe experimental design of the study.  Include sequential 
and/or parallel phases of the study, including durations, and 
delineate which interventions are standard of care and which are 
research.   
• Include a description of how anticipated risk will be minimized and 
include an analysis of risk vs. potential benefit. 
• Provide description of the study population (delineate all categories 
of subjects – patients, providers, family members, employees, etc.). 
Include anticipated enrollment numbers 
• Include rationale for including or excluding certain populations – in 
particular vulnerable populations. 
• As applicable, provide information on any added protections for 
vulnerable populations.  
The setting will be primary care and diabetes specialty clinics at the two facilities.  This 
will be a naturalistic intention-to-treat study in which all patients in these clinics are eligible to 
consent to participate and data from all arms will be included in the analysis regardless of 
whether the patient listens to the audio. Also, there will be no exclusion criteria. We have no 
preconceptions about who could benefit: for instance, counter-intuitively, a severely hearing 
impaired patient who has difficulty following face-to-face encounters might benefit if they choose 
to share the audio with a caregiver.  
  
We have designed a three arm RCT (figure 2, below), with each arm varying as to who 
gets access to the audio after the visit and who is aware of the fact that audio recording is 
occurring during the visit. The design establishes the conditions for achieving the first two aims 
of the study, with each aim framed as testable hypotheses. Arm 1 is the intervention, with both 
provider and patient aware of the recording. It is designed to represent how OAA would be 
utilized in actual practice. In Arm 2 all parties are also aware of the audio recording, but this time 
neither gets access to the audio after the visit (although the research team does). Arm 2 is a 
control for isolating the effect of patient access after the visit to the audio in Arm 1. Both patient 
and physician are blind to whether they are in Arm 1 or 2 until after the visit.  Comparing these 
two arms will achieve the first aim of the study.  Arm 3 is a second kind of control, as the audio 
is again not shared with the patient. In this arm, however, the patient conceals the audio 
recorder, typically in an accessory such as an eye glass case (a method we have employed 
extensively14), so that the provider is unaware they are being audio recorded until after the visit, 
enabling the research team to achieve the second aim of the study by comparing Arm 3 to Arm 
2, isolating the effect of being aware they are being audio recorded on the provider’s 
interactions with the patient. Whereas in Arm 3 the physician is unaware they are being 
recorded, in Arms 1+2, they are aware and also know there is a 50% chance the patient will 
have access to the audio. This design enables Arm 2 to serve both as a control for Arm 1 to 
achieve Aim 1 (since patient only gains access to the audio in Arm 1) and as comparator with 
Arm 3 – to achieve Aim 2 -- since the physician believes the patient may gain access to the 
audio. 

Version#3 8/30/2022 
VA IRB Protocol  
Page 12 of 33 
 
 
The superscripts associated with each outcome in the “How Answered” boxes in figure 2 refer to 
the following strategies for testing the hypotheses in Aims 1 and 2: (a) A survey called the 
Patient Activation Measure (PAM), (Appendix A-1) is completed by patient 1-2 weeks after their 
visit22; (b) The “no show rate” is the inverse of Return Visit Adherence (RVA) rate;23 (c) the Refill 
rate is “proportion of days covered” (PDC), a metric CMS utilizes in its plan ratings;24 and (d) BP 
and HgB A1c are checked if they are repeated within 6 months; (e) The SEGUE Framework for 
evaluating and scoring communication behavior, as rated off of the audio recording by a 
research assistant utilizing a checklist (Appendix A-2)25 and (f) Content Coding for 
Contextualization of Care, also called “4C”26.  Note: (b-d) are extracted from the CDW and 
SEGUE and 4C completed while accessing the medical record and listening to the audios. 
Finally, for all three arms, the members of the research team collecting the data and coding the 
audio are blinded as to study arm.  There is no arm in which the patient is unaware a visit is 
audio recorded. Blinding the patient is not pertinent to achieving the aims of the study. 
Figure 3, the flow diagram below, allocates patients in a parallel three arm randomized 
controlled trial (RCT) to most efficiently achieve the aims of the study (the calculations for the 
specific sample sizes are justified later in the protocol – and shown in Tables 4.1 and 4.2. on pp 
24+25). 
Figure 3: Flow diagram for parallel RCT 
 
Figure 2: Study Design 
 
Arm 3 
• Visited recorded 
• Patient aware 
• Provider unaware 
• No provider or 
patient access 
Arm 1 
• Visit recorded 
• Patient aware 
• Provider aware 
• Patient and 
provider have 
access 
Arm 2 
• Visit recorded 
• Patient aware 
• Provider aware 
• No patient or 
provider access 
Hypotheses 1A-1D: OAA increases pt 
activation (1A), treatment adherence (1B), and 
self-mgmt of chronic disease (DM & HTN) 
(1C)? 
Hypotheses 2A-2B: Awareness visit is being 
recorded for patient improves physician 
communication (2A) and contextualization of 
care (2B)?  
How Answered: Compare measures in Arms 
1 and 2 for: 
•Patient activationa*  
•Treatment Adherence: no show rate b,  refill 
rate c 
•Glycosylated Hgb and BP 6 month following 
visitd   
 
How Answered: Compare measures in Arm 2 
with 3 for:  
•Physician communication behavior scorese 
rated by research team members blind to arm. 
•Physician attention to contextual factors 
essential to care planning.f  
Physician 
and patient 
blinded to 
Arm during 
visit 
*Superscripts reference instruments to test 
hypotheses as described below. 

Version#3 8/30/2022 
VA IRB Protocol  
Page 13 of 33 
 
 
 
Random assignment to arms will be made using computer-generated sequences within the 
diabetes clinics (1:1 randomization to Arms 1 and 2 in blocks of 4) and primary care clinics 
(5:5:2 randomization to Arms 1, 2, and 3, in blocks of 24, with one partial block of 8 patients). 
Solid lines represent unconditional allocation of participants to Arms; dashed lines represent 
random sampling of participants to create efficient unbiased subsamples for key hypothesis 
tests. The rationale for the design, starting at the top of the figure and working down, is as 
follows: 
• 
We start with two separate pools, diabetes and primary care clinics, because measurement 
of the effect of OAA vs control (Arm 1 vs Arm 2) on Hgb A1c is most efficiently accomplished 

Version#3 8/30/2022 
VA IRB Protocol  
Page 14 of 33 
 
just in the diabetes clinics where every patient has diabetes, many have an A1c>8, and A1c 
is re-checked at frequent intervals. Hence, analysis of an effect of OAA on diabetes control 
will occur only in the diabetes clinic population.  
• 
All other measures to achieve Aim 1 will occur in analysis of data collected in both the 
diabetes and primary care clinics. Hence data on the effect of OAA on show-rates (visit 
adherence), medication refill rates, blood pressure control etc…will be pooled for patients 
randomized to Arms 1 and 2 in both the diabetes and primary care clinics.   
• 
The PAM will be applied to the smallest possible subsample of Arms 1 and 2 to test 
hypothesis 1A (figure 2) with 80% power (Table 4.1, p 24) because it requires a phone call 
and takes about three minutes of RA and patient time.  Subsamples will be drawn at random 
without replacement by computer. As shown in the Gantt chart (Table 2, p. 18), these calls 
will be spread out and administered 1-2 weeks post visit (Table 4.0, p 23). This is in contrast 
to measures of blood pressure, Hgb A1c, and visit and medication adherence, which are 
extracted from the data warehouse.  
• 
Arm 3, essential to achieve Aim 2, will only occur exclusively in the primary care clinic 
because it requires blinding physicians to being audio recorded which is not relevant to 
measuring the effect of OAA on diabetes control, and would confound that measurement.  
• 
Similar to PAM, the SEGUE and 4C analysis require RA time and effort, so will be applied to 
the smallest possible samples.  Whereas PAM is intended to measure the effect of OAA 
(Arm 1) compared to control (Arm 2) on patient activation, SEGUE and 4C are intended to 
measure the effect of the physician knowing they are being recorded (Arm 2) versus not 
knowing (Arm 3) on the process (SEGUE) and content (4C) of the physician’s decision 
making. The SEGUE checklist and the 4C coding occur while the RA is listening to the audio 
of a visit, randomly sampling 250 audios from the 827 primary care visit audios in Arm 2 for 
comparison with the 250 audios from Arm 3. Random sampling from Arm 2 primary care 
visits will be performed by computer without replacement and without regard to whether they 
are also part of the hypothesis 1A subsample.   
 
Research activities for testing each aim: Anticipating and mitigating risks 
Aim 1: Assess the impact of an open access audio program on two behaviors (patient 
activation, treatment plan adherence), two services (ED utilization, inpatient admissions), and 
two chronic condition measures (glycosylated hemoglobin, blood pressure). 
Aim 1 hypotheses will be tested by comparing patients randomized to arms 1 (intervention) 
and 2 (control) on the outcomes specified in figure 2.  Following an informed consent process 
(see “Informed Consent Procedures” p 20, below), the Veteran will carry an audio into the 
encounter, regardless of the arm to which they are assigned.  They’ll return the recorder when 
they leave.  In Arm 1, the Veteran and provider will each be given access to a passcode to a 
website that will stream the audio from any internet enabled device.  Audio is streamed rather 
than downloaded, to prevent inadvertent sharing.  Also, with streaming, the platform can track 
for the research team whether, when, the number of times, and how long the audio is listened to 
by either party. In Arm 2 neither the Veteran nor provider will have access to the audio, but the 
research team can listen to it.  
The decision to give the provider access to the audio whenever the patient has access is 
based on conversations with providers during the proposal planning process who have said they 
would not feel comfortable having a patient listen to a visit that they couldn’t also listen to if they 
wished to do so. In Open Access Notes, both providers and patients have access to 
documentation of the visits, and so it seems logical that Open Access Audio should work in 
parallel, particularly given that current technology makes this feasible.  The research team will 
be able to track whether and when either party access the audio. This will not contaminate 
isolating the impact of the audio on the patient (e.g. were the provider to call the patient after 

Version#3 8/30/2022 
VA IRB Protocol  
Page 15 of 33 
 
listening to the audio to correct an error or elaborate), since we’ll know when providers log in, 
enabling us to control for it as a co-variate or exclude if it’s a rare event.   
 
Aim 2: Assess the impact of open access audio on provider communication behavior and on 
their attention to patient contextual factors (i.e. individual Veterans’ needs and circumstances 
relevant to planning effective care).   
Isolating the effect of the OAA on providers requires including an arm in which the provider 
is unaware of the recording, but which the research team can analyze, to serve as a control.  
This will occur in Arm 3 (figure 2), in which the audio will be carried and concealed by the 
Veteran so that the provider is not aware of it. As in Arm 2, neither Veteran nor provider will 
have access to the audio after the visit, but the research team will be able to listen to it. This will 
enable comparisons of provider behavior (communication and contextualization of care) in Arm 
3 (control) with Arm 2. Although the patient will not actually obtain the audio in Arm 2 after the 
visit, the physician does not know that, as at the time of the encounter they are blinded as to 
whether they are in Arm 1 (in which the patient will gain access to the audio) or Arm 2 (in which 
they will not).  
For assessing provider behavior change based on awareness that the patient may get the 
audio recording and listen to it to better understand their care, we are focusing on both the 
process and content of the interaction.  We selected SEGUE both because of its high evidence 
for concurrent and construct validity, and inter-rater reliability, and because it can be scored by 
an independent third party off of audio using a checklist.27 Whereas SEGUE is a process 
measure (how effectively a provider communicates without regard to the content of the 
discussion), 4C is a content measure, i.e. whether they follow up on what the patient needs 
regardless of how they go about it. 4C is coded by a trained RA with access to the medical 
record and audio recording; the RA documents clues that a patient is struggling with life issues 
complicating their care (“contextual red flags”), whether the provider asks about them 
(“contextual probes”), whether the patient reveals the underlying life challenges (“contextual 
factors”) and, if so, whether the physician addresses them (“contextualized care”).28 Contextual 
red flags are specific pre-determined variable extracted both from the medical record (including 
specific thresholds for missed appointments, emergency department visits, and medications that 
have not been refilled). 4C has strong published evidence for both construct validity and inter-
rater reliability (90%),26 and contextualized care significantly predicts a range of favorable 
patient health care outcomes.14 Notably, we have observed that physicians are more likely to 
contextualize care when they know they are being recorded.13 We have previously documented 
that communication performance and contextualization of care are separate constructs that do 
not correlate -- so must both be measured to fully assess patient centered provider behaviors.29 
Note that at the time of the coding, including completion of the SEGUE instrument, the audio 
coder will only have access to the designated codes for provider and patient, which will be 
utilized for tracking purposes.  This is because the first step when audio recordings are 
uploaded following each encounter is to assign them codes with provider and patient names 
stored separately to a cross walk file.  
Audio recording technology  
For the audio recording device, we plan to use VHA approved audio recorders with 
accompanying software, the DS-9000 and DS 9600 Olympus encrypting recorders with a pin 
code for security.  The audios will be uploaded by the RA to a workstation desktop on to our 
secure VHA research server, hosted in our Center of Innovation (COIN), using the VHA 
approved Olympus/ODMS Dictation Module 7. Each audio file name will include the Veterans 
last name and date of the visit., for Arm 1), The project manager will then trim the audio to  
remove all the identifiers using the audacity software and upload the deidentified audio directly 
to the HealthPAL website(https://va.audiohealthpal.com) managed by the research team at the 

Version#3 8/30/2022 
VA IRB Protocol  
Page 16 of 33 
 
Department of Biomedical Data Science at the Geisel School of Medicine at Dartmouth.    At 
that point an automated script sends an email to the patient and provider (in Arm 1) providing 
them a link to login to access the audio from any internet enabled device.  The Dartmouth 
system, HealthPAL, 21 which is specifically designed for sharing audios with patients, also 
enables patients to selectively share their audio with others, such as family or caregivers, using 
a highly secure process that enables them to send an invite email and text message with a six 
digit code, that enables the user to set up their own unique password. This feature additionally 
enables the research team to track how patients are using audios including whether they are 
listening to them, how often, whether they share them and whether those individuals access the 
audios as well. Because many of the patients at participating sites are older and have low levels 
of technology literacy, we will make an attempt to reach by phone any subject who has not 
accessed HealthPAL within 7 days following their visit. These data will be variables in our 
analysis for the outcomes in Aim 1, e.g. for patient activation, medication adherence etc… 
    The data encryption and storage methods utilized in HealthPAL are highly secure. Health 
PAL is a web application that runs on Amazon’s cloud infrastructure, Amazon Web Services 
(AWS). The root account under which the application is hosted is managed by Dartmouth 
College Central IT, which has a Business Associates Agreement in place with Amazon for the 
use of Amazon’s HIPAA-eligible cloud services with Protected Health Information. All HealthPAL 
data are encrypted in-transit using HTTP over SSL, and all HealthPAL data are encrypted at 
rest using AES-256 encryption. and are monitored for safety using standard software storage 
review processes, including log analysis.      
     In addition, as detailed in the proposal, participating patients will complete a survey indicating 
any concerns they have following participation. In addition, they will be provided with a number 
to call if they have either technical questions or concerns, that goes directly to voicemail that the 
site research assistant accesses daily. Appropriate and adequate subject recruitment will be 
monitored by the research team which will convene across sites by conference call monthly. 
There is little reason to anticipate the need to change sites, given the multitude of patients at 
clinics at each site. Were a new site required, the principal investigator would likely contact one 
of several sites that are currently or have recently participated in a separate audio recording 
project and hence could efficiently adopt this study if needed. 
Because HealthPAL is managed by Dartmouth, not the VA, we will include in the consent 
document that VA will not have control of the data at all times. A recent precedent is an ongoing 
study conducted by this project’s implementation specialist (Sherry Ball) titled Home Monitoring 
for Early Detection of Chronic Disease Exacerbation, and approved by the Cleveland facility 
IRB, which will also review this study if funded. That study requires placement of hardware and 
software developed by an outside vendor in Veterans’ homes with internet connectivity for 
exchanging data on health and function daily with the VA research team.  The IRB/R&D 
decision to approve utilizing non-VA infrastructure was based on the premises that (a) the risk of 
data breaches is low, (b) the research question is important, (c) the patient who chooses to 
participate is fully informed that the VA has not approved and cannot monitor the security of all 
of the data, and (d) the VA does not have the required infrastructure to do the study. All are 
applicable for this proposed study as well. 
 
Aim 3: Assess patient, provider, and leadership perceptions of the extent to which the program 
is safe, not burdensome, and worthwhile at both the start and at two years into the program.  
In our prior research on the implementation of patient-collected audio in the clinical 
encounter, which utilized a RE-AIM framework to study these constructs with respect to the 
implementation of audio technology in the clinical setting, we identified three key determinants 
of 
implementation 
success 
encompassing 
reach 
(or 
adoption), 
acceptability 
and 
appropriateness (or “fit”): the extent to which stakeholders, including patients, providers and 

Version#3 8/30/2022 
VA IRB Protocol  
Page 17 of 33 
 
organizational leaders perceive the audio recording and data use process as (a) safe to do so, 
(b) not burdensome and of (c) value30,31. We also found that provider perceptions often evolve, 
starting with some initial concern that the program may interfere with their practice routine, 
and/or that the audio may be used for purposes other than intended. Once the project is 
established and has been active for some time, these concerns typically resolve.  
Although we have seen this pattern consistently in our prior work, we believe it is 
necessary to reassess during this study because the purpose of the data collected and its use 
differs. In the past, the audio has been collected solely for the purpose of assessing provider 
decision making during the encounter. In this study the purpose of the audio data is to assess 
the value of a new technological resource for patients. On the one hand, this should generate 
less provider anxiety, as providers are not the primary focus. On the other hand, in this project 
the audio is shared with the patient, which may raise concerns for some providers.  
Finally, we believe that it is important to collect attitudinal data about OAA from facility 
leaders with decision-making authority, to gauge its potential for widespread dissemination. This 
will be carried out both early on and near the end of the study, when they are most uncertain 
and most informed about the pros and cons of the program, respectively.  
 
Table 1 details each of the three principles as they apply to the proposed study, the 
planned practices for advancing the principles, and the evaluative methods.  
 
Table 1: Principles and practices for adoption of OAA and methods of evaluation (Aim 3)  
Principle  
Practice  
Evaluation 
Project must feel safe for 
providers/staff and Veterans:  
• Providers and staff will feel 
comfortable that patients will 
use the audio exclusively to 
enhance their care.  
• Leadership 
will 
feel 
comfortable having staff in their 
service line or facility participate 
• Veterans will feel confident 
the audio is secure, with a clear 
understanding 
of 
who 
can 
access it and how. They will 
also feel confident that the 
information they disclose when 
completing the PAM and 4C 
data will be securely stored, 
and shared after identifiers 
have been removed. 
 
Providers will have access to 
audio 
whenever 
their 
patients do and will always 
be informed when a visit was 
recorded -- in arm 3 that will 
occur after the visit.  
The encryption methods and 
access 
security 
features 
employed will be shared with 
patients during the informed 
consent process.  
PAM and 4C coding results 
are stored using a crosswalk 
file so as to be separated 
from provider identifiers. 
Email 
link 
to 
an 
anonymous 
optional 
survey of providers and 
staff near start and end of 
project, 
regarding 
their 
experience with program: 
Do 
they 
feel 
safe? 
(Appendix A-3, Provider 
survey) 
Also, conduct more in-
depth 
provider 
focus 
groups 
near 
start 
(Appendix 
A-4, 
Pre 
Provider 
focus 
groups) 
and 
end 
of 
project 
(Appendix 
A-5, 
Post 
Provider focus groups). 
Examine 
on 
the 
organizational level from 
the standpoint of leaders 
(Leadership Interviews, A-
6). 
Patient 
exit 
survey, 
completed during a 3 
month block each year 

Version#3 8/30/2022 
VA IRB Protocol  
Page 18 of 33 
 
(Appendix A-7) includes 
perceived 
safety 
assessment items.  
Project should be embedded 
in 
ongoing 
activities 
for 
providers/staff and Veterans. 
• Providers/staff: Because of 
competing responsibilities for 
patient care, providers and staff 
should guide decisions about 
when to participate. 
• Leadership will be aware of 
any potentially emerging issues.  
• Veterans: Accessing audio 
should be easy and convenient 
so that it fits into daily pattern of 
using 
portable 
device 
(cell 
phone, ipad) or home computer. 
Veterans are prepped and 
given the audio recorder 
prior to encounter. Process 
is aborted if patient is called 
for visit so as not to delay 
care process. Other than the 
presence of the recorder, 
nothing is added to the 
encounter. 
Patients are provided with a 
number to call and an email 
address if they are having 
technical 
problems 
accessing 
their 
audio.  
These will be accessed daily 
by the RA using a VA 
voicemail account.  
Include items regarding 
perceived 
burden 
in 
provider 
surveys 
and 
focus 
groups. 
(Appendices A-3 through 
A-5) See above.  
Examine 
on 
the 
organizational level from 
the standpoint of leaders 
(Leadership Interviews, A-
6). 
Patient 
exit 
survey 
includes 
effort 
assessment (Appendix A-
7). 
Value of project should be 
evident 
for 
providers/staff 
and Veterans:  
• Providers/Staff: 
To 
appreciate the value of audio 
recorded data, clinicians and 
staff should receive updates on 
the adoption of the program. 
• Leadership will advise on the 
sustainability of the program 
based on value.  
•  Veterans: 
Participating 
veterans 
should 
perceive 
sufficient value that they will be 
motivated to utilize the resource 
when 
they 
have 
questions 
about their care by logging into 
to listen to the audio. 
 
At each site, project team 
will 
provide 
quarterly 
updates on the adoption of 
the program, including how 
often 
their 
patients 
are 
accessing the audio. This 
can be done via email to 
participating providers and 
staff. 
 
Include items regarding 
perceived 
value 
in 
provider 
surveys 
and 
focus 
groups. 
(Appendices A-3 through 
A-5). See above. 
Examine 
on 
the 
organizational level from 
the standpoint of leaders 
(Leadership Interviews, A-
6). 
Patient survey includes 
value 
assessment 
(Appendix A-7). 
 
Table 2: The following Gantt chart outlines each phase of the study (and initials of responsible 
personnel): 
 
Year 1 
Year 2 
Year 3 
Hire site RA for Cleveland Facility (CF-Y) 
Open Audio team (Dartmouth) to customize and pilot test 
with assistance of teams in Cleveland and Chicago 
(WH,PB,SW,CF-Y) 

Version#3 8/30/2022 
VA IRB Protocol  
Page 19 of 33 
 
Open Audio Data Collection (GS, BK, Cleveland RA) 
Outcomes and demographics extracted from CDW for A1c, 
BP, Adherence measures (BB) 
Survey providers/staff (GS, BK, Cleveland RA) 
exit patient surveys (GS, BK, Cleveland RA) 
PAM survey phone calls post visit (GS, BK, Cleveland RA) 
Focus groups staff (SB) 
SEGUE + 4C Coding of sample of recorded visits (ABC, GS, 
BK) 
Site leadership interviews (SB) 
Data analysis of outcomes, contextual error rates for each 
arm of study (AS) 
Dissemination (SW, CF-Y) 
 
How anticipated risk will be minimized and an analysis of risk vs. potential benefit 
 
• 
Patients: The benefits of audio recording with analysis of specified outcomes, which are 
essential to the study, is that it will enable an assessment of whether this technology 
enhances patient health care outcomes as well as understanding of their care.  
Anticipated risks for patients included that: (a) carrying an audio recorder into a visits 
could be an inconvenience, could make them feel uncomfortable about disclosing private 
information. We minimize these risks by assuring you that if you are always welcome to 
turn of the audio recorder during the visit if you decide you don’t want to participate for 
any reason. We also minimize these risks by informing you that your audio recording is 
managed and stored with a comparable level of data protection and security as your 
medical record; (b) There is the possibility that there could be a breach in confidentiality 
due to a data security failure related to the storage or transmission of an audio file, 
including during collaboration with the Dartmouth team. The reason we are working with 
this group is that they have developed the technology, which is currently not available 
within the VA. They have minimized the risk to loss of confidentiality by using the best 
available software systems and data security protocols which are the same or 
comparable to VA systems.  While we cannot say how likely a data breach is, we 
estimate the risk to be small; (c) Some of the audio recordings  will be listened to by one 
or more research assistants. This will occur to a random sample of 10% of audio 
recorded visits.  The research team will minimize risk of breach of confidentiality by 
directly uploading the audio file to the secure  HealthPAL server. (d) A VA data analyst 
will look up patient specific information in the CDW about  blood pressure, blood sugar, 
emergency department and hospitalization usage, prescription refill rates and 
appointment attendance, but will not have access to the medical record. The privacy will 
be minimized by keeping patient names separate using a crosswalk file. (e) The audio 
recording process could possibly undermine a patients relationship with their doctor. We 
have minimized this possibility by only including physicians and other health care 
providers who want to participate and have consented to participate; (f) As explained 
above there are three brief questionnaires patients will or may be asked to complete. 
The first is the brief assessment of their knowledge and comfort with medical 
terminology. All participating patients do that one.  The second is a brief survey just as 
they exit their visit and return the audio recorder. That questionnaire will ask them about 
whether they found the experience valuable, how much effort it took, and whether they 
were comfortable participating. There is about a 25% chance they will be asked to do 
this questionnaire. The third questionnaire is conducted by phone 1-2 weeks from their 

Version#3 8/30/2022 
VA IRB Protocol  
Page 20 of 33 
 
visit and is intended to find out how confident and comfortable they are managing their 
health needs and care plan. There is a 45% chance they will be called to complete this 
questionnaire.    On all three questionnaires the research team will substitute a study 
identification code in place of their name, so that if anyone were to find the data they 
would not know that it is about a particular patient. We’ll keep the study ID code sheet 
that contains the names of all study participants with their study IDs, in a separate 
secure location on a server that also belongs to the VA (i.e. a crosswalk file). That sheet 
will be deleted at the end of the study. There may be some risk in sharing slightly 
personal data, including how knowledgeable you are about medical terms, and how 
comfortable you are managing their own care. They may find sharing this information 
personally uncomfortable.  We minimize this risk by assuring you that if, for any reason, 
you wish not to answer specific questions you need not do so.  
 
• 
Providers: The benefits of audio recording with analysis of provider experience, is that it 
will enable an assessment of whether this technology enhances patient health care 
outcomes as well as understanding of the providers perspective, which is essential to 
long term implementation planning. Risk and mitigation of risk include that providers: (a) 
may feel some unease that their interaction with their patient is being recorded and that 
they may listen to it and share it with others, such as caregivers and family. Any audio 
shared with patients will be streamed so patients will not have access to an audio file 
that can be uploaded to a third-party site; (b) There is the possibility that there could be a 
breach in confidentiality due to a data security failure related to the storage or 
transmission of an audio file. We will be using a crosswalk file as detailed directly above, 
with assignment of a discrete code for physician and patient, assigned to each audio and 
related instruments used for audio coding, such as the SEGUE. Audio data in arm 1 – 
the arm that shares audio with the patient and provider -- will be uploaded directly to  the 
HealthPAL website(https://va.audiohealthpal.com)  managed by Geisel School of 
Medicine at Dartmouth where a research team specializes in managing patient audio 
recorded data. The VA cannot vouch for the security of the system nor take 
responsibility for any breaches.  The reason we are working with this group is that they 
have developed the technology, which is currently not available within the VA. They 
have minimized the risk to loss of confidentiality by using the best available software 
systems and data security protocols which are the same or comparable to VA systems.  
While we cannot say how likely a data breach is, we estimate the risk to be small; (c) 
Providers will be invited to participate in twobrief survey’s over the course of the three 
year study. Additionally, the providers will be invited to participate in two focus groups 
where their names will be replaced by a code and linked to a crosswalk file. Because the 
information we are obtaining is not sensitive – it is solely about the experience of the 
providers participating in this study, , risks of participation are small but there could be 
some inconvenience. They are welcome to abort them at any time. 
 
Leadership interviews:  The benefits of leadership interviews is that the data collected on 
their perspective will inform efforts to disseminate and implement patient collected audio 
more broadly if the findings demonstrate it is effective.  These interviews, which will be 
recorded, will be voluntary, coded and linked to a cross walk document.  The recording 
will be heard by members of the research team. There is a minimal risk of breach of 
confidentiality, and the data collected is not sensitive as it does not include discussions 
of any specific patient’s care or of any particular provider. 
    
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 21 of 33 
 
5.2 Recruitment Methods 
• State how many subjects will be needed.  
The numbers and distribution of patient subjects is presented in Figure 3, above.  As noted, a 
total of 1904 will be recruited from the primary care clinics and 404 from the diabetes clinics, 
totaling 2308, all above 18 years of age.  
 
The primary subjects are Veterans. However, since providers will be audio recorded as well, 
and since we will be studying the effect of being audio recorded on the behavior and outcomes 
of their patients, providers will be considered as human research subjects as well. Since they 
are not a unit of study, there is no specific sample size.   
 
We also plan to interview hospital leaders about the program. They are also subjects, but given 
the low risk to them, will likely be exempted from a formal consent process. We anticipate 
interviewing five leaders at each of the two facilities, for a total of 10.   
 
• Describe when, where, how and by whom potential subjects will be 
identified and recruited.  
• Describe materials that will be used to recruit subjects, e.g., 
advertisements.  Include materials as an appendix or separate 
attachment. 
• Describe any payments to subjects, including the amount, timing (at 
the end of the study or pro-rated for partial study participation), 
method (e.g., cash, check, gift card), and whether subjects will 
experience a delay in receiving the payment. 
 
 
Recruitment will proceed as follows:  
 
(A) Providers (physicians or nurse practitioners) will be notified of the project by the site 
investigator by attending their section meetings (at the invitation of the section chiefs — who 
have already indicated support of the project) and providing a brief description and study 
brochure (same as the one shared with patients). Those who wish to participate will sign a sheet 
providing their name and contact information. In addition, the PI will send out an email to 
providers who were not able to attend the meeting. The site investigator and/or RA will meet 
with all who indicate interest to complete the informed consent process.  
 
NOTE: When clinical section meetings are held virtually because of the COVID-19 pandemic, 
these activities will also be conducted virtually. Handouts and sign-up sheets will be emailed. 
 
Recruitment of providers for focus group participation will occur via email invitation, and is 
optional for all participating providers.  Providers are first informed of the focus groups in during 
the informed consent process for the study, as detailed in the consent document.  They are 
informed that they will contacted about participation, that participation will be voluntary, that the 
focus groups will be audio recorded, and that to minimize risk, they will be asked not to use their 
names or the names of any patients during the focus groups. They are also informed that lunch 

Version#3 8/30/2022 
VA IRB Protocol  
Page 22 of 33 
 
will be served.  The email text is provided with this protocol and informs providers that “if you 
are interested, please reply to this message.”  
 
(B) Recruitment of patients will occur in primary care and diabetes clinics in Chicago (Jesse 
Brown) and Cleveland (Louis Stokes). Since the former is nearly three times larger than the 
latter, it is anticipated that recruitment will be proportional. However, it will continue at both site 
until target numbers are achieved. The method of recruitment is as follows: 
 
In primary care, the research team will obtain the panel lists of all participating providers.  These 
are accessed through CPRS under each providers schedule.  They will then randomly select the 
subset of patients necessary to contact via mail 2-4 weeks prior to their scheduled appointments 
informing them of the study and explaining how to opt out. Based on a 2014-16 study we 
conducted involving audio recording visits (SDR 12-280) we plan to mail 2767 prospective 
subjects, taking into account that 20% opt out, decline during the outreach call, are not 
reachable, don’t show up in clinic, or opt out on site.   About a week prior to the visit the RA will 
then phone those who do not opt out, to achieve recruitment of 1904 patients to be distributed 
across 2 sites (Chicago and Cleveland) and divided by randomization, utilizing the computer-
generated sequences detailed in the research to allocate 827, 827, and 250 subjects to Arms 1, 
2, and 3 respectively. Those who are reached and affirm will agree to arrive approximately 20 
minutes early for their appointment.  
 
For the diabetes clinic patients, an additional 404 patients will be randomly assigned to Arms 1 
and 2. The same opt out strategy described above for primary care will be employed to recruit 
patients to these clinics as well.  
 
(C) Upon arrival, they will complete the informed consent process and the brief literacy screen 
(REALM-SF), receive the audio recording device, and instructions based on the arm of the 
study to which they are randomized, as to whether to keep the audio recording out in the open 
(Arms 1 and 2), or to conceal it (Arm 3). A code, assigned to the encounter, will be utilized as 
the ID on the literacy screen, and linked to the patient name and medical record number stored 
in the cross walk file on a separate VHA approved server. After the encounter, patient and 
provider participants will receive a link to access the audio if they were in Arm 1.  If they are 
called prior to completing the process, their appointment takes precedence and they will not 
participate. 
 
(D) In addition to pre-visit outreach, the RA will recruit patients at an information desk on site 
when they arrive in the waiting area with a poster and a brochure inviting Veterans to learn more 
if they are interested.* This will occur until recruitment targets are met. Those who are seeing a 
participating provider and want to participate will complete the informed consent process and 
the brief literacy screen (REALM-SF). They will receive a $20 gift card to remunerate them for 
the additional work of completing the survey after the visit and for randomly participating in a 
follow up call for the PAM survey.  If the Veteran is called prior to completing the process, their 
appointment takes precedence and they will not participate.  
 
*COVID-19 MODIFICATION: As long as infection transmission remains a risk the 
following protocol modification will be in place:   
• 
Only phone based recruitment will occur. 
• 
The informed consent process will be completed over the phone prior to the visit, 
except for the final signing process.  
• 
The RA will still meet the patient in the waiting area, both masked.  However, it will 
be a very brief meeting and they will remain 6 feet apart.  There will be a small fold 

Version#3 8/30/2022 
VA IRB Protocol  
Page 23 of 33 
 
up table in the waiting area. The patient will sign the consent form, collecting the 
audio recorder, and complete the 7 item REALM-SF which consists of reading out 
loud 7 words, and takes less than 1 minute.   
• 
Post visit, the patient will return the audio recorder to the table and collect $20 gift 
card from the research assistant.  Again, the RA will maintain a physical distance 
of at least 6 feet and will remain masked.  
 
5.3 Informed Consent Procedures 
• Indicate if informed consent will be obtained and/or if you are 
requesting 
a 
waiver 
of 
informed 
consent 
or 
waiver 
of 
documentation of informed consent.  If the research involves 
multiple phases, specify for which phases of the research the 
waiver(s) is being requested and/or the informed consent will be 
sought. 
• Describe who will be obtaining informed consent, if applicable, and 
any circumstances that may need to be addressed (e.g. subjects 
with impaired decision making ability and the use of a legally 
authorized representative, etc.) 
• If applicable, indicate how local site study personnel will be trained 
regarding human subjects protections requirements and how to 
obtain and document informed consent. 
Providers: As described in recruitment section above, participating physicians 
and nurse practitioners will complete an informed consent process. As noted, 
those who wish to participate will sign a sheet providing their name and contact 
information. The site investigator and/or RA will meet with them to complete the 
informed consent process. 
Patients: As described in recruitment section above, prior to participation but 
upon arrival for their appointment, patients will complete the informed consent 
process. As noted, the RA will meet patients (who have already assented by 
phone) prior to their appointment to complete the consent process.  
All personnel will complete all mandatory VA training for human subject research and 
have current certificates.   
Hospital Leaders: As explained above, we plan to interview hospital leaders 
about the program. They are also subjects, but given the low risk to them, we 
have requested for a waiver of documentation of consent. We will be contacting 
them via email asking them to participate in the research study by attaching the 
Informed consent explaining the research study and their participation.. 
 
5.4 Inclusion/Exclusion Criteria 

Version#3 8/30/2022 
VA IRB Protocol  
Page 24 of 33 
 
• Describe the criteria that determine who will be included in or 
excluded from the study.  
There is no plan to exclude any category of patients in participating clinics who wish to 
participate and complete the informed consent process. Women and minorities will be included.  
There is no plan to include children in this study.  
Hence we plan to include individuals who consent from the following groups:  
Patients who seek medical services at either the primary care clinics or the diabetes clinic; 
Providers who are either physicians or independent practitioners such as Advance Practices 
Nurses in these clinics; and hospital leaders whose portfolio of responsibilities includes 
management or oversight of clinical care.  
5.5 Study Evaluations 
• Describe all evaluations to be conducted (including screening; 
tests/questionnaires that will be administered; any procedures that 
subjects will be required to complete) and data collection methods.  
Include materials as an appendix or separate attachment. 
There will be several sources of research material. Instruments are included in 
appendices A-1 through A-8 as indicated below and as attached: 
A) Audio recordings of encounters made with written consent by both patients 
and providers. The audios are collected for research purposes. 
B) The patient’s medical record: Following a 10% (n=250) random sample of 
audio recorded visits (arm 3), again with written patient subject consent, a trained 
coder will extract discrete pre-determined variables for “4C” coding from the 
medical record after the visit, based on the referenced 4C manual. 
C) Phone calls to a random (40%) sample of patients to complete the Patient 
Activation Measure. 
C) Surveys of all patients (a brief literacy screen), and of both patients (25% of 
encounters) and all providers about their experience participating in an OAA 
program. (for patients: A-1, A-7, A-8; for providers: A-2, A-3).  
D) Provider focus groups about their experience participating in an OAA 
program. (A-4, A-5) 
E)  Structured interviews with facility leaders about their experiences with the 
OAA program. (A-6) 
 
5.6 Data Analysis 
• Provide 
sample 
size 
determination 
and 
analysis 
(include 
anticipated rate of screen failures, study discontinuations, lost to 
follow-up etc.). 
• Describe how, where and by whom the data will be analyzed. 

Version#3 8/30/2022 
VA IRB Protocol  
Page 25 of 33 
 
Sample size calculations:  
The samples sizes shown in each arm and sub-arm in Figure 3, above, were determined by the 
proportions of patients in participating clinics with the conditions or behaviors of interest. Table 
3, below, provides that information, starting with the sizes of the clinic populations, drawn from 
the CDW.  Note that for blood pressure and Hgb A1c we include the baseline rates of patients 
who meet BOTH the criteria of a specific threshold of poor control AND repeat measurement 
within a 6-month window, since six months is the window we plan to use for follow up 
assessment of the efficacy of the intervention compared with control.  
 
Table 3: Clinical characteristics of participating clinical sites in 2018 
 
Chicago 
Cleveland 
Primary care clinic population 
(2 visits in past 2 years) 
32,791 
11,796 
Diabetes clinic population 
  1,245 
  1073 
Total 
34,036 
12,833 
Primary care + Diabetes 
Most Recent SBP>140 or 
DBP>90* with repeat in < 6 
Mo 
10,959 (32%) 
1513 (12%) 
Proportion of Days Covered* 
88% 
84% 
Return Visit Adherence** 
93% 
94% 
Diabetes Clinic only 
Hgb A1c>8 with repeat in< 
6Mo  
821 (66%) 
751 (70%) 
*A measure of medication adherence from the CDW: proportion of days patients had pills to 
cover medication needs based on refill rate over a 120 day interval.  
**A measure of appointment adherence from CDW: The percentage of all visits scheduled to 
any clinics by patients seen in these clinics in 2018. 
Determining the specific sample sizes show in the flow diagram from the data in table 3 required 
determining the power needed to test each hypothesis with at least 80% power to detect the 
expected effect. 
We describe next the planned analysis for testing each hypothesis to achieve each aim:  
Aims 1 and 2: 
As summarized in Table 4, we will apply several measures to assess the effectiveness 
of the intervention, testing hypotheses to achieve Aims 1 and 2. For each hypothesis, 
comparisons between arms will be made using mixed-effect linear or logistic regression as 
appropriate for the outcome, with a fixed effect of study arm, VA site, and primary vs. secondary 
care setting, and a random effect of provider to accommodate clustering of visits within 
providers. Of primary interest is the direction, magnitude, and significance of the regression 
coefficient associated with study arm. Primary analysis will compare arms on an intention-to-
treat basis, without regard to whether audio made available in Arm 1 is actually accessed by 
anyone; secondary analyses will include covariates representing whether, how often, and by 
whom (patient or provider) audio is actually accessed in Arm 1 to assess the impact of audio 
use (vs. availability). As patients will be randomized to arms, we do not expect demographic or 
other patient differences among arms, but will conduct tests for differences, and if any variables 
differ significantly among arms, we will include them as covariates in the regression models. 
 

Version#3 8/30/2022 
VA IRB Protocol  
Page 26 of 33 
 
Table 4. Hypotheses and evaluation strategy for effectiveness of intervention (Aims 1 and 2)  
Hypothesis 
Evaluation Strategy 
1A: OAA increases patient activation. 
Administer PAM (Appendix A-1) 1-2 weeks 
post visit via phone call and compare 
scores in Arm 1 vs. Arm 2 
1B: OAA increases care plan adherence, 
including fewer no-shows and more on-time 
refills. 
Utilizing the data warehouse measure 
Return Visit Adherence (RVA) and refill 
rates (PDC) 24,32,33 in Arms 1 vs. 2.  
1C: OAA reduces Hgb A1c and BP 
Same as above: Utilizing data warehouse 
measure A1c and BP.  
2A: Providers obtain higher scores on measures 
of communication behavior when they know they 
are being recorded.  
 
Completed by trained RA off of Audio: 
SEGUE 
checklist 
for 
communication 
completed by research team (Appendix A-
2)25, comparing randomly sampled visits in 
Arm 2 vs Arm 3.  
2B: Providers obtain higher scores on measures 
of attention to patient contextual factors essential 
to care planning when they know they are being 
recorded.  
4C 
for 
contextualization 
of 
care34, 
comparing randomly sampled visits in Arm 
2 vs Arm 3.  
 
As noted earlier, the PAM will be completed over the phone by the RA initially at one week 
after the visit (this will be extended to 10-14 days if it turns out a substantial number of patients 
listen to the audio more than one week after the visit).  
Table 4.1 lists the calculated power for the intention-to-treat analyses for Hypotheses 1A-1C 
by outcome. These analyses are based on data from 1029 combined primary care and diabetes 
clinic visits each in Arms 1 and 2 (i.e. sample size 2058), except for measuring the effect of 
OAA on Hgb A1c (in 1C) which is based only on 202 diabetes clinic visits each in Arms 1 and 2 
(total 404). For patient activation (1A), we assume veterans with no audio access will have a 
mean PAM score of 56 and standard deviation of 19.5, based on a 2016 national study of VA 
health care users;35 clinically meaningful differences in activation have been described as 4-6 
points.36 Since testing hypothesis 1A requires a phone call to administer the PAM, we will 
randomly contact 924 participating subjects, an average of 9 calls/week across sites, which will 
achieve 80% power, instead of calling all 2058 subjects.  Baseline rates of each undesirable 
outcome for 1B are derived from our previously published study in the VA using recordings 
without provider awareness or patient access (as in Arm 3), and we expect to find an odds ratio 
of 0.5 for each outcome when patients have access to audios.14 Baseline rates of each 
undesirable outcome for 1C are derived from the data warehouse (Chicago and Cleveland 
facilities – see table 3, above). 
 
Table 4.1: Expected differences and power for planned intention-to-treat analyses for 
hypothesis 1A-1C 
Hypothesis Outcome 
Baseline without 
audio 
available 
Expected with 
audio 
available 
Sample 
size 
Power 

Version#3 8/30/2022 
VA IRB Protocol  
Page 27 of 33 
 
(Arm 2) 
(Arm 1) 
(total) 
1A 
Patient activation (PAM 0-
100 score) 
56.0 
60.0 
924 
80% 
1B 
Visit no-show rate  
7% 
4% 
2058 
95% 
Medication Proportion of 
days not covered 
13% 
7% 
2058 
80% 
1C 
Hgb A1c > 8 with repeat in 
< 6 Mo  
66-70% 
33-35% 
404 
80% 
Most Recent SBP>140 or 
DBP>90* with repeat in < 
6 Mo 
39% 
24% 
2058 
99% 
 
Also, while not included in Table 4.1, because it is not a primary objective of Aim 1, we will 
explore whether OAA is associated with fewer ED visits and hospitalizations.  With the current 
sample size, we will in fact have 80% power to detect a decrease in inpatient admissions from 
14%, the 2018 rate in our study population, to 9%.  
Table 4.2 lists the calculated power for the analyses for Hypothesis 2A and 2B (comparisons 
of Arm 2 vs. Arm 3) based on actual Veteran visits by outcome. Baseline SEGUE scores are 
based on SEGUE’s validation study among general internists (mean 17.5, sd 1.77); that study 
found that medical students in a skills lab achieved mean SEGUE scores of 21.1 (sd 2.1), and 
we take that as our minimal expected level of improvement from knowledge that one is being 
observed. Baseline rates of contextual probing and planning are based on our prior study in the 
VA using recordings without provider awareness or patient access (as in Arm 3), and 
incorporate the expectation that contextual red flags suitable for probing will be present in 2/3 of 
encounters and contextual factors that should be incorporated in planning will be present in ¼ of 
encounters overall.14 Expected increases in probing and planning are based on odds ratios from 
our prior study comparing these measures in undercover visits to providers with the same cases 
presented to medical students in a testing setting (and are thus conservative measures of how 
much better providers would likely perform if they were aware vs. unaware they were being 
tested).13 To obtain at least 80% power for each of these comparisons, we will randomly sample 
250 primary care clinic visits (from 827 Arm 2 primary care visits), and enroll all 250 primary 
care visits in Arm 3 for the SEGUE and 4C coding processes. For 4C, we expect 168 visits/arm 
with red flags suitable for probing and 63 visits/arm with contextual factors suitable for planning. 
 
Table 4.2: Expected differences and power for planned analyses for hypotheses 2A and 2B 
Outcome 
Baseline (doctor 
unaware, n=250) 
Expected (doctor 
aware, n=250) 
Sample 
size 
(total) 
Power 
SEGUE 
provider 
score  
17.5 
21 
500 
99% 
Contextual probing 
rate13,26 
32% 
47% 
500 
80% 
Contextual 
59% 
82% 
500 
82% 

Version#3 8/30/2022 
VA IRB Protocol  
Page 28 of 33 
 
planning rate13,26 
 
All effects measured in Aim 1 and 2 will be adjusted for patient sociodemographic 
characteristics which will be obtained from the CDW (including age, gender, and race) or from 
administering the REALM-SF to assess literacy.  Internet access, for those in the intervention 
arm, will be ascertained by census track using geocoding of patient addresses, based on FCC 
data, a method developed by Perzynski et al.37,38 
 
Aim 3:  
Qualitative work will address safety, burden and value (as operationally defined in table 
1, pp 17-18 above) in regard to adoption and implementation of OAA. This analysis will be 
based on the data collected utilizing the instruments referenced in table 1 and contained in the 
appendices: the surveys of both patients and providers about their experience participating in an 
OAA program. (for patients: A-1, A-7, A-8; for providers: A-2, A-3); the provider focus groups 
about their experience participating in an OAA program. (A-4, A-5); and structured interviews 
with facility leaders about their experiences with the OAA program. (A-6) 
 
Data will be analyzed using thematic synthesis of leadership interviews, focus groups 
and patient surveys by 2 qualitative analysts (SB and ABC) with experience in qualitative 
methodology and ready access to qualitative data analysis computer software. We will create an a 
priori framework that includes safety, burden and value on three levels: provider participant, 
Veteran, and leadership. Further creation of codes and themes will be iterative. To ensure rigor 
and validity of findings each analyst will independently code and develop emerging themes. 
Each theme will be discussed between the two analysts until consensus is reached.    
 
5.7 Withdrawal of Subjects 
• Describe any anticipated circumstances under which subjects will 
be withdrawn from the research without their consent.  
None 
• Describe the consequences of a subject's decision to withdraw from 
the research and the procedures for orderly termination of 
participation by the subject (e.g., the subject contacting the 
investigator for an end-of-study visit). 
No consequences. No consequences. For instance, if a patient 
decides that they don’t want to record their visit, the just default to 
usual care.  
• Describe procedures if a subject is withdrawn or withdraws from the 
intervention portion of the study but agrees to continue in the 
follow-up phases or for safety outcome purposes. 
Not applicable. For instance, if a patient decides not to record their 
visit, there will be nothing to follow up.   
6.0 
Reporting 

Version#3 8/30/2022 
VA IRB Protocol  
Page 29 of 33 
 
• Include procedures for reporting unanticipated problems, serious adverse 
events, and protocol deviations. 
 
Any loss of research data (unsigned surveys submitted by physicians or patients), or 
notes related to focus group interviews will be reported to the IRB.  
 
Any incident that could indicate the protocol adversely effects patient care or compromises 
patient privacy will prompt immediate suspension of the research.   
 
If an SAE/UAP occurs during the course of the project, a Form 119 Report of Serious Adverse 
Events and Unanticipated Problems Involving Risks to Participants or Others will be filed with 
the CIRB. Information will be collected at the time of the incident or as soon as possible after the 
occurrence of the incident is made known to the study team within the required 5 days. Any 
incident impact the health or health care of a patient subject will be reported to their PCP in 
same time frame.  
 
• 
All non-compliance with the study protocol or applicable requirements will be reported in 
accordance with VHA Handbook 1058.01. This will be achieved as follows: 
 
The study team will discuss the importance of reporting non-compliance with the 
study protocol with all site investigators and will emphasize the importance of 
reporting non-compliance to the study team. 
 
 The PI/Study Coordinator will report instances of non-compliance in accordance with 
the VHA Handbook 1058.01. 
 
If appropriate, the instance of non-compliance will be discussed between the study 
team and all site investigators to avoid duplication of non-compliance as well as to 
emphasize the importance of reporting non-compliance. 
• Include information about whether the study has a Data Monitoring 
Committee and if so, how often it will meet. 
 
Per requirement of HSR&D this project has been assigned to a VA Data 
Safety Monitoring Board (DSMB) since it is considered to be an interventional 
clinical trials. We have submitted and received approval for a Data Analysis 
Plan (DAP) to the DSMB via the ART website. The DSBM determined its own 
schedule for reviewing protocols based on VA ORS practices and policies.  
7.0 
Privacy and Confidentiality 
 
• Describe whether the study will use or disclose subjects’ Protected Health 
Information (PHI).  
Patient PHI: The study will be collecting PHI, consisting of audio recordings of 
encounters and patient email addresses, to send them links to their audio 
files. As noted above, the data analyst will extract the patient outcomes and 
demographic data from the Central Data Warehouse. He will have access to 
PHI briefly as he runs search in the Corporate Data Warehouse. 

Version#3 8/30/2022 
VA IRB Protocol  
Page 30 of 33 
 
For data tracking purposes, we will create a master spreadsheet with a row 
for each patient encounter.  A code will be assigned to each encounter linked 
to a cross-walk file (see below).  
The three patient surveys (literacy survey, exit survey, and patient activation 
measure) will be handled similarly: they will be assigned an encounter 
specific code, with a link to the cross-walk file.  
 
Provider Subject PHI: Survey and focus group data is not PHI. However it is 
possible that during an employee focus group discussion about the OAA project, that 
a clinician could mention a patient’s name or other identifier. This is not likely as the 
topic of the focus groups is the OAA project not the care of any specific individual.   
 
• Describe the steps that will be taken to secure the data (e.g., training, 
authorization of access, password protection, encryption, physical 
controls, Certificates of Confidentiality, and separation of identifiers and 
data) 
The data encryption and storage methods are highly secure,but are monitored for 
safety using standard software storage review processes, including log analysis.  
All study team members will be required to maintain necessary research training, 
and the PI of the study team will oversee completion and maintenance of any required 
training.  
 
The cross-walk file will be kept in a separate file from the master file, accessible 
only with permission from the PI with access enabled by the IT data manager. Access is 
based exclusively on need, as follows:  The data analyst who access the CDW to look 
up A1c and BP, RVA and PDC; The RA who completes the PAM 1-2 weeks post visit; 
and the RA who completes the SEGUE and completes the 4C coding, both off the audio.  
 
The research records and data will be stored and destroyed in compliance with 
record Control Schedule (RCS) 10-1. 
8.0 
Communication Plan 
• Include plan for ensuring all required local site approvals are obtained and 
notifying the Director of any facility where the research in being conducted 
but the facility is not engaged. 
Local R&D paperwork will be submitted well in advance of implementation at 
each site. Implementation at each site will not occur until R&D approvals are 
in hand from all sites. 
The PI will ensure all required local site approvals are obtained prior to the 
start of the study. 
• Include plan for keeping all engaged sites informed of changes to the 
protocol, informed consent, and HIPAA authorization 

Version#3 8/30/2022 
VA IRB Protocol  
Page 31 of 33 
 
All amendments and modifications to the protocol will be communicated to the engaged 
participating sites.  This will be achieved as follows: 
 
Distributing any revised and approved documents to site investigator at all engaged 
participating sites via email and a telephone call alerting the site investigator to the 
change.  
 
Routine phone calls between the project manager, PI/study chair, and site 
investigator will also serve as a venue to communicate changes. 
 
• Include plan for informing local sites of any Serious Adverse Events, 
Unanticipated Problems, or interim results that may impact conduct of the 
study. 
Any adverse event related to the research will be reported to the site specific IRB.  
o The study PI will report any SAEs to the site specific IRB via a memo within 48 
hours. 
o The study PI will also share any memos developed related to SAEs within 48 hours 
as well as calling a meeting between the study team and local site investigator to 
discuss the issue as well as any resulting actions that might need to take place. 
• Include plan for ensuring the study is conducted according to the IRB-
approved protocol. 
The principle investigator will be checking in monthly with each site to review the 
protocol. 
• Include plan for notifying all local facility directors and LSIs when a multi-
site study reaches the point that it no longer requires engagement of the 
local facility (e.g., all subsequent follow-up of subjects will be performed by 
the PI from another facility). 
The engagement of each facility will be essential to the protocol through nearly the 
duration of the study.   
o 
When a study site has completed the study or engagement with study team is no longer 
necessary, the study team will work with the site investigator to work through the proper 
channels at their facility to close out the work at their site. 
o 
The study team will notify the site specific IRB when local sites have completed the 
study or are no longer engaged with the study team. 
9.0 
References 
 
1. Ley P. Memory for medical information. Br J Soc Clin Psychol 1979;18:245-55. 
2. Jansen J, van Weert J, van der Meulen N, van Dulmen S, Heeren T, Bensing J. Recall in 
older cancer patients: measuring memory for medical information. Gerontologist 
2008;48:149-57. 
3. McGuire LC. Remembering what the doctor said: organization and adults' memory for 
medical information. Exp Aging Res 1996;22:403-28. 
4. Rice GE, Okun MA. Older adults' memory for important medical information. Exp Aging Res 
1991;17:90. 

Version#3 8/30/2022 
VA IRB Protocol  
Page 32 of 33 
 
5. Belyeu BM, Klein JW, Reisch LM, et al. Patients' perceptions of their doctors' notes and 
after-visit summaries: A mixed methods study of patients at safety-net clinics. Health Expect 
2018;21:485-93. 
6. Lyles CR, Gupta R, Tieu L, Fernandez A. After-visit summaries in primary care: mixed 
methods results from a literature review and stakeholder interviews. Fam Pract 2018. 
7. Neeman N, Sehgal NL. Meaningful utilization of after-visit summaries in the ambulatory 
setting. Am J Med 2015;128:828-30. 
8. Tsulukidze M, Durand MA, Barr PJ, Mead T, Elwyn G. Providing recording of clinical 
consultation to patients - a highly valued but underutilized intervention: a scoping review. 
Patient Educ Couns 2014;95:297-304. 
9. Ford S, Fallowfield L, Hall A, Lewis S. The influence of audiotapes on patient participation in 
the cancer consultation. Eur J Cancer 1995;31A:2264-9. 
10. Elwyn G, Buckman L. Should doctors encourage patients to record consultations? BMJ 
2015;350:g7645. 
11. Elwyn G, Barr PJ, Grande SW. Patients recording clinical encounters: a path to 
empowerment? Assessment by mixed methods. BMJ open 2015;5:e008566. 
12. Weiner SJ, Schwartz A. Contextual Errors in Medical Decision Making: Overlooked and 
Understudied. Acad Med 2016;91:657-62. 
13. Schwartz A, Weiner SJ, Binns-Calvey A. Comparing announced with unannounced 
standardized patients in performance assessment. Jt Comm J Qual Patient Saf 2013;39:83-
8. 
14. Weiner SJ, Schwartz A, Sharma G, et al. Patient-centered decision making and health care 
outcomes: an observational study. Ann Intern Med 2013;158:573-9. 
15. Hamilton AJ, Caskey FJ, Casula A, Inward CD, Ben-Shlomo Y. Associations with Wellbeing 
and Medication Adherence in Young Adults Receiving Kidney Replacement Therapy. Clin J 
Am Soc Nephrol 2018;13:1669-79. 
16. Kinney RL, Lemon SC, Person SD, Pagoto SL, Saczynski JS. The association between 
patient activation and medication adherence, hospitalization, and emergency room 
utilization in patients with chronic illnesses: a systematic review. Patient Educ Couns 
2015;98:545-52. 
17. Wolever RQ, Dreusicke MH. Integrative health coaching: a behavior skills approach that 
improves HbA1c and pharmacy claims-derived medication adherence. BMJ Open Diabetes 
Res Care 2016;4:e000201. 
18. Weiner SJ, Schwartz A, Weaver F, et al. Contextual errors and failures in individualizing 
patient care: a multicenter study. Ann Intern Med 2010;153:69-75. 
19. Agha Z, Lofgren RP, VanRuiswyk JV, Layde PM. Are patients at Veterans Affairs medical 
centers sicker? A comparative analysis of health status and medical resource use. Arch 
Intern Med 2000;160:3252-7. 
20. Rodriguez V, Andrade AD, Garcia-Retamero R, et al. Health literacy, numeracy, and 
graphical literacy among veterans in primary care and their effect on shared decision 
making and trust in physicians. Journal of health communication 2013;18 Suppl 1:273-89. 
21. Barr PJ, Dannenberg MD, Ganoe CH, et al. Sharing Annotated Audio Recordings of Clinic 
Visits With Patients-Development of the Open Recording Automated Logging System 
(ORALS): Study Protocol. JMIR Res Protoc 2017;6:e121. 
22. Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation 
Measure (PAM): conceptualizing and measuring activation in patients and consumers. 
Health Serv Res 2004;39:1005-26. 
23. Weiner S, Ashley  N, Binns-Calvey  A, Kelly  B, Sharma  G, Schwartz A. . Content Coding 
for Contextualization of Care. Version 6.0. Harvard Dataverse Network Project [on line]. 
December 2014. 

Version#3 8/30/2022 
VA IRB Protocol  
Page 33 of 33 
 
24. Crowe, M. Do You Know the Difference Between These Adherence Measures? Pharmacy 
Times, 
July 
05, 
2015. 
Last 
accessed 
Nov 
4, 
2018 
https://www.pharmacytimes.com/contributor/michael-crowe-pharmd-mba-csp-
fmpa/2015/07/do-you-know-the-difference-between-these-adherence-measures. 
25. Makoul G. The SEGUE Framework for teaching and assessing communication skills. Patient 
Educ Couns 2001;45:23-34. 
26. Weiner SJ, Kelly B, Ashley N, et al. Content Coding for Contextualization of Care: 
Evaluating Physician Performance at Patient-Centered Decision Making. Med Decis Making 
2013. 
27. Zill JM, Christalle E, Muller E, Harter M, Dirmaier J, Scholl I. Measurement of physician-
patient communication--a systematic review. PLoS One 2014;9:e112637. 
28. Weiner SJ, Ashley N, Binns-Calvey A, Kelly B, Sharma G, Schwartz A. Content Coding for 
Contextualization of Care: Training & Coding Manual. Harvard Dataverse Network. 
http://dvn.iq.harvard.edu/dvn/dv/4C (last accessed 11.11.2019). 
29. Weiner SJ SA, Cyrus K, Binns-Calvey A, Weaver FM, Sharma G, Yudkowsky R 
Unannounced Standardize Patient Assessment of the Roter Interaction Analysis System: 
The Challenge of Measuring Patient-Centered Communication. Journal of General Internal 
Medicine 2012:In Press. 
30. Glasgow RE, Dickinson P, Fisher L, et al. Use of RE-AIM to develop a multi-media 
facilitation tool for the patient-centered medical home. Implement Sci 2011;6:118. 
31. Weiner SJ SA, Sharma G, Binns-Calvey A, Ashley N, Kelly B, Weaver FM. Patient collected 
audio for performance assessment of the clinical encounter. Jt Comm J Qual Patient Saf 
2015;42:273-8. 
32. Hibbard JH, Greene J. What the evidence shows about patient activation: better health 
outcomes and care experiences; fewer data on costs. Health Aff (Millwood) 2013;32:207-14. 
33. Content Coding for Contextualization of Care. Dataverse Network Project. at 
http://dvn.iq.harvard.edu/dvn/dv/4C (accessed December 24, 2012).) 
34. Ross PT, Fitzgerald JT, Ravindranath D, et al. Assessing Residents' Veteran-Centered Care 
Skills in the Clinical Setting. Journal of graduate medical education 2018;10:279-84. 
35. Etingen B, Miskevics S, LaVela SL. Assessing the Associations of Patient-Reported 
Perceptions of Patient-Centered Care as Supplemental Measures of Health Care Quality in 
VA. J Gen Intern Med 2016;31 Suppl 1:10-20. 
36. Alexander JA, Hearld LR, Mittler JN, Harvey J. Patient-physician role relationships and 
patient activation among individuals with chronic illness. Health Serv Res 2012;47:1201-23. 
37. Perzynski AT, Roach MJ, Shick S, et al. Patient portals and broadband internet inequality. 
Journal of the American Medical Informatics Association : JAMIA 2017;24:927-32. 
38. Fixed 
Broadband 
Deployment 
Data 
from 
FCC 
Form 
477. 
https://www.fcc.gov/general/broadband-deployment-data-fcc-form-477. 
Last 
Accessed: 
October 30, 2019.