Evaluating the Comparative Effectiveness of Telemedicine in Primary Care: Learning From the COVID-19 Pandemic
Conditions: Asthma, Chronic Obstructive Pulmonary Disease (COPD), Congestive Heart Failure, Diabetes, Hypertension
Sponsor: Weill Medical College of Cornell University

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
1 
 
 
STUDY PROTOCOL 
Evaluating the Comparative Effectiveness of Telemedicine in Primary Care: 
Learning from the COVID-19 Pandemic 
 
 
 
ADMINISTRATIVE INFORMATION 
 
 
Title (SPIRIT 1) 
Evaluating the Comparative Effectiveness of Telemedicine in Primary Care: Learning 
from the COVID-19 Pandemic  
Trial registration (2a and 2b). 
This study will be registered on ClinicalTrials.gov. 
Protocol version (3) 
v 3.0   10.21.21  
Funding (4) 
This work was funded by a Patient-Centered Outcomes Research Institute (PCORI) 
Project Program Award (1234-56789). 
Author details (5a) 
Jessica S Ancker, MPH, PhD 
Principal Investigator 
Weill Cornell Medicine 
 
Rainu Kaushal, MD, MPH 
Principal Investigator 
Weill Cornell Medicine 
Name and contact 
information for the trial 
sponsor funder (5b) 
PCORI  
Program Officer: Penny Mohr 
Phone: (202) 627-1884 
Email: Fundedpfa@pcori.org or  
Role of sponsor {5c} 
PCORI is the funder of this study.  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
2 
 
 
 
 
STUDY PROTOCOL 
Evaluating the Comparative Effectiveness of Telemedicine in Primary Care: 
Learning from the COVID-19 Pandemic 
 
 
INTRODUCTION 
 
A. BACKGROUND AND RATIONALE 
During the COVID-19 pandemic, telemedicine has quickly emerged as the primary method of providing outpatient care 
in many regions with shelter-in-place and social distancing policies. It is critical to understand the impact of this rapid 
and widespread transition from in-person to remote visits on disparities in access to primary care, especially in chronic 
disease where ongoing communication between providers and patients is essential. Also, these newly developed or 
expanded telemedicine programs vary widely, raising important questions about the effect of these differences on 
uptake of telemedicine among different patient populations and on patient-centered outcomes. Leveraging a natural 
experiment approach, we will examine rapidly changing telemedicine and in-person models of care during and after the 
COVID-19 crisis to determine whether certain patients could safely choose to continue telemedicine or telemedicine-
supplemented care, rather than return to in-person care. Our overarching goals are to describe the features of 
telemedicine programs in primary care during the COVID-19 pandemic and to use natural experiment methods to 
provide rigorous evidence on the effects of these programs.  
 
B. RESEARCH OBJECTIVES AND DESCRIPTION OF STUDY DESIGN 
The study, which brings together an experienced multidisciplinary team with expertise in telemedicine research, 
leverages the PCORnet National Patient-Centered Clinical Research Network (CRN) infrastructure through a 
collaboration among three CRNs and two health plans representing diverse urban and rural populations. Our aims are: 
 
Aim 1: What are the features of primary care telemedicine programs newly implemented or expanded in response to 
the COVID-19 pandemic, and how is telemedicine offered to and experienced by diverse populations? In Aim 1A, we 
will conduct a comprehensive landscape survey of primary care leaders nationally, using a theory-informed survey 
instrument. Aim 1A will address an important knowledge gap on the extent and characteristics of telemedicine 
implementation during the COVID-19 pandemic. It will also provide crucial stakeholder-informed data about 
participating sites to support Aim 2. In Aim 1B, a qualitative study with patient and provider stakeholders will explore 
experiences with, barriers to, and facilitators of telemedicine and provide insights into disparities in delivery and use.  
 
Aim 2: What is the comparative effectiveness of primary care telemedicine programs and capabilities on patient-
centered outcomes and disparities in outcomes in chronic disease? In Aim 2A, we will apply natural experiment 
methods, leveraging CRN electronic health record (EHR) and claims data, to compare the effectiveness of (1) 
synchronous telemedicine alone; (2) telemedicine-supplemented in-person care; and (3) in-person care alone on 
patient-centered outcomes, including the quality of health care and the frequency of avoidable emergency department 
visits. In Aim 2B, we will examine how comparative effectiveness varies as a function of telemedicine program features 
(such as patient technical assistance and provider training). In Aim 2C, we will examine disparities in telemedicine use 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
3 
 
and impact among patient subgroups, including socioeconomically vulnerable patient populations. Patient-reported 
outcomes include satisfaction with care and telemedicine usability. 
 
This study will provide rapid, actionable guidance about the comparative effectiveness of telemedicine relative to in-
person visits for primary care chronic disease management, and about organizational implementation choices that 
support telemedicine effectiveness. Our findings will provide patients, providers, and policymakers with rigorous 
evidence about alternative models of health care delivery in a time of innovation, expansion, and development of 
telemedicine. 
 
C. AIM 1 METHODS: PARTICIPANTS, INTERVENTIONS, OUTCOMES 
Overview  
Because the COVID-related transition to telemedicine was so abrupt and universal, there is little existing evidence on 
barriers to and facilitators of telemedicine implementation in primary care from either patient or provider perspectives. 
In this phenomenological study,1 we will conduct inductive research informed by the voices of the participants, in the 
absence of relevant existing social theory. 
 
Aim 1A is a telephone survey of primary care practice leadership to collect nuanced information about sites and 
characteristics of telemedicine initiatives. The 15-minute survey will cover characteristics of the sites and of the 
telemedicine offerings, including features, implementation processes, workflows, technical support, integration of 
interpreter services, and related programs such as remote patient monitoring. Preliminary survey domains and scales 
(see Appendix A) are guided by the Triangle Model and the Penchansky and Thomas access to care framework.2,3  
 
Aim 1B is a qualitative interview study with patients and primary care providers (PCPs). Interviews will explore 
experiences, advantages and disadvantages of telemedicine, and barriers to and facilitators of telemedicine. A clinical 
research coordinator will administer informed consent and trained interviewers will conduct 30-45-minute interviews in 
English or Spanish.  
 
For both studies, the interview instrument will be finalized with input from stakeholders (see Engagement section) and 
pilot tested. The semi-structured format allows detailed, complex data to be categorized later, improves engagement to 
optimize response rate, and minimizes risk of social desirability response bias.4,5 
 
The products of Aim 1 will be: (1) a comprehensive taxonomy of types and features of telemedicine offered in response 
to the COVID-19 pandemic; (2) a nuanced understanding of the perspectives of those seeking to use telemedicine, 
including advantages and disadvantages, facilitators and barriers, and perceived value; and (3) stakeholder-informed 
input for use in Aim 2 analyses, including details of site characteristics and telemedicine interventions as well as 
outcomes prioritized by stakeholders. 
 
 
Data Sources 
 
This will be a primary data collection using telephone interviews, for which participants will receive a stipend of $40. 
 
Data collection: For both patients and providers, semi-structured interview instruments will be designed to explore 
experiences with, advantages and disadvantages of, and barriers to and facilitators of telemedicine. The interview guides 
will be finalized with input from the Stakeholder Advisory Board and pilot tested for clarity. Trained interviewers will 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
4 
 
conduct 30- to 45-minute individual interviews in English or Spanish. All interviews will be audio-recorded and 
professionally transcribed.  
 
Confidentiality and Data Management: As stated in the PCORI Methodology Standards, researchers have an obligation 
to ensure confidentiality.6 All personally identifiable data will be redacted during transcription, and the code list linking 
transcripts to participant identity and consent forms will be stored separately and accessible only to the site PI. Audio 
recordings will be stored on protected servers accessible only to the site PI and team and will be destroyed after data 
analysis is complete. 
 
Sample 
 
We will recruit both patients and providers from primary care clinics that transitioned to telemedicine to ensure 
representation from individuals who have had experience with the technology. Both PCPs and patients will help us 
understand the features of telemedicine utilized (e.g. synchronous video, telephone calls) and their relative experiences 
with these features. Our approach to sampling PCPs from different practices and purposive patient sampling serves our 
goal of understanding telemedicine experience by diverse populations. 
 
For Aim 1A, we will use two sampling frames. First, we will survey one leader at each eligible adult primary care practice 
within the CRNs participating in Aim 2 (INSIGHT, OneFlorida, and STAR). The target sample size of practices is 250 
practices. If our total sample size does not reach 250, or if we want to do targeted recruitment for specific practice 
types, we will draw additional primary care practices nationally using an Agency for Healthcare Research and Quality file 
of group practices.7  
 
For Aim 1B, we will recruit PCPs from both sampling frames with experience using or attempting to use telemedicine, 
sampling for maximum variability across practice environments (e.g., high vs. low Medicaid patient share), locations 
(e.g., urban vs. rural), and clinical roles. We will recruit adult patients (English or Spanish speaking) with a chronic 
condition under study who have attempted to use or have used telemedicine. We will recruit patients through our 
participating CRNs, a patient registry at Mount Sinai (REMIND study (HS: 15-00792) and through provider nominations. 
We will conduct purposive sampling on the basis of age, primary language, insurance type, illness severity, and location. 
The target sample size is 75 patients and 25 providers.  
 
In our inductive approach (see analysis section below), we will iteratively conduct interviews and analyze data 
until the team concurs that thematic saturation has been reached (when new interviews cease to yield new concepts).8 
In addition, during the analysis, we will also critically assess our participant sampling selection to identify additional 
participants to interview to reflect specific perspectives or viewpoints. Although this means we cannot establish a 
precise sample size a priori, we conservatively plan for 25 PCPs and 75 patient interviews. 
 
The advantage of using thematic saturation as a stopping criterion is that it allows us to avoid establishing a 
fixed sample size a priori in order to be able to respond flexibly to the data collection and analysis process.1,8 Using this 
approach, we plan to continue sampling and analyzing the data until we reach saturation. This means that we can 
increase the sample size of PCPs to more than 25 if we determine it is needed. (In addition, we can also increase or 
reduce the patient sample size, currently estimated at 75, if needed.) 
 
Outcomes and Covariates 
 
For Aim 1A, primary outcomes and covariates to be collected will be start and end dates of implementation of 
telemedicine programs at the sites, features of the telemedicine programs launched including type of platform used and 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
5 
 
availability of technical support and training, and characteristics of the practice site including ownership, pre-COVID-19 
experience with telemedicine, and type of patient population served. See Appendix A for more details. 
 
For Aim 1B, patients and providers will be asked about their experiences with telemedicine and whether it has been 
successful or unsuccessful, their attitudes toward telemedicine, and their perceptions. Covariates to be collected include 
demographics, previous experience with telemedicine and information technology, and location. 
 
 
 
D. AIM 1 METHODS: DATA COLLECTION MANAGEMENT, ANALYSIS  
 
Analytic Methods 
Aim 1A quantitative results will be summarized with descriptive statistics. Bivariate analyses will be conducted to assess 
association of telemedicine features with practice characteristics. In addition to a taxonomy based on the Triangle 
Model and the Penchansky and Thomas access to care framework,2,3 we will also explore a data-driven approach to a 
taxonomy of implemented telemedicine programs. Here, a 3-step approach will be followed applying two separate data 
reduction techniques as previously described.9 First, multiple correspondence analysis10 will be applied to study 
relationships between the (categorical) items in each of the seven conceptual domains of our survey. Where needed, 
continuous variables will be transformed into categorical variables; this will be distribution-based or based on natural 
cutoffs. In this step, for each of the seven domains, a select number of factorial axes will be retained for subsequent 
analyses. This will be based on the percentage of explained variance, interpretability, as well as visual representation of 
variables and organizations.11 In the second step, the factorial axes that result from the multiple correspondence 
analysis will be integrated into a principal component analysis. Here, principal components will be retained based on a 
scree plot (visualizing the percent variance explained by the number of principal components) using the ‘elbow 
criterion.’12 In the third step, telemedicine programs will be classified using an ascending hierarchical classification 
method.13 This group partitioning technique involves minimizing intra-class variance and maximizing inter-class variance. 
To obtain the classification, we will inspect the resulting dendrogram (which represents the hierarchy tree) and and the 
inertia quotient (interinertia/total inertia) graph. The latter typically visualizes increases in the inertia quotient (y-axis) 
with increases  in the number of classes (x-axis). Here, we will select the number of classes based on the point after 
which any increase in the inertia quotient tends to level off. In addition to statistical considerations, decisions regarding 
the number of classes to retain will also take into account the interpretability of classes9. In other words, the number of 
partitions used in the final classification will take into account both the statistical criterion (inertia quotient) and 
the theoretical/clinical plausibility of the final groupings. All analyses will be conducted using SAS v.9.4 statistical 
software (SAS Institute, Cary, NC). These analyses will be conducted and overseen by coinvestigators Dr. Jashvant Poeran 
and Dr. Iván Díaz.  
 
In Aim 1B, qualitative data will be analyzed through an iterative process of thematic content analysis.14 (Although we 
embrace an inductive analytic approach for this phenomenological study and plan to apply using the constant 
comparative method for analysis,15 the approach does not meet the definition of grounded theory15 because the end 
product is not novel social theory, but rather coherent themes that express the lived experiences of the participants. 
Therefore, it is more accurate to describe the study approach as thematic analysis.14)  
 
The qualitative leadership team will each independently read the initial 5 interviews and conduct open coding to 
begin to identify issues and concerns in relation to participants’ experiences with and beliefs about telemedicine in 
primary care. Concurrently, we will also identify emergent concepts in meetings with our stakeholder partners. Through 
regular research meetings, the qualitative team will compare these two sources of data, collaboratively construct a list 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
6 
 
of preliminary concept codes, and ensure consensus on code meanings. Then each transcript will be coded by a single 
coder using Dedoose (Version 8.0.35. SocioCultural Research Consultants, LLC, Los Angeles, CA: www.dedoose.com). To 
ensure fidelity, the coding of every 10th transcript will be reviewed by the qualitative leadership team. Themes will be 
developed through continued team meetings as researchers conduct axial coding to develop the relationships between 
and inferences from the concept codes into themes. Through this iterative process, we will continue to update the 
coding scheme if new concepts emerge and determine whether new codes suggest the need for updating or enriching 
the emergent themes. We will iteratively reframe our semi-structured interview guide as needed to elaborate questions 
that address barriers to and facilitators of telemedicine implementation in primary care. 
The credentials and experience of the qualitative researchers are relevant to this project.6 We have assembled a 
team with extensive experience with qualitative work from a phenomenological perspective. Dr. Horowitz was trained in 
qualitative research by medical anthropologists as a fellow at the University of Washington and has conducted 
numerous qualitative studies involving individual interviews, focus groups, photovoice, and document analysis.16-19 She 
taught the qualitative research course at Mount Sinai for 5 years, and developed and taught a qualitative research 
course for community partners. Dr. Benda is a human factors engineer working in health care whose qualitative research 
has examined treatment of ED patients with limited English proficiency, provider perspectives on decision support, and 
health professional/patient use of novel technologies. Dr. Lin is a primary care physician experienced with caring for 
patients with chronic conditions, who has also published qualitative research on patient-provider communication in 
cancer. PI Dr. Ancker, an informaticist and biostatistician whose K award provided an opportunity to cross-train in 
human factors and qualitative research, has worked extensively with both quantitative and qualitative methods, in 
particular in studies of how patients and providers adapt to new information technologies.20-23 
 
Trustworthiness and credibility: To ensure trustworthiness and fidelity,6,24 all recruitment, data collection, analysis 
meetings, and writing will be documented to create an audit trail. To maximize credibility,6,24 we will rely upon multiple 
researchers to make decisions about coding and interpretation, and engage in “persistent observation” (in other words, 
constant rereading of the data, and multiple discussions and revisions of the concepts). In addition, to maximize 
credibility, after the team has agreed that thematic saturation has been reached, we will conduct member checks25 with 
(a) the stakeholder partners and (b) a subset of interview participants who have agreed to be contacted for follow-up. In 
this member check, we will present summarized results, themes, and quotes for verification. If the member check 
suggests misinterpretations of participant experience or gaps in the data, we will return to the data collection and 
analysis phase of the project, until we reach a point at which a member check no longer suggests the need for new data 
collection and analysis. 
 
Primary Analysis 
 
Our primary analysis will focus on developing a nuanced understanding of patient and provider experiences with and 
perspectives on telemedicine, particularly in the context of the rapid transition to telemedicine during the COVID 
pandemic. 
 
Secondary Analyses 
 
Although our primary analysis will focus on themes common to the entire group of interviewees, we anticipate that 
several subgroups may be large enough to produce thematic saturation. Our secondary analyses will therefore focus on 
similarities and differences in perspectives across large subgroups of interest, specifically, patients versus providers. It is 
also likely that we will be able to examine differences across other relatively large subgroups. Examples may include 
older patients versus younger patients, patients with more severe illness versus patients with less severe illness, 
different insurance coverage groups, or primary language, provided the sample sizes are sufficient.  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
7 
 
 
 
Limitations 
 
Power 
 
With qualitative research, study power is not formally computed. Instead, the iterative process of recruiting and 
analyzing allows us to continue recruiting until we achieve thematic saturation, i.e., failure to elicit new thematic 
information from new interviews. This approach is a widely accepted method of establishing sufficient sample size. 
Nevertheless, it is possible with any qualitative project that recruiting more patients would broaden the project and 
allow us to identify more themes. 
 
Missingness 
 
With qualitative research, participants are free to direct the interview to a certain extent, including skipping certain 
questions. This means it is likely that we will not collect precisely the same set of responses from all participants. We will 
administer a brief demographic questionnaire to all participants to try to encourage self-report of key information. 
Nevertheless, it is possible that certain pieces of information will be missing for certain participants. If necessary, the 
iterative sampling and data analysis approach would allow us to continue recruiting if it appears that missing data for 
certain participants would threaten our ability to reach thematic saturation within certain subgroups. 
 
Other 
 
Qualitative research is by definition strongly grounded in the experiences and perspectives of the participating 
individuals and communities. There is always the possibility that themes and findings found within one community 
would be different from those found in a different community. As a result, the term “generalizability” is not used of 
qualitative research. Instead, qualitative researchers promote “transferability,” the generation of findings that could 
potentially be applied to other settings, by carefully describing not only the findings but also the context in detail, so 
that the results are meaningful, and their applicability or lack of applicability to other settings can be determined. 
 
 
E. AIM 2 METHODS: PARTICIPANTS, INTERVENTIONS, OUTCOMES 
Overview 
 
Using information about practices collected during Aim 1, together with EHR and claims data, we will conduct a 
longitudinal retrospective cohort study to assess the comparative effectiveness of new or expanded telemedicine 
services on important patient-centered outcomes including unnecessary emergency department care.  
 
Study Setting 
 
Primary care practices affected by the COVID-19 pandemic across four health systems in the INSIGHT (Mount Sinai 
Health System and Weill Cornell Medicine), OneFlorida (University of Florida Health), and STAR (University of North 
Carolina Health) CRNs.  
 
Eligibility Criteria  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
8 
 
 
We will include primary care practices. Primary care is defined as practices in the fields of general practice, family 
practice, (ambulatory) internal medicine, preventive, and geriatric medicine. (Our primary care definition has been 
adapted from the Medicare specialty designation with some adjustments. For the purposes of this study, hospice and 
palliative care, and osteopathic medicine have been excluded due to lack of chronic disease management, and pediatric 
medicine has been excluded due to target adult population in this study.) Women’s and men’s health clinics will be 
included on a case-by-case basis if they are staffed primarily by internists, rather than obstetrician-gynecologists or 
urologists. 
 
The study population encompasses patients that are attributed to primary care clinics in one of the four health systems 
defined above. Patients are included in the study if they are ages 19 or older and received two or more outpatient visits 
at a participating practice during a one-year period before the COVID-19 pandemic, and had one or more of five chronic 
illnesses (asthma, chronic obstructive pulmonary disease (COPD), congestive heart failure (CHF), diabetes, hypertension) 
as defined by the Medicare Chronic Conditions Warehouse algorithm.27 For the claims analyses, we additionally require 
that patients are continuously enrolled over the entire study time period. 
 
Interventions and Comparators 
 
We will use natural variation in the availability of in-person and telemedicine visits across clinics over time during the 
pandemic in a difference-in-differences framework to compare the clinical effectiveness of three care models: (1) 
telemedicine-only; (2) telemedicine-supplemented in-person visits; and (3) in-person visits only.  
 
While these arms will be defined at onset of the pandemic, clinics may not remain in the same arm over time. We will 
evaluate changes in total telemedicine and in-person visits in each month and capture the status of each clinic by month 
for secondary analyses. In additional secondary analyses, we will explore heterogeneity in effects across patient and 
organizational subgroups.  
 
The intervention population encompasses patients that are attributed to primary care clinics that switched to delivering 
telemedicine-only visits as their primary model of care delivery during the pandemic. We will compare this population to 
patients that are attributed to clinics that delivered telemedicine-supplemented in-person visits or in-person visits only 
during the pandemic. We will define the clinics’ primary model of care delivery, and the timing of their switch in clinical 
modality, using the survey results from Aim 1.  
 
Our decision to make the clinic the primary unit of analysis is because of the known sociodemographic and clinical 
differences in patients (pre-pandemic) who chose to use telemedicine. Patients that use telemedicine for instance, tend 
to be younger and are more likely to be female than patients that used other sites of care.28 Therefore, in our primary 
analysis, we will exploit the change in availability for telemedicine services at the clinic level as an independent 
determinant of telemedicine use, leveraging an intent-to-treat approach to evaluate the association between clinic-level 
telemedicine availability and patient outcomes. We will further improve the internal validity of the study by matching 
clinics using clinic-level characteristics. This is further described in the analytic methods below.  
 
Outcomes  
 
We will analyze measures of health care utilization and quality from Medicare and commercial (Anthem and Humana) 
claims and CRN EHR data. Our primary outcome measures include avoidable emergency department (ED) admissions, 
potentially preventable hospital admissions, and continuity of care.2,3 Secondary measures will include evidence of 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
9 
 
controlled disease (as indicated by condition-specific measures endorsed by the National Quality Forum (NQF) or HEDIS) 
and days at home. A list of outcome measures is included in Table 1 below. These primary and secondary outcome 
measures will be constructed at the patient-month level, and we will examine 30-day, 60-day, 6-month and one-year 
outcomes where data are available. We have operationalized the majority of these measures through a previous PCORI 
health system demonstration grant, in collaboration with OneFlorida.  
 
Additionally, we will collect patient-reported outcomes (PROs) through REDCap surveys of 1,000 patients at Weill Cornell 
Medicine, delivered via the patient portal. The recruitment methods for this survey include, but are not limited to, the 
use of the Consent to be Contacted for Research (CCR) (WCM IRB protocol 1812019822; PI: Thomas Campion). For those 
patients who have opted into CCR participation and meet the inclusion criteria for this study, the study team will send 
recruitment message(s) via approved methods in accordance with CUIMC-NYP-WCM governance.  By requesting CCR 
status for patients meeting study eligibility criteria, the study team agrees to contact only patients approved by the 
Institutional Review Board (IRB) for this specific study. The study team agrees to adhere to EpicTogether and the 
Tripartite processes. Contingent on the Tripartite committee(s)'s decision, all the relevant materials will be submitted in 
subsequent submissions to the IRB. The recruitment methods for patients who are approved to be contacted include, 
but not limited to, use of Epic’s research recruitment message, phone call, email, and/or mail. The study team also 
agrees to maintain a record of patients contacted for, enrolled in, declined for, and withdrawn from the study, which the 
team will share at regular intervals with TRAC and the IRB. For patients who have not consented to be contacted for 
research, the study team will pursue the alternative methods of recruitment approved by the IRB office. Additional 
recruitment methods for the PRO survey include inviting patients who were recruited for the qualitative interviews, or 
patients who are nominated by their primary care providers. Invitations will be sent to patients via email, phone call 
and/or mail.  
 
We will collect PROs on patient experiences based on the Patient Satisfaction Questionnaire (PSQ-18) (5-point 
subscale),29 which includes patient satisfaction, communication quality with providers, and accessibility/convenience of 
care. For individuals who accessed a telemedicine visit, we will ask questions adapted from the validated Telemedicine 
Usability Questionnaire (TUQ), including the ease of use and access to the telemedicine service, quality of the 
interaction with the provider, and satisfaction.5 We will leverage stakeholder input and the Aim 1A survey to inform the 
duration of patient exposure to each of the three treatment arms.  
 
 
Table 1 
Measure 
Data Source 
PRIMARY OUTCOMES 
 
• 
Avoidable ED Admissions30 
Claims 
• 
Potentially Preventable Hospital Admissions*31,32 
Claims 
• 
Continuity of Care 
 
o 
Bice-Boxerman Continuity of Care Index 
Claims 
o 
Breslau Usual Provider of Care 
Claims 
o 
Attendance at Follow-up Appointment 
EHR 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
10 
 
• 
Patient-Reported Outcomes 
REDCap 
SECONDARY OUTCOMES 
 
• 
Days at Home (days per month not in hospital or institutional setting) 
Claims 
• 
Evidence of controlled disease** 
EHR/Claims 
*Potentially preventable hospital admissions are defined as hospital admissions for ambulatory care-sensitive conditions, which can be treated with primary care  
**In subset analyses for which we have a consistent panel of EHR data, we will assess condition-specific measures of disease control, including measures accredited by the NQF or included in 
HEDIS. These may include laboratory measurements, vitals, or medication use. The choice of these measures will be informed by provider and patient participants in our Accelerator 
(Stakeholder Board) in order to ensure we select outcomes that are important to stakeholders. 
 
Participant Timeline 
 
The main outcomes of interest will be assessed 30-days, 60-days, 6 months, and 1 year after the exposure to one of the 
comparator arms of clinic-level telemedicine use. Our study will include claims data for 2017-Q1 2021 and CRN EHR data 
for 2017-Q1 2021. 
 
Sample Size  
 
The estimated breakdown of the number of primary care patients in each of the four participating health systems by 
race and ethnicity is shown in Table 2. 
 
Table 2 
Part A. Baseline estimates of demographics of primary care patients from 4 health systems  
Ethnic Category  
Females  
Males 
Sex/Gender 
Unknown 
or Not 
reported  
Total  
Hispanic  
6772 
3475 
0 
10247 
Not Hispanic  
89657 
59527 
2 
149186 
Unknown (individuals not reporting ethnicity)   
26863 
19569 
2 
46434 
Ethnic Category: Total of All Subjects 
123292 
82571 
4 
205867 
 
Racial Categories  
American Indian 
120 
78 
0 
198 
Asian  
5733 
3736 
0 
9469 
Native Hawaiian or Other Pacific Islander 
184 
93 
0 
277 
Black or African American 
22084 
11060 
0 
33144 
White 
60479 
44323 
0 
104802 
More than one race 
6236 
3248 
0 
9484 
Unknown or not reported  
28456 
20033 
4 
48493 
Racial Categories: Total of All Subjects  
123292 
82571 
4 
205867 
Part B. Primary care patients from 4 health systems with Hispanic/Latino ethnicity 
Racial Categories  
Females 
Males 
Sex/Gender 
Unknown 
Total  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
11 
 
or Not 
reported 
American Indian  
23 
12 
0 
35 
Asian 
41 
29 
0 
70 
Native Hawaiian or Other Pacific Islander 
23 
13 
0 
36 
Black or African American 
190 
79 
0 
269 
White  
1547 
887 
0 
2434 
More than one race 
2260 
973 
0 
3233 
Unknown or not reported  
2688 
1482 
0 
4170 
Racial Categories:  Total of Hispanics or Latinos 
6772 
3475 
0 
10247 
*Estimates are counts of ambulatory patients seen at the 4 sites since January 1 2020; due to time constraints, this table includes some patients 
seen in specialty settings who will not be included in the final analysis. 
 
An additional PCORI deliverable for this project (scheduled for 2021) is to provide a more detailed estimate of patients 
meeting our eligibility criteria and covered by the two commercial insurers.  
 
 
 
Power Calculation 
 
Across participating sites, we estimate about 205,000 patients in primary care with one or more of the five conditions 
(see tables above). For the ED visit outcome, this sample size provides 93% power to detect a 1% difference between 
the treatment and comparison arms in a two-tailed test with an alpha of 0.05.  
 
For comparison, a recent study evaluated the impact of a Veterans Health Administration care coordination program 
enhanced with home telemedicine on ED use and other outcomes for veterans with diabetes.6 In this study of 537 
patients, 69.9% of patients in the pre-telemedicine period had an ED visit, while only 62.45% of patients in the post-
period incurred an ED visit, an 11% relative difference (p = .04). We believe that this is a clinically meaningful change in 
ED use before and after the implementation of a care coordination program targeted at a specific chronic care condition, 
and that it is feasible to aim for a difference of similar magnitude. Although there are few other highly relevant studies 
to draw from, we also refer to a 2013 systematic review of non-ED interventions intended to reduce ED utilization. 
Across 9 studies, the interventions were associated with ED utilization decreases from 9% to 54%.7 We do note that the 
intervention studied in Evaluation & the Health Professions6 involved the addition of a telemedicine care coordination 
program to existing care. As such, it is most relevant to the contrast between the telemedicine-supplemented in-person 
arm and the in-person-only arms of our study. It may be less relevant to our telemedicine-only arms. Similarly, the 
intervention studied in JAMA Internal Medicine33,3433,3433,3433,3433,3433,3435, 3636 involved an outpatient program that 
augmented, rather than replaced, standard primary care, and may be most relevant to the telemedicine-supplemented 
in-person arm and the in-person only arms of our study. 
 
For the patient satisfaction outcome, with our planned survey sample size of patients with patient-reported outcomes (n 
= 1,000), we have 93% power to detect a 5% difference in patient satisfaction between the treatment and comparison 
arms in a two-tailed test with an alpha of 0.05. Based on an RCT examining the impact of an intensive outpatient care 
program on patient-reported satisfaction among a cohort of high-need patients, we need sample size of 206 patients to 
provide 80% power to detect a 0.26 difference in patient satisfaction (effect size determined by the study) using a 2-
tailed hypothesis at an α level of .05.34 Our sample size of 1,000 patients will provide greater than 80% power to detect 
this difference. The 0.26 difference in patient satisfaction was derived from a recent research study evaluating the 
impact of a medical home program on patient-reported satisfaction.8 The measure of patient satisfaction was based on a 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
12 
 
5-point scale measuring patients’ general satisfaction with care, the accessibility/convenience of care, and perceived 
quality of communication with their provider. Among patients who participated in both baseline and follow-up surveys, 
there was a statistically significant increase in mean self-reported satisfaction with the augmented medical home 
program compared to the standard primary care program by 0.26 points (2.90 to 3.16; P = .04). Although this program 
did not utilize any telemedicine components, we consider it reasonable to expect comparable changes in patient 
satisfaction among programs that have a virtual care management component that promotes frequent contact between 
patients and providers and allows patients to receive care in their own homes. Of note, a recent review article reports 
that in nearly all studies, both patients and clinicians reported high satisfaction with telemedicine visits, and either no 
difference between virtual and office visits in terms of the overall quality of the visit, or higher satisfaction with 
telemedicine visits compared to in-person only care.35 Additionally, very recent work on the first 4 weeks of response to 
the COVID-19 crisis, which assessed patient satisfaction of telemedicine visits using a 5-point scale, found that 67% of 
patients who had a video or phone visit reported that the visits were “good/better” than regular in-person visits, and 
96% reported that they were “somewhat/very satisfied with medical care” they received.36  
 
 
Our sample sizes still provide sufficient power should we adopt a non-inferiority framework to test whether 
telemedicine visits are no less effective than in-person visits. For the ED visit outcome, if we assume that the minimum 
sample size is 25,000 in each of the three groups, and that between 20% and 5% of patients in the in-person arm have 
unplanned ED visits, we will have between 80% and 99% power to determine whether the telemedicine arm is no worse 
than 1 percentage point (absolute) lower than the in-person comparator. (This calculation was done with a z-test to 
determine a difference in proportions with a one-sided 2.5% significance level.) In fact, our study will have more power 
if the groups have more than 25,000 patients. For the patient satisfaction outcome, with 500 in each group, there will be 
90% power with one-sided level of significance of 0.025 to determine if the decrease in mean level of patient satisfaction 
for the telemedicine group is no more than 0.2 standard deviations compared to the control group (in-person). We 
believe a 0.2 standard deviation effect size is reasonable for a non-inferiority margin because it lies below the 0.25 effect 
size that is typically deemed meaningful for patient satisfaction and quality of life instruments. 
 
F. AIM 2 METHODS: DATA COLLECTION, MANAGEMENT, ANALYSIS 
 
Data Collection Methods and Data Sources 
 
Our primary analysis will be the Medicare claims data analysis. We anticipate this will be the most robust analysis 
because of the large sample size (relative to the commercial claims data) and ability to comprehensively track patients 
across care settings (relative to the CRN EHR data). We will obtain 100% of 2017 through Q1 2021 Medicare fee-for-
service claims data for the three states in which CRN-participating research sites are located (FL, NC, and NY). Therefore, 
we will able to follow patients regardless of where they seek care, which will improve patient attribution, as well as 
capture the entirety of a patient’s follow-up care for the utilization, continuity of care, and quality of care measures. The 
Medicare data will be accessed via the Centers for Medicare and Medicaid Services Virtual Research Data Center (VRDC), 
because (in contrast to physical copies of the data), the VRDC allows for data to be used for analysis with little lag 
(approximately three months following claims run-out). Use of the VRDC will enable the research team to include up to 
11 months of data following the initial apex of the pandemic in April 2020. The patient attribution process will be 
separately conducted for each enrollee population. 
 
Unfortunately, it will not be feasible to use federal (T-MSIS) Medicaid claims data for this project. The T-MSIS data are 
currently running at a 2-year lag, meaning we would not be able to obtain the data for much of the period of interest 
until after the grant had concluded.  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
13 
 
 
In the secondary analysis, we will replicate the Medicare analysis using commercial claims data for commercially insured 
populations. Commercial claims data will be analyzed for Anthem and Humana commercial enrollees at the NY, FL, and 
NC sites for 2017 through Q1 2021, using the same analytic approach.  
 
An additional secondary analysis will examine measures of disease control (see Table 1), utilization, and continuity of 
care drawn from the CRN data. Measures of disease control require EHR data that in general are not available from the 
commercial claims providers. In addition, the CRN data are all-payer. However, the main limitation is that if patients 
seek care outside of the CRN, we will not capture their utilization within CRN data. Therefore it would not be 
appropriate to seek to extract the ED or hospital admission data from this source. 
 
*After submission of this grant, the three participating CRNs agreed to begin collecting data that would allow us to 
distinguish between telemedicine and in person visits, an item that was not available when this study was designed. This 
may enable us to conduct a separate analysis of telemedicine versus non-telemedicine visits at the visit level. 
 
Finally, we will collect PROs using REDCap surveys through the WCM REDCap database. REDCap is a secure website for 
building and managing online surveys and databases. It is available at each site and harmonized across institutions to 
facilitate data storage and management. REDCap will generate automated emails or text messages to patients 
containing hyperlinks to access surveys. The REDCap team will be responsible for maintenance during the study period. 
Table 4 summaries the primary data sources and data elements. 
 
 
Table 4 
Data Sources 
Data Elements 
Medicare, Commercial claims, CRN structured EHR data 
Diagnosis codes 
Medicare, Commercial claims, CRN structured EHR data 
Procedure codes 
Medicare, Commercial claims, CRN structured EHR data 
Demographic characteristics 
Medicare, Commercial claims, CRN structured EHR data 
Health care utilization 
Commercial claims, CRN structured EHR data 
Laboratory testing results 
Medicare, Commercial claims, CRN structured EHR data 
Medications 
REDCap surveys 
Patient experience and satisfaction 
 
Missing data: We will assess the data source adequacy and patterns of missing data in compliance with PCORI 
Methodology Standards for preventing and handling missing data. More specifically, we will report the reasons for 
which missing data occur, which may include but are not limited to incomplete data on demographic characteristics in 
both the clinical and claims data, incomplete utilization and clinical data in the CRN data, and nonresponse to the 
REDCap surveys. We will also report the percentage of missing data in each of our data sources. We believe that there 
will be limited missing data among Medicare beneficiaries due to the completeness of Medicare FFS claims. Among 
commercial claims, there is also unlikely to be missing data among patients. Due to the low probability of missing data, 
we will use listwise deletion to deal with missing data among these data sources. Among the clinical data and PRO data 
sources, we will use validated statistical methods to address missing data, including multiple imputation and likelihood-
based methods. Among all data sources, we will conduct sensitivity analyses to evaluate changes in the study sample 
and outcomes with and without missing data.  
 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
14 
 
Data linkage: Decisions around the analytic plan for this study are informed by the need for expediency to meet all 
deliverables in this rapid-turnaround, 2-year project. To ensure the primary analysis can be completed with up-to-date 
data covering the pandemic period, we must use the Medicare VRDC to access Medicare data. If the CRNs are able to 
provide high-quality Medicare identifiers for the included cohort (SSN, MBI, HIC, Name, DOB, Zip code, or other), it will 
be possible to link the measures of disease control derived from clinical laboratory testing data from the CRNs to the 
Medicare data set so that the secondary analysis can be completed with the same analytic files. To accomplish the data 
linkage each study site (WCM, UNC, MS, and UF) will submit a finder file with identifiers to HealthAPT (General Dynamics 
Information Technology), per CMS’s 'Submission of Medicare Data Finder and Crosswalk Files‘ guidelines. In parallel, 
investigators from WCM will transmit the de-identified CRN data with clinical measures from Datacore to the VRDC, so 
that CMS can perform the crosswalk, merge the datasets, and discard the identifiers. The collaborating institutions and 
investigators will not have access to these identifiers in their analysis. 
 
However, even if such identifiers can be provided by all CRNs, previous experience with this data suggests that up to 
15% or more of each sample will fail to match either because of invalid identifiers or because of lack of overlap between 
the samples. This has the potential to introduce bias, as well as to decrease sample size. We will therefore conduct 
quality control on any provided identifiers before using them for the secondary analysis; large amounts of missingness 
or errors may mean that it will be more rigorous to conduct the secondary analysis separately on a CRN data set.  
 
Data Management Plan 
 
Data Collection: See section above for Data Collection description. Because of the need for expediency in this 2-your 
project, data queries for CRN and claims data will be conducted across distributed partners. HealthCore, a wholly-owned 
and independent outcome research subisidary of Anthem, will be responsible for querying and analyzing Anthem data 
using the same methodology developed by the WCM team for analysis of the CRN and Medicare data. A similar 
distributed data model will be used by the Humana team for analysis of the Humana claims data. 
 
Data Organization: All final analytic files for CRN and claims data analyses will be organized as patient-month records 
and generated and stored as comma-separated values files. We will collect PROs through survey data using RedCAP, a 
HIPAA-compliant IRB-approved platform. For the PRO analysis, analytic files will be organized as patient-encounter 
records and stored as comma-separated values files. 
 
Data Handling: Medicare claims data will be accessed within the VRDC, and will be accessed and managed according to 
the Medicare specifications described in the DUA. Only 4 analysts will have access to VRDC. The VRDC will handle data 
backups. Commercial claims data will be handled within the commercial entity, with specified users described in the 
DUA.  
 
CRN data and PRO data will both be handled within the Weill Cornell Data Core, a highly secure data repository and 
analytic environment maintained by Weill Cornell Information Technology Services. This restricted-access data center 
that meets the security requirements for PHI and other data designated as restricted access. Data Core’s computing 
system belongs to a Windows domain that exceeds the U.S. Defense Department’s C2 standards for trusted computing 
environments. Access is limited to authorized users who have signed a DUA with the Data Provider and a Computing 
System User Agreement with HPR Data Core, as confirmed by the WCM Office of Sponsored Research Administration 
and IRB. Data is backed up daily.  
 
For all data sources, privacy and confidentiality will be maintained in accordance with IRB requirements. To limit risk, 
analyses will be performed on limited data sets, including no individually identifying information other than dates.  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
15 
 
 
Data Description: We will utilize existing PCORnet CDM and Medicare LDS data dictionaries. A separate data dictionary 
will be developed for the PRO data. For in-house data sets, metadata will be documented within the Weill Cornell Data 
Catalog, which utilizes the elements from Dublin Core, with additional elements added from the following standards: 
• 
Common metadata elements for Cataloging Biomedical Datasets 
• 
DataCite Metadata 
• 
Dryad Metadata 
• 
W3C Data Catalog Vocabulary 
• 
NIH BioCADDIE Metadata v1 
This data catalog is maintained by the Weill Cornell’s Samuel J. Wood Library and C.V. Starr Biomedical Information 
Center, which assists researchers in recording appropriate metadata needed for possible sharing and reuse of the data. 
 
Data Maintenance: Raw data sources and analytic files used for this analysis will be maintained within the Data Core 
and/or within the secure computing environments of partner institutions that originally sourced the data for two years 
following conclusion of the project, at which point they will be destroyed.  
 
Data Sharing: These analyses will use PHI, so the researchers will not share any data that is at the patient-level or 
patient-month level. 
 
 
 
 
Data and Safety Monitoring Plan 
 
A detailed Data and Safety Monitoring Plan (DSMP) will be submitted separately in an upcoming deliverable (12/21/20) 
in accordance with PCORI Policy on Data and Safety Monitoring Plans for PCORI-Funded Research.  
 
Standards for Data Networks as Research Facilitating Structures 
 
The study leverages the PCORnet National Patient-Centered Clinical Research Network (CRN) infrastructure through a 
collaboration among three CRNs to compare telemedicine’s effectiveness on patient-centered outcomes, including the 
quality of health care and the frequency of avoidable emergency department visits and complications.  
DN-1: Requirements for the design and features of data networks: The INSIGHT and OneFloridaCRNs require 
that affiliated sites structure their data according to the Observational Medical Outcomes Partnership (OMOP) data 
model, and transfer that data via Secure File Transfer to secure Data Centers managed by the CRN coordinating centers. 
Data from affiliated sites are transformed to fit the requirements of the PCORnet Common Data Model (CDM), which 
conducts nationally standardized data checks to ensure the validity and integrity of the data. The STAR CRN is a 
distributed network: each member institution maintains a local datamart meeting the PCORnet CDM specifications. 
While STAR does not house data within one central site, data may be transferred on a project by project basis via secure 
file transfer protocol or other secure mechanism acceptable to both parties. PCORnet utilizes the privacy-preserving 
record linkage solution provided by Datavant, providing software for each site that is installed locally to generate 
encrypted hash tokens based on personally-identifiable information held within local source systems, and then linking 
and standardizing patient data across health systems while protecting patient level identifiers.  

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
16 
 
The INSIGHT CRN leverages a rigorous set of privacy and security protocols that adhere to the HIPAA privacy 
rules and definitions of “Safe Harbor” and “Expert Determination” to undergird its limited, centralized database. A 
national expert has certified INSIGHT as meeting strong federal privacy and security requirements, and INSIGHT regularly 
undergoes audits to ensure compliance with NIST 800-66 and HIPAA standards. The INSIGHT CRN database is managed 
in the AWS cloud. OneFlorida makes use of a dedicated, secure ‘bubble’ for all project data. All access requires an 
encrypted connection over VPN and through the use of a VDI. STAR CRN institutions abide by local institutional security 
policies. At UNC-Chapel Hill (the STAR site participating in this project), data are housed as a HIPAA-limited dataset on a 
secure server and are only accessible to designated honest brokers. Each request for data requires an IRB approval or 
exemption designation; data request; review by a local data request review committee; and, if data is to be shared 
externally, a data use agreement. An honest broker extracts data on a project by project basis. Data are provisioned to 
local study teams on a secure file server restricted to the UNC VPN, or are securely transferred to approved, external 
parties.  
INSIGHT, OneFlorida and STAR require researchers to follow policies and procedures for privacy and security. 
Researchers and project staff are trained regarding privacy and security, and data is only held on servers approved for 
holding PHI. Researchers and their institution are required to sign agreements which bind them to the CRN’s policies and 
procedures outlining requirements around data sharing, prohibiting re-identifying the data, requiring secure storage and 
disposal of the data, and requiring the reporting of breaches and security incidents. The CRNs maintain data dictionaries 
that are available to all researchers, which use a standardized terminology system to communicate the specific data 
elements and data structure contained in their respective databases. A PCORnet wide data dictionary annotating 
metadata within the CRNs is available to researchers as well. Consistent terminology used includes ICD-9 and ICD-10 
codes for diagnoses, and LOINC codes for laboratory results among other commonly used coding terminologies in 
healthcare research. The Patient Centered Outcomes Research Network, of which the NYC INSIGHT network, OneFlorida 
and STAR are all members, is well-positioned to address our research questions through networks that have a proven 
track record of success in integrating large data sets from disparate institutions into a common data model suitable for 
developing and disseminating queries to be run at sites across the country.  
DN-2: Selection and use of data networks: We have chosen to work with 3 established PCORI CRNs: INSIGHT, 
STAR, and One Florida. These networks represent very different patient populations, ranging from rural to urban, with 
great socioeconomic, racial, and ethnic diversity. Two of these networks (INSIGHT and One Florida in particular) have a 
track record of working together. 
Very few networks have complete capture of the necessary variables and outcomes of interest within their 
Common Data Models for their patient populations. EHR data might be incomplete for several reasons: patients have 
ambulatory and inpatient encounters from health systems not affiliated with the CRNs, the post-acute care encounters 
(eg., hospice and home health) may not be covered by the EHR data. Since this project focuses on patient access to care 
and avoidance of unnecessary care and complications, it is of great importance to use data with complete patient 
utilization across multiple health systems, care settings, and geographic regions. We will therefore supplement the EHR 
data from CRNs with commercial and Medicare claims data. Claims data captures patient information as long as a 
patient is insured, regardless of the care settings and geographic regions. Using both EHR and insurance claims data 
therefore provided more complete and high-quality information for this project.  
 
 
Statistical Methods 
 
Causal Model and Overall Hypothesis 
 
As described in more detail earlier, our population of interest is primary care patients with chronic medical conditions. 
The intervention of interest is the switch to primary care telemedicine prompted by the COVID epidemic. The 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
17 
 
comparators are in-person care, telemedicine primary care, and telemedicine-supplemented in-person care; whether a 
practice is within any of these comparative groups may be time varying. The primary outcomes (described in more detail 
above) are unnecessary utilization outcomes listed above. The timing of the outcomes will be assessed at 30 days, 60 
days, six months, and 12 months. The settings are primary care practices within our collaborating institutions. 
 
We propose a causal model informed by the Wagner Chronic Care Model in which productive disease management 
interactions between informed patients and prepared care teams are promoted by effective health systems as well as 
communities.37 Critical relevant elements of the health system are processes to provide self-management support, 
design of the healthcare delivery system, and technologies such as clinical information systems and decision-support. In 
our causal model, for patients with chronic disease, access to care is a mediator in the causal pathway between the 
disruption in care from the pandemic and the outcomes in our study (e.g., unnecessary utilization such as unplanned ED 
visits, and satisfaction with care). To the extent that telemedicine improves access to care, it mediates this relationship. 
Furthermore, patient-provider communication is an additional mediator of this relationship, through its effect on self-
management support as well as trust in the healthcare system.  
 
However, one reason this study is important is that the abrupt nationwide transition makes it challenging to determine 
in advance whether telemedicine would be expected to have positive or negative impacts on care access and patient-
provider communication, overall and within important patient subgroups. The pandemic prompted telemedicine use by 
patients who would otherwise not have been early adopters of telemedicine, increasing the diversity of patient users, 
including those from disadvantaged social circumstances or with older age who may encounter more barriers to 
successful use of telemedicine (stemming from limited device access, technology literacy, or broadband access, or 
primary language). These barriers would cause telemedicine to have a negative effect on care access and patient-
provider communication, as well as patient satisfaction. Conversely, however, there are known instances in which 
people who would not have opted into the technology themselves nevertheless become effective users of the 
technology after being provided with an initial access experience.26 Thus, the forced use of telemedicine may possibly 
have successfully promoted engagement with telemedicine and therefore improved access to care and patient-providers 
communication among some patients from less advantaged patient groups.  
 
Another complexity making it challenging to predict the directionality of the effect of the telemedicine transition is that 
the rapid nationwide transition appears to have led to diverse types of telemedicine programs, ranging from relatively 
mature programs with well-established technology and user support, to rapidly implemented programs with less well-
established technology or support. This heterogeneity may influence whether telemedicine was effective in improving 
access to care or patient-provider communication. It is well-established that innovations that are not highly usable, well-
matched to user needs, and supported with organizational support are unlikely to promote their desired outcomes.38,39 
 
In light of the complexities described above in the causal model, we propose a nondirectional overall hypothesis that 
telemedicine-only, telemedicine-supplemented in-person visits, and in-person only groups will have different healthcare 
utilization and patient satisfaction outcomes. 
 
Primary Analysis 
 
Our causal model will leverage variation over time and across sites in the availability of telemedicine visits to assess the 
impact of a change in clinic-level availability in telemedicine on patient-level outcomes. We will use a generalized 
version of the difference-in-differences methodology (also known as an event study design) in an intent-to-treat 
framework to compare patients receiving care at clinics that have switched to telemedicine-only visits at a given time 
with patients at comparable clinics that have not or that used a mixture of telemedicine and in person care. This event 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
18 
 
study design allows for multiple treatment periods and for the relationship between treatment and outcomes to vary 
over time.  
 
The causal identifying assumption in difference-in-differences models is that outcomes in each comparator arm would 
have followed the same trends in the absence of exposure to telemedicine, after the onset of the pandemic.40 While we 
cannot directly test this assumption, we can test whether there significantly different trends in outcomes during the 
months prior to the onset of the pandemic. We expect that this parallel trends assumption may hold in our study, 
because the transition to telemedicine during the pandemic was rapidly implemented and potentially unrelated to 
patient outcomes and the underlying quality of the clinic.  
 
In order to further improve the comparability of clinics across each of the treatment arms, we will use propensity scores 
to match clinics based on clinic-level characteristics, including rurality; size; mean clinical risk (HCC score, Elixhauser 
comorbidity index); and proportions of patients with Medicaid coverage, dually eligible for Medicaid and Medicaid, 65 
and older, male, and black or Hispanic. In an additional sensitivity analysis, we will exclude patients who tested COVID-
positive. We will match clinics based on their baseline characteristics in the pre-pandemic period and their initial 
treatment arm status after the onset of the pandemic. In addition, we will explore the sensitivity of our estimates to re-
matching each month, since, as noted in the intervention section above, treatment status will be time-varying. As part of 
our descriptive analysis, we will evaluate how often clinics switch treatment arms from their initial treatment status, 
which will inform our sensitivity analyses.  
 
We recognize that it is possible that the matching procedure might result in the loss of clinics and sample size. If it does, 
we would instead apply alternative methods that accomplish the goal of propensity score matching (addressing 
confounding to improve causal estimation) without loss of sample size. The first alternative is inverse probability 
weighting, which uses the propensity score to re-weight the outcomes, creating a pseudo-sample in which exposure to 
the intervention is unrelated to potential confounders.41 The second alternative is doubly robust estimation, which 
incorporates (i) a propensity score model and (ii) and outcome regression model in order to gain efficiency (more power) 
in comparison to an inverse weighting, and to reduce reliance on the ability of correctly positing the propensity score 
model.42 We can substitute these approaches for the originally proposed propensity score weighting, or we can instead 
apply them in a sensitivity analysis. 
 
We will define the degree of telemedicine exposure (dose) in two ways. First, based on the comprehensive landscape 
survey of primary care leaders in Aim 1, we will develop a composite score indicating the robustness of the telemedicine 
program at each clinic site under study. Among other features, this score will take into account the total number of 
components and delivery modes of the telemedicine program, its focus on providing coordinated care to a patient 
population with chronic care needs, and staff and clinical support from the organization.  
 
Second, we will define telemedicine exposure based on the proportion of all visits at a given clinic that are delivered via 
telemedicine in each month. Specifically, we will classify clinics into quartiles of telemedicine exposure based on the 
distribution of all telemedicine provision among clinics in this treatment arm. We will then separately analyze the 
relationship between our outcomes of interest and the amount (dose) of telemedicine being offered at a clinic. We will 
cluster all primary and secondary analysis at the clinic level, as this is the level at which the exposure occurred.43 
 
Finally, in addition to the primary intent-to-treat analysis, we are proposing an additional secondary analysis. This 
secondary “as treated” analysis is designed to evaluate the relationship between patient-level telemedicine exposure 
and our outcomes of interest. In this secondary analysis, we will characterize patients’ exposure to telemedicine after 
the first wave of shelter-in-place measures were put into place based on the treatment that they actually received 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
19 
 
rather than based on the availability of telemedicine at the primary care clinic to which they were attributed. In this 
analysis, we will adjust for observed clinic and patient-level characteristics as described in the response above that may 
impact both the availability and choice to use telemedicine, as well as patient outcomes. Additionally, we will provide 
estimates from a fixed effects model in which each patient serves as their own control.  
 
Primary findings will be presented alongside descriptive statistics assessing the external and internal validity of the 
analysis, including balance across relevant covariates before and after matching. We will also use descriptive statistics to 
characterize the analysis population, identify sources of missingness in the data across outcomes and covariates as well 
as whether missingness may introduce selection bias into our analysis. In sensitivity analyses, we will multiply impute 
covariates for which there is meaningful evidence of missingness. 
 
Heterogenous Treatment Effects 
 
We hypothesize that patients in disadvantaged sociodemographic groups will derive less benefit from telemedicine than 
patients in majority groups because of barriers to use of telemedicine described in the conceptual model. We 
additionally hypothesize that more robust telemedicine programs (as defined in the Aim 1 taxonomy) will provide more 
benefit to patients than less robust ones.  
 
We will explore variation in effects for the outcomes listed in Table 1 among pre-specified sociodemographic subgroups 
(race/ethnicity, rurality, median income for zip code) and clinical subgroups (COVID-19 patients vs. all other patients; 
low vs. high burden of illness; stable vs. unstable disease, as indicated by clinical measures of disease progression). 
Finally, we will use the Aim 1 survey for stratified analyses of telemedicine factors, such as historical use of telemedicine 
and patient technical assistance. Additionally, the Medicare data will allow us to conduct a subgroup analysis for our 
claims-based outcomes on dually eligible patients. Therefore, we will conduct eight separate subgroup analyses in all. 
We will use the same analytic approach described in the the primary analysis to conduct these subgroup analyses and 
present the differences in treatment effect estimates with standard errors. We will also conduct univariate subgroup 
analyses with interaction effects to statistically test for differences in outcomes (as outlined below) among these 
subgroups. 
 
The goal of these subgroup analyses is to evaluate the effects of each of the three telemedicine arms across patient 
populations which may have unequal access to telemedicine visits, experience differential treatment effects of 
telemedicine use, and vary in experiences with regards to using telemedicine. In particular, individuals with lower digital 
health literacy and higher illness burden may be affected differentially by the adoption of telemedicine-supplemented or 
telemedicine-only care relative to in-person care only. Previous work has also shown that older adults (aged 65 and 
older) have lower digital health literacy, and individuals in rural areas are more likely to face greater barriers accessing 
telemedicine.44,45 Previous work has also reported that black patients were less likely to use an inpatient portal relative 
to white patients, although recent work has shown that during the COVID-19 pandemic, blacks may be more likely than 
whites to report using telemedicine.46,47  We hypothesize that the comparative effectiveness of the treatment arms will 
differ significantly across the following subgroups: race/ethnicity, rurality, income as proxied by payer and median 
income for zip code.  
 
G. ETHICS AND DISSEMINATION – AIMS 1 AND 2 
Research Ethics Approval  
 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
20 
 
BRANY, an independent fully AAHRPP-accreditated IRB, will serve as the IRB of record for this study. An exempt approval 
was received by BRANY on 11/16/2020. Local IRBs will be obtained at the participating sites by the expected deliverable 
date (2/15/21). 
 
All protocol modifications will be disseminated to relevant parties, including all participating investigators and sites, the 
IRB, and PCORI. 
 
Confidentiality 
 
Investigators will maintain strictest standards for confidentiality and privacy of all research data. For Aim 1 interviews, 
data will be recorded, stored, and analyzed using study-specific IDs rather than participant name; the list linking name to 
ID will be stored securely and accessible only to the site PIs. Transcription will be done by a HIPAA compliant vendor. 
Any mention of identifying information will be stripped from the interview transcript before it is saved and analyzed. 
Direct access to the data will be limited to authorized research personnel named on the IRB. For Aim 2 analyses, 
quantitative analyses will be performed on limited data sets containing no individually identifying information other 
than dates. Data security will be maintained as described above in the data management plan. No efforts will ever be 
made to re-identify any research data. 
 
Declaration of Interests 
 
All investigators have completed a Conflict of Interest Disclosure Form. None of the Principal Investigators (Jessica 
Ancker, Rainu Kaushal, Chris Harle, Carol Horowitz, Saif Khairat, Winnie Chi) have any financial or competing interests.  
 
Access to Data 
 
As described under the Data Management Plan, primary data analysis will occur within Weill Cornell’s Data Core, a 
secure research computing environment that meets all relevant standards for working with potentially identifiable 
patient information. The Weill Cornell coinvestigators named on the Medicare DUA will have access to the data set for 
the primary analysis. In light of the need for speed in this 2 year project, secondary analyses involving commercial claims 
data will be conducted by the payer coinvestigators, who will have full access to these secondary data sets. Raw data 
sources and analytic files will be maintained within the secure computing environments for two years following 
conclusion of the project. 
 
Dissemination Policy 
These studies have great potential for dissemination and implementation in light of national interest in telemedicine, 
driven by patient questions about the benefits of telemedicine, primary care practice concerns on how to effectively and 
safely provide services after COVID-19 shelter-in-place orders expire, and the desire by public and private payers for 
actionable guidance on whether to extend coverage and reimbursement for telemedicine. The findings from our 
research will include: a taxonomy and explanation of pragmatic telemedicine initiatives as deployed in the field (which 
will have value to policymakers and researchers seeking clarity about terms and definitions in use); a nuanced 
understanding of provider and patient perspectives, including both advantages and barriers to use (of interest to patient 
and provider stakeholders); and rigorous evidence about the comparative effectiveness of telemedicine relative to 
traditional models of in-person care (of value to policymakers, payers, providers, and patients). 
 
Results from these studies will be disseminated in multiple ways. We anticipate publishing results to top medical and 
scientific journals. The research team has an impressive publication record of hundreds of peer-reviewed scientific 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
21 
 
articles, including many in high-profile journals. In addition, all are active members of national and international 
scientific organizations and routinely present interim and final findings at the PCORI Annual Meeting, AcademyHealth, 
the American Medical Informatics Association, the American Public Health Association, and many other venues. Findings 
will be disseminated in more approachable, non-technical language through social media and study-specific websites. 
Finally, we will work with our Stakeholder Board to assess stakeholder interest in an actionable toolkit of 
recommendations for telemedicine implementation.  
 
All reports and publications will be designed to provide sufficient information to allow for assessment of the studies 
internal and external validity. 
 
Authorship eligibility guidelines will be determined using the International Committee of Medical Journal Editors' 
(ICMJE) definition of authorship, which is based on the following 4 criteria: 
1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation 
of data for the work; AND 
2) Drafting the work or revising it critically for important intellectual content; AND 
3) Final approval of the version to be published; AND 
4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or 
integrity of any part of the work are appropriately investigated and resolved. 
 
One goal of our work with patient and stakeholder partners will be to define barriers to telemedicine implementation. 
The pragmatic design of our research and the broad inclusion criteria will make our findings more generalizable. We 
anticipate that our findings will result in actionable guidance. Some health care organizations, however, may face 
significant financial constraints, which limit their ability to act on all study findings and recommendations. To address 
this concern, our research design is intended highlight which patient groups are most likely to benefit from specific 
telemedicine interventions, which could help providers target telemedicine initiatives efficiently. We will identify other 
potential barriers to implementation and address them through Aim 1 and our engagement plan. 
 
 
H. ENGAGEMENT APPROACH 
Stakeholder input will be elicited to identify telemedicine leaders, patients, and providers to participate in Aim 1, 
propose new comparator groups for Aim 2, suggest additional outcomes for analysis in Aim 2, disseminate study results 
to stakeholder groups, and discuss applications of findings to inform practice and policy. Stakeholders we will engage 
include: (a) patients with the five chronic diseases of interest; (b) primary care providers, (c) primary care practice 
leaders and technology leadership who build, support and resource telemedicine; (d) allied health professionals (i.e., 
social work, nursing) who support implementation; and (e) payers who seek to decide what telemedicine components 
they will cover. We have already begun to recruit stakeholders and the engagement leads for the three collaborating 
CRNs, led by Ms. Goytia for INSIGHT, will identify other stakeholders. The engagement approach will follow the 
accelerator model, a stakeholder-engaged research model in which stakeholders from different disciplines come 
together to share expertise, experience, and resources as equals.48 Coinvestigators Dr. Carol Horowitz (Mount Sinai site 
PI), Crispin Goytia (INSIGHT engagement lead), and Dr. Kaushal (multiple PI) have years of experience with this 
approach.49 Other accelerators in operation are supporting the PCORI-funded natural experiment in diabetes research,50 
and other big data analyses similar to the proposed project.50 In the accelerator, a national Stakeholder Board will 
inform all aspects of the study through regular meetings. Workgroups reporting to the board will include additional 
members from research sites and focus on specific study aspects, including outcome selection, results interpretation, 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
22 
 
and dissemination. For co-learning, we will build capacity of all team members through team science work and 
educational activities.51-53 Board members will be invited to co-author and co-present research findings to scientific and 
lay audiences. We will foster trust, transparency, and honesty through clear delineation of roles and responsibilities, 
regular communication, and anonymous feedback about meeting effectiveness and operations.  
To ensure representation from specialty care, we have assembled a Specialty Care Stakeholder Committee. The 
Specialty Stakeholder Committee will hold two, 1-hour meetings each year of the 2-year grant. In year 1, the meetings 
will focus primarily on research design, particularly sampling issues and questions to be asked in the interviews. In year 
2, the meetings will focus on reviewing preliminary and final findings and helping to put them in context. Committee 
members are also available for targeted questions as the research program progresses.  
● Fernando J Martinez, MD, MS, chief of pulmonary and critical care medicine, Weill Cornell, has a particular practice 
and research focus on COPD. His research is funded by the NHLBI and other sources, and he is active in the COPD 
Foundation, the American Thoracic Society, the European Respiratory Society, the American College of Chest 
Physicians, and the Fleischner Society. Dr. Martinez will serve in-kind.  
● Ruth Masterson Creber, RN, PhD, is a nurse informaticist at Weill Cornell who specializes in heart failure and works 
on other ongoing PCORI initiatives in telehealth, in collaboration with Dr. Kaushal. She will devote 3% effort.  
● Parag Goyal, MD, MSc, is a Weill Cornell cardiologist with expertise in heart failure, coronary artery disease, rhythm 
disorders, and other chronic heart conditions. He will devote 3% effort. 
● Stacey-Anne Brown, MD, MPH, is a pulmonologist at the Icahn School of Medicine at Mount Sinai with training in 
outcomes research and a clinical focus on COPD and asthma. She is currently working with research informaticists to 
evaluate remote platforms to deliver pulmonary rehabilitation services to the home. She is an active member of the 
American Thoracic Society and the American College of Chest Physicians. She also has participated in clinical trials of 
COPD and the novel coronavirus disease 2019 (COVID 19). She will devote 4% effort. 
● Ronald Tamler, MD, is a board-certified endocrinologist at Mount Sinai and is certified as a nutrition-support 
physician and diabetes educator. He specializes in Type 1 and Type 2 diabetes and diabetic complications and also 
serves as director of digital health implementation for Mount Sinai Health System. He will devote 5% effort. 
In addition, the COPD Foundation (https://www.copdfoundation.org/) has provided a letter of support for this 
project (see attachment). The COPD Foundation was established to improve the lives of people with chronic obstructive 
pulmonary disease through research, advocacy and education, and community support. We will be consulting with 
foundation leadership about qualitative approaches (such as developing the final interview guides for patients and 
providers) and interpretation of quantitative findings, as well as approaches for disseminating actionable results to 
appropriate stakeholder audiences. 
 
 
 
REFERENCES 
 
1. 
Cresswell JW, Poth CN. Qualitative Inquiry and Research Design, Fourth Edition. Thousand Oaks, CA: 
Sage Publications, Inc.,; 2018. 
2. 
Ancker JS, Kern LM, Abramson E, Kaushal R. The Triangle Model for evaluating the effect of health 
information technology on healthcare quality and safety. Journal of the American Medical Informatics 
Association. 2012;19(1):61-65. 
3. 
Pechansky R, Thomas W. The concept of access. Medical care. 1981;19(2):127-140. 
4. 
Dillman DA, Smyth JD, Christian LM. Internet, phone, mail, and mixed-mode surveys: the tailored design 
method. John Wiley & Sons; 2014. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
23 
 
5. 
Nederhof AJ. Methods of coping with social desirability bias: A review. European journal of social 
psychology. 1985;15(3):263-280. 
6. 
PCORI Methodology Committee. PCORI Methodology Standards. Washington, DC: Patient-Centered 
Outcomes Research Institute,;February 2019. 
7. 
Jones D, Machta R, Rich E, Peckham K, Mathematica. Comparative health system performance 
initiative: Compendium of US health systems, group practice linkage file. 2019. 
8. 
Morse J. The significance of saturation. Qualitative Health Research. 1995;5:147-149. 
9. 
Borges Da Silva R, Pineault R, Hamel M, Levesque JF, Roberge D, Lamarche P. Constructing taxonomies 
to identify distinctive forms of primary healthcare organizations. ISRN Family Med. 2013;2013:798347. 
10. 
Friendly M. Visualizing Categorical Data. 2000. 
11. 
Perreault K, Dionne CE, Rossignol M, Poitras S, Morin D. Physiotherapy practice in the private sector: 
organizational characteristics and models. BMC Health Serv Res. 2014;14:362. 
12. 
Inc. SI. SAS/STAT® 13.1User’sGuide The PRINQUAL Procedure. SAS Institute Inc; 2013. Accessed 
12/14/2020. 
13. 
Ward JH. Hierarchical Grouping to Optimize an Objective Function. Journal of the American Statistical 
Association. 1963;58(301):236-244. 
14. 
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 
2006;3(2):77-101. 
15. 
Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded 
Theory. 2nd ed. Thousand Oaks, CA: Sage; 1998. 
16. 
Balakrishnan R, Kaplan B, Negron R, Fei K, Goldfinger JZ, Horowitz CR. Life after Stroke in an Urban 
Minority Population: A Photovoice Project. Int J Environ Res Public Health. 2017;14(3). 
17. 
Lunde B, Littman L, Stimmel S, Rana R, Jacobs A, Horowitz CR. "Just Wear Dark Underpants Mainly": 
Learning from Adolescents' and Young Adults' Experiences with Early Discontinuation of the 
Contraceptive Implant. J Pediatr Adolesc Gynecol. 2017;30(3):395-399. 
18. 
Vedanthan R, Tuikong N, Kofler C, et al. Barriers and Facilitators to Nurse Management of 
Hypertension: A Qualitative Analysis from Western Kenya. Ethnicity & disease. 2016;26(3):315-322. 
19. 
Ives B, Nedelman M, Redwood C, et al. Vision Voice: A Multimedia Exploration of Diabetes and Vision 
Loss in East Harlem. Prog Community Health Partnersh. 2015;9(3):335-343. 
20. 
Ancker JS, Witteman HO, Hafeez B, Provencher T, Van de Graaf M, Wei E. "You get reminded you're a 
sick person": Personal data tracking and patients with multiple chronic conditions. Journal of Medical 
Internet Research. 2015;17(8):e202. 
21. 
Ancker JS, Witteman HO, Hafeez B, Provencher T, Wei E. The invisible work of personal health 
information management among people with multiple chronic conditions: Qualitative interview study 
among patients and providers. Journal of Medical Internet Research. 2015;17(6):e137. 
22. 
Benda NC, Das T, Abramson E, et al. “How did you get to this number?” Stakeholder needs for 
implementing predictive analytics: a pre-implementation qualitative study. Journal of the American 
Medical Informatics Association. 2020;27(5):709-716. 
23. 
Ancker JS, Stabile C, Carter J, et al. Informing, Reassuring, or Alarming? Balancing Patient Needs in the 
Development of a Postsurgical Symptom Reporting System in Cancer. Proc AMIA Annu Fall Symp. 
2018;in press. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
24 
 
24. 
Korstjens I, Moser A. Practical guidance to qualitative research. Part four: Trustworthiness and 
publishing. European Journal of General Practice. 2018;24(1):120-124. 
25. 
Morse JM, Barrett M, Mayan M, Olson K, Spiers J. Verification Strategies for Establishing Reliability and 
Validity in Qualitative Research. International Journal of Qualitative Methods. 2002;1(2):13-22. 
26. 
Ancker JS, Nosal S, Hauser D, Way C, Calman N. Access policy and the digital divide in patient access to 
medical records. Health Policy and Technology. 2017;6(1):3-11. 
27. 
(CMS) CfMMS. Chronic Conditions Data Warehouse. 2020. 
28. 
Uscher-Pines L, Mehrotra A. Analysis of Teladoc use seems to indicate expanded access to care for 
patients without prior connection to a provider. Health Aff (Millwood). 2014;33(2):258-264. 
29. 
Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). Rand Santa Monica, 
CA; 1994. 
30. 
Billings J, Parikh N, Mijanovich T. Emergency department use in New York City: a substitute for primary 
care? Issue Brief (Commonw Fund). 2000(433):1-5. 
31. 
Torio CM, Elixhauser A, Andrews RM. Trends in Potentially Preventable Hospital Admissions among 
Adults and Children, 2005-2010: Statistical Brief #151. Healthcare Cost and Utilization Project (HCUP) 
Statistical Briefs. Rockville (MD)2006. 
32. 
Quality AfHRa. Prevention Quality Indicators Overview. 2020; 
https://www.qualityindicators.ahrq.gov/modules/pqi_resources.aspx. 
33. 
Zulman DM, Pal Chee C, Ezeji-Okoye SC, et al. Effect of an Intensive Outpatient Program to Augment 
Primary Care for High-Need Veterans Affairs Patients: A Randomized Clinical Trial. JAMA Intern Med. 
2017;177(2):166-175. 
34. 
Zulman DM, Chee CP, Ezeji-Okoye SC, et al. Effect of an intensive outpatient program to augment 
primary care for high-need Veterans Affairs patients: a randomized clinical trial. JAMA internal 
medicine. 2017;177(2):166-175. 
35. 
Kruse CS, Krowski N, Rodriguez B, Tran L, Vela J, Brooks M. Telehealth and patient satisfaction: a 
systematic review and narrative analysis. BMJ open. 2017;7(8):e016242-e016242. 
36. 
Serper M, Nunes F, Ahmad N, Roberts D, Metz D, Mehta S. Positive Early Patient and Clinician 
Experience with Telemedicine in an Academic Gastroenterology Practice during the COVID-19 
Pandemic. Gastroenterology 2020. 
37. 
Coleman K, Austin BT, Brach C, Wagner EH. Evidence On The Chronic Care Model In The New 
Millennium. Health Affairs. 2009;28(1):75-85. 
38. 
Ancker JS, Kern LM, Abramson E, Kaushal R. The triangle model for evaluating the effect of health 
information technology on healthcare quality and safety. JAMIA. 2011;18:749-753. 
39. 
DeLone WH, McLean ER. The Delone and McLean model of information systems success: a 10-year 
update. Journal of Management Information Systems. 2003;19(For):9-30. 
40. 
Angrist  JD PJ-S. Mastering Metrics: The Path From Cause to Effect. Princeton, NJ: Princeton University 
Press; 2015. 
41. 
Curtis LH, Hammill BG, Eisenstein EL, Kramer JM, Anstrom KJ. Using Inverse Probability-Weighted 
Estimators in Comparative Effectiveness Analyses with Observational Databases. Medical Care. 
2007;45(10):S103-S107. 
42. 
Funk MJ, Westreich D, Wiesen C, Stürmer T, Brookhart MA, Davidian M. Doubly Robust Estimation of 
Causal Effects. American Journal of Epidemiology. 2011;173(7):761-767. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
25 
 
43. 
Bertrand M, Duflo E, Mullainathan S. How Much Should We Trust Differences-In-Differences 
Estimates?*. The Quarterly Journal of Economics. 2004;119(1):249-275. 
44. 
Ancker JS, Grossman LV, Benda NC. Health Literacy 2030: Is It Time to Redefine the Term? Journal of 
General Internal Medicine. 2019. 
45. 
Levy H, Janke AT, Langa KM. Health literacy and the digital divide among older Americans. Journal of 
General Internal Medicine. 2015;30(3):284-289. 
46. 
Walker DM, Hefner JL, Fareed N, Huerta TR, McAlearney AS. Exploring the Digital Divide: Age and Race 
Disparities in Use of an Inpatient Portal. Telemedicine journal and e-health : the official journal of the 
American Telemedicine Association. 2020;26(5):603-613. 
47. 
Campos-Castillo C, Anthony D. Racial and Ethnic Differences in Self-Reported Telehealth Use during the 
COVID-19 Pandemic: A Secondary Analysis of a U.S. Survey of Internet Users from Late March. Journal 
of the American Medical Informatics Association. 2020. 
48. 
Poger J, Mayer V, Duru O, et al. Network Engagement in Action: Stakeholder Engagement Activities to 
Enhance Patient-centeredness of Research. Medical Care. 2020;58:S66-S74. 
49. 
Horowitz CR, Shameer K, Gabrilove J, et al. Accelerators: Sparking Innovation and Transdisciplinary 
Team Science in Disparities Research. Int J Environ Res Public Health 2017;14(225). 
50. 
Goytia CN, Kastenbaum I, Shelley D, Horowitz CR, Kaushal R. A tale of 2 constituencies: exploring 
patient and clinician perspectives in the age of big data. Medical care. 2018;56(10 Suppl 1):S64. 
51. 
Gabrilove JL, Fattah L, Bloom L, Della Ventura C. 3509 Developing a Leadership Alumni Forum to foster 
a culture of leadership at Mount Sinai. Journal of Clinical and Translational Science. 2019;3(s1):65-66. 
52. 
Gabrilove JL, Ventura CD, Fattah L, et al. 2517 Learning to LEAD: Leadership emerging in academic 
departments. Journal of Clinical and Translational Science. 2018;2(S1):58-58. 
53. 
Mazumdar M, Messinger S, Finkelstein DM, et al. Evaluating academic scientists collaborating in team-
based research: A proposed framework. Academic medicine: journal of the Association of American 
Medical Colleges. 2015;90(10):1302. 
54. 
Broens TH, Huis in't Veld RM, Vollenbroek-Hutten MM, Hermens HJ, van Halteren AT, Nieuwenhuis LJ. 
Determinants of successful telemedicine implementations: a literature study. Journal of telemedicine 
and telecare. 2007;13(6):303-309. 
55. 
Ranganathan C, Balaji S. Key Factors Affecting the Adoption of Telemedicine by Ambulatory Clinics: 
Insights from a Statewide Survey. Telemedicine and e-Health. 2019. 
56. 
Agha Z, Schapira RM, Maker AH. Cost effectiveness of telemedicine for the delivery of outpatient 
pulmonary care to a rural population. Telemedicine Journal and e-Health. 2002;8(3):281-291. 
57. 
Breslow MJ, Rosenfeld BA, Doerfler M, et al. Effect of a multiple-site intensive care unit telemedicine 
program on clinical and economic outcomes: an alternative paradigm for intensivist staffing. Critical 
care medicine. 2004;32(1):31-38. 
58. 
Sartori DJ, Olsen S, Weinshel E, Zabar SR. Preparing trainees for telemedicine: a virtual OSCE pilot. 
Medical education. 2019;53(5):517-518. 
59. 
Obstfelder A, Engeseth KH, Wynn R. Characteristics of successfully implemented telemedical 
applications. Implementation science. 2007;2(1):25. 
60. 
Hanson D, Calhoun J, Smith D. Changes in provider attitudes toward telemedicine. Telemedicine and e-
Health. 2009;15(1):39-43. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
26 
 
61. 
Whitten P, Love B. Patient and provider satisfaction with the use of telemedicine: overview and 
rationale for cautious enthusiasm. Journal of postgraduate medicine. 2005;51(4):294. 
62. 
Totten AM, Womack DM, Eden KB, et al. Telehealth: mapping the evidence for patient outcomes from 
systematic reviews. 2016. 
63. 
Conway A, Inglis SC, Clark RA. Effective technologies for noninvasive remote monitoring in heart 
failure. Telemedicine and e-Health. 2014;20(6):531-538. 
64. 
Flodgren G, Rachas A, Farmer AJ, Inzitari M, Shepperd S. Interactive telemedicine: effects on 
professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2015(9). 
65. 
Hopp F, Woodbridge P, Subramanian U, Copeland L, Smith D, Lowery J. Outcomes associated with a 
home care telehealth intervention. Telemedicine Journal & e-Health. 2006;12(3):297-307. 
66. 
Inglis SC, Clark RA, Dierckx R, Prieto‐Merino D, Cleland JG. Structured telephone support or non‐
invasive telemonitoring for patients with heart failure. Cochrane Database of Systematic Reviews. 
2015(10). 
67. 
Steventon A, Bardsley M, Billings J, et al. Effect of telehealth on use of secondary care and mortality: 
findings from the Whole System Demonstrator cluster randomised trial. Bmj. 2012;344:e3874. 
68. 
Jansen-Kosterink S, Dekker-van Weering M, van Velsen L. Patient acceptance of a telemedicine service 
for rehabilitation care: A focus group study. International journal of medical informatics. 2019;125:22-
29. 
69. 
Faruque LI, Wiebe N, Ehteshami-Afshar A, et al. Effect of telemedicine on glycated hemoglobin in 
diabetes: a systematic review and meta-analysis of randomized trials. Cmaj. 2017;189(9):E341-E364. 
70. 
Almathami HKY, Win KT, Vlahu-Gjorgievska E. Barriers and Facilitators That Influence Telemedicine-
Based, Real-Time, Online Consultation at Patients’ Homes: Systematic Literature Review. Journal of 
medical Internet research. 2020;22(2):e16407. 
71. 
Sanders C, Rogers A, Bowen R, et al. Exploring barriers to participation and adoption of telehealth and 
telecare within the Whole System Demonstrator trial: a qualitative study. BMC health services 
research. 2012;12(1):220. 
72. 
Schulz T, Richards M, Gasko H, Lohrey J, Hibbert M, Biggs BA. Telehealth: experience of the first 120 
consultations delivered from a new R efugee T elehealth clinic. Internal Medicine Journal. 
2014;44(10):981-985. 
73. 
Brebner J, Brebner E, Ruddick-Bracken H. Experience-based guidelines for the implementation of 
telemedicine services. Journal of telemedicine and telecare. 2005;11(1_suppl):3-5. 
74. 
Stevenson L, Ball S, Haverhals LM, Aron DC, Lowery J. Evaluation of a national telemedicine initiative in 
the Veterans Health Administration: Factors associated with successful implementation. Journal of 
telemedicine and telecare. 2018;24(3):168-178. 
75. 
Ellimoottil C, An L, Moyer M, Sossong S, Hollander JE. Challenges and opportunities faced by large 
health systems implementing telehealth. Health Affairs. 2018;37(12):1955-1959. 
76. 
Fortney JC, Pyne JM, Kimbrell TA, et al. Telemedicine-based collaborative care for posttraumatic stress 
disorder: a randomized clinical trial. JAMA psychiatry. 2015;72(1):58-67. 
77. 
Prendergast M, Honey M. The Barriers and Facilitators for Nurse Educators Using Telehealth for 
Education. Studies in health technology and informatics. 2019;264:1323-1326. 
78. 
Fairbrother P, Ure J, Hanley J, et al. Telemonitoring for chronic heart failure: the views of patients and 
healthcare professionals–a qualitative study. Journal of clinical nursing. 2014;23(1-2):132-144. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
27 
 
79. 
Porath A, Irony A, Borobick AS, et al. Maccabi proactive Telecare Center for chronic conditions–the care 
of frail elderly patients. Israel journal of health policy research. 2017;6(1):68. 
80. 
Due-Christensen M, Kaldan G, Almdal TP, Glindorf M, Nielsen KE, Zoffmann V. Out-of-office hours 
nurse-driven acute telephone counselling service in a large diabetes outpatient clinic: A mixed methods 
evaluation. Patient education and counseling. 2015;98(7):890-894. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Appendix A: Aim 1 Survey Components 
Telemedicine domains (I-VII) and sub-domains to be included in our survey; domains are based on the Triangle Model 
evaluation framework for health information technology (1). References (mostly reviews) are provided for each sub-
domain indicating a role in patient outcomes or successful implementation of a telemedicine program. For a subset of 
sub-domains, relevant Penchansky and Thomas ‘access-to-care’ framework elements (2,3) are noted that indicate the 
potential for care access improvement related to these sub-domains. These include: A) affordability, B) accommodation, 
C) availability, D) accessibility, and E) acceptability. 
  
Evidence indicating 
positive effect on 
outcomes / 
successful 
implementation 
Relevant 
Penchansky & 
Thomas 
‘access-to-care’ 
framework 
elements 
I. ORGANIZATION (PRIMARY CARE PRACTICE) CHARACTERISTICS 
 
 
Location (rural/urban, in states with varying legislation/policies on telemedicine) 
54,55 
A / D 
Patient volume 
56,57 
C 
Ownership (e.g. health system-owned/physician-owned) 
55 
B 
Practice type (e.g. community, academic, FQHC) 
55 
B 
Provider mix (faculty/trainees) [also under II. PROVIDER] 
58 
B / C 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
28 
 
Patient mix (insurance, race/ethnicity, age / primary language) [also under IV. 
PATIENTS] 
59 
A / E 
II. PROVIDER CHARACTERISTICS 
 
 
Comfortable with telemedicine (e.g. indirectly measured through previous use of 
telemedicine in practice) 
60[previous use a 
negative impact] 
- 
Provider satisfaction with telemedicine (through proxy sources such as 
complaints or requests for support) 
61 
- 
Provider trainee status (faculty/trainee) [also under I. ORGANIZATION] 
58 
 
III. TECHNOLOGY (TELEMEDICINE) CHARACTERISTICS 
 
 
Type 
 
 
-Remote monitoring 
62-67 
B / C / D 
-Store-and-forward 
62,64 
B / C / D 
-Interactive telemedicine (video/audio) 
62 64,65 
B / C / D 
-Interactive telemedicine (audio-only) 
62, 63,64,66 
B / C / D 
Mode 
 
 
-Patient portal 
68 [potential 
negative impact] 69 
B 
-App/Software 
70 
B 
-Electronic medical record integration 
54,71 
- 
-Interpreter services (general availability and mode of operation during 
telemedicine visits) 
72 
B / C / E 
IV. PATIENTS 
 
 
Patient mix (insurance, race/ethnicity, age / primary language) [also under I. 
ORGANIZATION] 
59 
A / E 
Patient access to technology 
54,73 
B / E 
V. ORGANIZATION—TECHNOLOGY RELATIONSHIP 
 
 
Leadership engagement 
74 
- 
Needs driven implementation of telemedicine 
14,54,59,75 
B / C 
Supplementation or substitution of usual care (e.g. in-person visit) 
76 
- 
Identification of ‘champions’ 
54,74,75,77 
- 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
29 
 
Raising awareness among (potential) patients / marketing / outreach (e.g. to get 
registered onto the electronic medical record platform) 
54 
B / E 
Clarity on candidate criteria (‘who is a potential candidate for telemedicine?’) 
54,78 
B 
Proactive/pre-emptive vs. reactive scheduling of telemedicine visits 
79 
B / C 
Continuous evaluation of data (processes and outcomes) 
75 
- 
Availability outside of office hours 
80 
B 
De novo implementation or expansion of telemedicine 
54 
- 
Roll-out characteristics (acute/spread out) 
73,74 
A / B 
Long-term plan (e.g. future expansion) 
 
C 
Disruption of services (e.g. during encounter) 
54,71,74,78 
B / C 
Patient support services / education (e.g. patient preparation through online 
resources, on the expected process, troubleshooting mechanisms, instructions in 
case of disruptions during the encounter, etc.) 
54,71,73,75,78 
B / C / E 
Data security 
54,71 
 
Financial aspects of telemedicine / reimbursement (what is reimbursed and for 
who and does it matter in terms of telemedicine practice?) / Suitable budget 
available (e.g. application for FCC COVID-19 telemedicine funding) 
54,55,73,77 
A / C / E 
Collaboration between promoters and users 
59 
B 
VI. PROVIDER—TECHNOLOGY RELATIONSHIP 
 
 
Seamless integration into daily routines including services such as scheduling, 
billing, documentation, check-in/-out process, handling of urgent visits, ordering 
of laboratory tests/imaging 
54,55,59,74 
B 
VII. ORGANIZATION—PROVIDER RELATIONSHIP 
 
 
Provider training 
59,73,74,77 
- 
Support staff 
71,73,77,78 
- 
 
 
 
 
 
 
 
 
 
 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
30 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Appendix B: Oral Consent Script for Practice Leadership – One-on-one Interview 
 
“My name is XXX. We are conducting this interview to learn more about when and how telemedicine has been 
deployed by your organization this year. Our ultimate goal is to assess the impact of telemedicine on patients 
with chronic disease, using claims data and EHR data. This study is part of a research project funded by the 
Patient Centered Outcomes Research Institute (PCORI).   
If you agree to take part in this study, we will conduct a 30-60-minute interview. We will be taking typed or 
handwritten notes, and with your permission, audio-recording the interview. The recordings will be transcribed 
to check the handwritten notes and for qualitative data analysis.  
This study does not involve any personal health information. We will follow strict protocols to protect all 
research information. There is always a finite risk for a confidentiality breach. To protect against this, you will 
be assigned a study ID number. Data collected will be referred to only by study ID number. All study data 
collected will be treated as confidential among the research team. Data will be reported in aggregate. Your 
individual responses will be anonymous.  
Your participation in this study is entirely voluntary. You may choose not to respond to any questions you do 
not feel comfortable answering and you may end your participation at any time.  You may or may not 
experience personal benefit from taking part in this study, but knowledge gained from your participation may 
benefit others. There will be no cost to you. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
31 
 
Should you have any questions or requests for information relating to this research, comments, or concerns 
about this study, you can contact me at XXXX, or the Institutional Review Board of Biomedical Research 
Alliance of New York.  The Institutional Review Board number is XXXX. 
Are you willing to participate? 
May we record this interview for the purposes of qualitative data analysis?” 
 
 
 
 
 
 
 
 
Appendix C: Consent Script for Providers 
 
“My name is XXX.  We are conducting this interview to learn more about your experiences with telemedicine. 
Our goals is to understand the impact of telemedicine on patients with chronic disease. This grant is funded by 
the Patient Centered Outcomes Research Institute (PCORI).   
If you agree to take part in this study, we will conduct a 30-60-minute interview. We will be taking typed or 
handwritten notes, and with your permission, audio-recording the interview. The recordings will be transcribed 
for the purpose of qualitative data analysis.  
This study does not involve any personal health information. We will follow strict protocols to protect all 
research information. There is always a finite risk for a confidentiality breach. To protect against this, all 
participants will be assigned a study ID number. Data collected will be referred to only by study ID number. All 
study data collected will be treated as confidential among the research team. Data will be reported in aggregate. 
Your individual responses will be anonymous and not be shared with any leadership.  
Your participation in this study is entirely voluntary. Your decision to participate or not participate will not be 
shared with any leadership. You may choose not to respond to any questions you do not feel comfortable 
answering and you may end your participation at any time.  You may or may not experience personal benefit 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
32 
 
from taking part in this study, but knowledge gained from your participation may benefit others. There will be 
no cost to you.  
Should you have any questions or requests for information relating to this research, comments, or concerns 
about this study, you can contact me at XXXX, or the Institutional Review Board of Biomedical Research 
Alliance of New York.  The Institutional Review Board number is 516-318-6877. 
Are you willing to participate? 
May we record this interview for the purposes of qualitative data analysis?” 
 
 
 
 
 
 
 
Appendix D: Consent Script for Patients  
 
“My name is XXX. We are conducting this interview to get your perspective on telemedicine.  You have been 
asked to participate because you have experience using or trying to use telemedicine. This grant is funded by 
the Patient Centered Outcomes Research Institute (PCORI).   
If you agree to take part in this study, we will conduct a 30-60-minute interview. We will be taking typed and/or 
handwritten notes, and with your permission, audio-recording the interview. The recordings will be transcribed 
for qualitative data analysis.  
This study does not involve any personal health information. We will follow strict protocols to protect all 
research information. There is always a finite risk for a confidentiality breach. To protect against this, all 
participants will be assigned a study ID number. Data collected will be referred to only by study ID number. All 
study data will be treated as confidential among the research team. Data will be reported in aggregate. You will 
not be individually identified in any reports or publications.   
Your participation in this study is entirely voluntary. You may choose not to respond to any questions you do 
not feel comfortable answering and you may end your participation at any time.  You may or may not 
experience personal benefit from taking part in this study, but knowledge gained from your participation may 
benefit others. There will be no cost to you. 

 
 
 
PRINCIPAL INVESTIGATOR: Ancker, Jessica S; Kaushal, Rainu 
 
 
 
33 
 
Should you have any questions or requests for information relating to this research, comments, or concerns 
about this study, you can contact me at XXXX, or the Institutional Review Board of Biomedical Research 
Alliance of New York.  The Institutional Review Board number is 516-318-68773. 
Are you willing to participate? 
May we record this interview for the purposes of qualitative data analysis?”