Pharmacist Management of Paxlovid eVisits
Phase: NA
Conditions: COVID-19, Quality of Care
Sponsor: Kaiser Permanente

1 
 
 
 
 
 
 
Pharmacist Management of Paxlovid eVisits:  
A Randomized Clinical Trial 
 
Study Protocol and Statistical Analysis Plan 
 
Tracy Lieu, MD, MPH, Kristine Lee, MD, E. Margaret Warton, MPH, Andrew deLaunay, 
PharmD, Milton Chan, MD, Thao Huynh, MD, Lisa Liu, MD, Michelle Mancha, BA, Stephanie 
Prausnitz, MS, Stacy Arcos, AA, Charles Quesenberry, PhD, Kyle Reader, MHA, Eric 
Smallberg, MA, Mary Reed, DrPH 
 
From The Permanente Medical Group (Division of Research, Virtual Care Team, Adult and 
Family Medicine regional leads, and TPMG Consulting Services) and Pharmacy Services, 
Kaiser Permanente Northern California, all in Oakland, CA  
 
 
Version date: February 25, 2024 tracked 
Submitted to ClinicalTrials.gov 
 
 

2 
 
Abstract 
 
Importance. Enhancing the management of messages from patients and providing virtual 
options for urgent care are top priorities for The Permanente Medical Group (TPMG) and Kaiser 
Foundation Hospitals/Health Plan (KFH/P). 
 
Objective. To compare pharmacist management of e-visit requests for Paxlovid for COVID-19 
with management by adult and family medicine physician pools regarding costs, time, clinical 
outcomes, and patient and clinician satisfaction.  
 
Design, setting, and participants. This cluster randomized clinical trial will include adults from 
17 medical facilities of Kaiser Permanente Northern California who make Paxlovid e-visits on 
weekdays from October 9 to December 11, 2023.   
 
Intervention. In the intervention group, a regional team of pharmacists will manage Paxlovid e-
visits following a standard protocol; in the comparison group, adult and family medicine 
physicians (AFMs) will manage these visits according to medical center-based protocols.  
 
Main Outcomes and Measures. The primary outcome is whether a patient with one or more 
common drug-drug interactions received counseling for any drug-drug interaction. Secondary 
outcomes are the hours from the e-visit request to the prescription among patients who receive 
one and the clinician time and cost per visit managed. 
 
Potential Results. We will test the hypotheses that Pharmacist Care compared with AFM Pool 
Care will have higher quality of care and lower costs based on the receipt of counseling for 
common drug-drug interactions, faster time to prescriptions, and lower clinician time and cost 
per visit managed.  
 
Potential Conclusions and Relevance. If pharmacist management has better or similar 
outcomes compared with AFM pool management, this will provide support to continue this 
practice for Paxlovid e-visits and to evaluate possible expansion of pharmacist management for 
similar clinical situations.  
 
1. Specific Aims and Hypotheses 
 
Enhancing message management and virtual urgent care options for patients are among TPMG 
and KFH’s top priorities. As team-based inbox management is further developed, evaluating its 
clinical effectiveness and outcomes in terms of patient care experience and costs will provide 
valuable information for operational planning. In addition, sharing what we learn with other 
health care systems through talks and publications will help highlight TPMG’s reputation as a 
group that innovates to support clinicians while increasing patient access through virtual care.  
 
This study will compare the clinical outcomes and costs of two alternative modes of 
management of patient requests for Paxlovid sent via e-visits: management by a regional team 
of pharmacists, vs. management by medical center-based pools of adult and family medicine 
physicians (AFMs) doing desktop medicine. We will conduct a cluster randomized trial in 17 
facilities in Kaiser Permanente Northern California to compare Pharmacist Care with AFM Pool 
Care for patients making Paxlovid e-visits with regard to: 
 
Aim 1. Quality of care 
  

3 
 
Example hypothesis: Patients with common potential DDIs will be more likely to receive 
relevant counseling in the Pharmacist Care group than the AFM Pool Care group.  
 
Aim 2. Clinician time and costs 
 
Example hypothesis: Pharmacist Care will have lower costs of clinician time per Paxlovid 
e-visit managed than AFM Pool Care. 
 
 
Aim 3. Exploratory outcomes 
 
Example hypothesis: Patients in the Pharmacist Care will report comparable satisfaction 
with the process compared with the AFM Pool Care group. 
 
2. Background and Significance 
 
Physicians experience substantial stress associated with managing their electronic health 
record (EHR) inboxes.1-3 Much of physicians’ time in the EHR inbox involves answering 
messages from patients.4 The volume of messages is a key driver of stress and attention 
switching.1,5  
 
Assisting physicians with message management is a top priority for The Permanente Medical 
Group (TPMG). A variety of approaches are possible. Team-based approaches that involve 
pharmacists and other clinical staff have been recommended, and anecdotal report suggests 
they have been successful in some settings.6 However, scant systematic evidence exists on 
whether such approaches are effective in terms of timeliness or quality of care, or in actually 
saving physicians’ time.  
 
This study will address this important gap in evidence via a cluster randomized trial comparing 
pharmacist management of patient e-visit requests for nirmatrelvir-ritonavir (Paxlovid) with 
management by pools of adult and family medicine (AFM) physicians. The latter was the default 
approach by most Kaiser Permanente Northern California (KPNC) medical centers before 2023. 
The findings will have broad implications for regional care delivery planning, as TPMG and KFH 
consider whether and how to have pharmacists and other clinical team members participate in 
message management. This study will inform operational decisions and practice for both Kaiser 
Permanente and other health care systems in the U.S. and beyond.  
 
3. Innovation 
 
Pharmacist management of secure messages from patients is innovative in not having been 
adopted or evaluated at large scale. Preliminary anecdotal and unpublished information from 
within TPMG suggests that pharmacists’ scope of practice is well-suited to message 
management for many specific conditions based on treatment algorithms. At our KP San 
Francisco medical center, pharmacists have been able to take charge of messages for a range 
of issues, including refill authorization requests and requests to manage chronic conditions 
(hypertension, osteoporosis, attention deficit disorder, gout) and time-limited conditions 
(sexually transmitted infections, latent tuberculosis, opioid tapering, and assessment of Paxlovid 
eligibility).  
 
E-visit management is another innovative aspect of this study. Kaiser Permanente Northern 
California (KPNC) is ahead of most other systems nationally in its implementation of e-visits. 

4 
 
Enhancing the accessibility of urgent care through digital and other means is a key priority for 
TPMG as well as KP generally.    
 
4. Approach  
 
4.1. Design, setting, and participants.  
 
This cluster randomized clinical trial will assign patient requests for Paxlovid made via e-visits to 
either Pharmacist Care or AFM Pool Care, which consists of management by medical center-
based pools of AFMs in adult and family medicine (AFM). The unit of randomization is the 
KPNC facility. KPNC is an integrated health care system with 4.6 million members served by 
more than 9,500 physicians and 87,000 employees in 15 service areas. It has a mature EHR 
(EPIC/HealthConnect, adopted circa 2008) and encourages members to use a wide array of 
options for outpatient care, including secure messages to physicians via the EHR portal, e-
visits, video, telephone, and in-person office visits.  
 
The target population is adults 18 years and older in facilities within participating KPNC service 
areas who submit requests for Paxlovid on weekdays during the study period using the e-visit 
protocol accessible via the electronic portal kp.org. Messages that are handled from 8:00 am to 
4:30 pm, Monday through Friday will be included. The study period will run from October 9 to 
December 11, 2023. Pharmacist Care will not be available for Paxlovid e-visits that need to be 
reviewed on weekends or holidays. These are managed by virtualist physicians in the regional 
call center and will be excluded from this study.  
 
Of KPNC’s 15 service areas, 4 have not implemented pharmacist management in any facility. 
These service areas and their facilities are eligible for this study; others must be excluded. The 
eligible service areas are SCL, SJO, SRF, and SSC. These service areas contain 17 facilities; 
each facility will be treated as a cluster in the random assignment process. 
 
4.2. Intervention.  
 
All procedures will be conducted as part of standard of care. In the Pharmacist Care group, 
each e-visit message requesting Paxlovid will be assigned to a pharmacist on the regional team. 
Pharmacists will review requests using a standard protocol. They will call patients via telephone 
to elicit additional information where needed, file Paxlovid prescriptions when appropriate, and 
arrange home delivery when requested.  
 
The protocol for pharmacists has a list of common potential drug-drug interactions and provides 
guidance for how these should be handled. Drug-drug interactions are documented by 
pharmacists using a smartphrase in Epic, the electronic health record (EHR). Some DDIs may 
lead to changes in medications while a patient is taking Paxlovid.  
 
In the AFM Pool Care group, each e-visit message requesting Paxlovid will be assigned to a 
pool of AFM physicians based at the medical centers in a given service area. Physicians will 
review requests using the existing standard regional protocol (as well as any existing medical 
center-level protocols). The regional Paxlovid E-visit User Guide illustrates how to use 
SmartSets to document the plan and respond to patients with the outcome (eligible, not eligible, 
or further discussion needed). Following this protocol, physicians call patients via telephone to 
elicit additional information where needed, file Paxlovid prescriptions when appropriate, and 
arrange home delivery when requested.   
 

5 
 
4.3. Randomization. In each of the participating service areas, facilities have been assigned at 
random to either the Pharmacist Care or AFM Pool Care group. We used covariate-constrained 
randomization, a method to prevent imbalances on important variables that may be associated 
with the outcomes.7 We conducted preliminary analyses to characterize facilities with regard to 
population size, socioeconomic status (using the Neighborhood Deprivation Index), race, 
ethnicity, and COVID-19 incidence in the service area since the start of the pandemic. We have 
chosen these variables for use in the restricted randomization given the expected association 
with study outcomes, along with the expected level of between facility variability which would 
increase the likelihood of between-arm imbalance. We also stratify randomization on service 
area for within-area balance on number of facilities assigned to the Pharmacist Care and AFM 
Care groups.  
 
Using these variables, we enumerated all possible outcomes of randomization and eliminated 
all allocations of the facilities to the two groups that do not meet specified between-group 
balance criteria.  Standard diagnostics were applied in selecting the covariate balance criteria, 
ensuring no threats to the validity of the study design, including extreme reductions in total 
possible allocations for the trial, and instances of cluster pairs never/rarely or always/often 
randomized to the same arm.  An allocation of facilities to the two arms for implementation was 
randomly selected from the set of allocations that meet the final covariate balance criteria.     
 
The balance criteria were: maximum 5% between arm difference in total membership, 30% 
difference in total Black, 20% for each of White, Asian and Hispanic race/ethnic groups, 25% 
difference in mean NDI, and 15% difference in mean COVID-19 incidence. We are applying no 
between arm balance criteria within strata/service area. 
 
The selected randomization scheme leads to good balance between the intervention and 
comparison groups, with 4.8% difference in total membership (~565K vs ~513K), 11.7% for 
Black (23K vs 29K), 2.2% for White (183K vs 175K ), 8.7% Asian (150K vs 126K), 7.8% 
Hispanic (127K vs 109K), 12.2% difference in mean NDI (-.49 vs - .44), and 2.7% difference in 
mean COVID-19 incidence (.23 vs .22).  
 
4.4. Main outcomes and measures 
 
4.4.1. Outcomes 
 
The primary outcome will be measured only among patients who have common potential drug-
drug interactions (DDIs) based on a pre-specified list of medications. The outcome will be 
whether the patient received counseling for any valid potential DDI. For patients with more than 
one DDI, the outcome will counted as present if they received counseling for any of their 
potential DDIs.  
 
Secondary outcomes will be: 
 
a. % of all patients who were prescribed Paxlovid (based on orders) 
 
b. Hours from the e-visit request to the prescription order being placed, among patients 
who were prescribed Paxlovid 
 
c. Clinician minutes per message handled. This will be analyzed using EPIC access log 
data. We will link each e-visit for Paxlovid with the relevant secure messages and/or 

6 
 
telephone encounters, including follow-ups that occurred before the initial request was 
resolved. Clinicians include pharmacists and physicians.  
 
d. Estimated personnel cost per Paxlovid message handled. The estimated cost per 
message handled will be calculated by multiplying the number of minutes spent per 
message by the standard wage rate for a pharmacist or AFM in Northern California.  
 
e. Patient satisfaction with care 
 
 
Exploratory outcomes will be: 
 
a. Number of Paxlovid e-visits handled by the regional pharmacy team, per week 
b. % of messages fully handled by the initial clinician (pharmacist or AFM) (based on 
review of visits) 
c. % of cases handled by pharmacists that required a physician consultation (based on 
review of visits) 
 
d. Pharmacist satisfaction with the process of care  
[Note: Deleted “AFM satisfaction” as a measure because there was not a logical group 
of AFMs to survey, nor logical questions to ask them.] 
 
e. % of all patients who obtained Paxlovid (based on fills) 
f. Days from request to receipt of medication among patients prescribed Paxlovid 
 
g. % of patients with common potential DDIs who were prescribed Paxlovid 
h. If feasible based on ICD-9 codes: Incidence of drug reactions due to Paxlovid DDIs 
within 10 days after a Paxlovid prescription 
 
4.4.2. Predictors and covariates 
 
The primary predictor is study group, i.e. Pharmacist Care or AFM Pool Care. Covariates are 
other variables that could potentially mediate, moderate, or confound the association between 
the primary predictor and the outcomes. In this study, covariates include patient age, sex, race, 
ethnicity, language spoken, and comorbidities, as well as the patient’s neighborhood-level SES.  
 
4.5. Data collection.  
 
The primary outcome will be measured only among patients who have common potential drug-
drug interactions (DDIs) based on a pre-specified list of medications. The outcome will be 
whether or not they received relevant counseling for any of their DDIs. Eligible patients in both 
groups will be selected using computerized pharmacy codes generated based on the protocol 
used by the regional pharmacy team. The computerized pharmacy codes will be used to identify 
patients with any of these potential DDIs in a consistent way in the Pharmacist Care and AFM 
Pool Care groups.  
 
Research assistants blinded to group assignment will review text extracts from the EHR using a 
structured protocol to identify whether there is documentation that the patient received 
counseling regarding a DDI. With each study group, a random sample of eligible patients will be 
identified; the sample will be stratified by facility to yield a representative sample. Masking of 
group assignment will be conducted by one research assistant prior to review by a different 

7 
 
research assistant. A patient with any potential DDI will be counted as having received relevant 
counseling if any counseling (for any potential DDI) was documented in the text extra prior to or 
coincident with a Paxlovid prescription, or in the absence of a Paxlovid prescription, occurring 
within 2 days after the e-visit. 
 
For the secondary outcomes, we will use data available from KPNC computerized sources, 
including HealthConnect (HC, aka EPIC) EHR data (from Clarity) and EPIC access logs. EHR 
data will be used to identify all e-visits for Paxlovid during the study period. For each patient who 
made an e-visit, the EPIC access log data for the subsequent 5-day period will be analyzed to 
identify how many minutes of pharmacist or physician time were associated with that request.  
 
For the exploratory outcomes, patient care experience will be evaluated by surveying a sample 
of patients selected at random in the service areas in the Pharmacist Care and AFM Pool Care 
groups. We have designed a short survey that takes no more than 5 to 10 minutes to complete. 
The survey will be fielded by email, mail, and telephone. Due to the fact that we are asking 
about a specific care experience and patients’ recall fades quickly after 2 weeks, we will field the 
survey using all 3 modes in a relatively efficient sequence. A letter containing a description of 
the study, language regarding informed consent and confidentiality, and the survey will be sent 
by mail on day 11 after the e-visit, and an email with a link to the online version will be sent on 
day 14. Telephone administration will be attempted (up to 4 attempts total) starting on day 17 for 
those who have not replied to the paper or online survey. A small gift to thank patients for 
considering participation will be included in the mailing. Responses will be treated as 
confidential and data will be aggregated, with no individual identifiers associated with data 
presented in reports.  
 
Patient-reported outcomes will be reported on a 5-point scale and will include:  
a. Satisfaction with the overall quality of care 
b. Satisfaction with the timeliness of care  
c. Confidence in the care provided 
 
Clinician satisfaction with the process of care will be evaluated by surveying all pharmacists on 
the regional team providing Pharmacist Care, plus the AFM site leads for Paxlovid e-visits in the 
service areas participating in the study. Because the AFMs in each service area cross-cover the 
Paxlovid e-visit requests from all facilities in that service area, it will not be possible to survey 
front-line AFMs in the two groups separately, and it would not be productive to ask the AFMs 
about their experience with Pharmacist Care because they would typically only have direct 
experience with Pharmacist Care if a problem arose that required their attention.  
 
The pharmacist and physician surveys have been designed to require no more than 3 to 5 
minutes to complete. These surveys will be fielded via Outlook email using a Teams form, as 
well as on paper via interoffice mail. A small gift to thank pharmacists and physicians for 
considering participation will be included in the mailing. Responses will be treated as 
confidential and data will be aggregated, with no individual identifiers associated with data 
presented in reports.  
 
Data collected from both pharmacists AFMs and will include: 
a. Satisfaction with the process of care (on a 5-point scale) 
b. Suggestions for improving the care process (open-ended) 
 
4.6. Statistical analysis plan. 
 

8 
 
During the survey and visit review components of this study, it became clear that more patients 
than projected had received pharmacist care. A preliminary analysis conducted on 12/18/2024, 
prior to evaluating any outcomes, found that the facility assigned to each e-visit usually did *not* 
match the home facility of the patient who initiated the e-visit. This was most likely due to an 
operational workflow (which is still being elucidated) that was unanticipated at the time this 
study was designed and facilities were assigned at random to the Pharmacist Care vs. 
Physician Care groups. Due to the fact that the patient’s “home” facility became imbalanced, we 
will include this as a covariate in the primary analysis (see below). 
 
The primary analysis will group each patient based on which type of clinician (pharmacist or 
physician) first accessed the eVisit. This is similar to a per-protocol analysis. A patient whose 
eVisit was first accessed by a pharmacist will be counted in the pharmacist group, even if the 
eVisit was referred to a physician. A patient whose eVisit was first accessed by a physician will 
be included in the physician group, even if the eVisit was referred to a pharmacist. We select 
this approach as the primary analysis because it best reflects what patients experience in real-
life care delivery, which this study is intended to evaluate.  
 
A secondary analysis will treat each patient as part of the group to which they were assigned 
based on the facility listed on the e-visit worklist (intention-to-treat). However, this approach may 
be worse at reflecting the actual patient experience because dozens of pharmacists and 
physicians were involved in executing the randomization assignments. It is possible that some 
patients will have received a different mode of care from the group to which they were assigned. 
 
The primary outcome is binary, i.e., whether or not a patient received counseling for a DDI. This 
will be compared between study groups using mixed effects logistic regression providing point 
and interval estimates of outcome odds ratios associated with study group. This analysis 
technique accounts for the within-facility correlation among patients to obtain valid estimates of 
treatment effects and associated standard errors. The pre-specified set of covariates for 
inclusion in regression models include: study group and covariates used in the randomization 
procedure (i.e. patient socioeconomic status (using the Neighborhood Deprivation Index), 
patient race/ethnicity, and service area level COVID-19 incidence). Study group and other 
covariates will be treated as fixed effects and the patient’s home facility and e-visit facility will be 
handled as random effects. 
  
In addition, exploratory/sensitivity analyses will examine the impact of including additional 
covariates in our estimation of treatment effect, with focus on variables with chance imbalance 
in distributions by treatment group, as determined in preliminary analyses.  Analyses resulting in 
an appreciable change in the estimate of treatment differences or increased precision of the 
treatment effect will be noted.   
 
For the secondary outcomes, we will compare the estimated cost per Paxlovid message, the 
minutes spent per message, and the hours from e-visit to prescription order between the 
Pharmacist Care and AFM Pool Care groups using linear mixed regression models, providing 
point and interval estimates of between-group differences in mean outcomes.  If cost or minutes 
have a skewed distribution, log transformation will be used. Analyses of other secondary 
outcomes with that are continuous will be similar. The approach to inclusion of covariates and 
sensitivity/exploratory analyses is as described for Aim 1.   
 
Patient satisfaction and other care experience scores will be compared between the study 
groups using linear mixed effects regression with log transformation if needed for skewed data, 
with analyses paralleling that described for the other secondary outcomes.  

9 
 
 
For all aims, secondary analyses will assess variation in the estimates of treatment effect by 
age, race/ethnicity/language group, and comorbidity score group (high, medium, low) via 
stratification by level of the potential effect modifier.  These stratified analyses will inform the 
fitting of regression models with appropriate cross-product terms between the potential effect 
modifier and study group indicator variable, allowing for a more formal assessment of 
heterogeneity in treatment effect.  
 
A service area or facility may drop out before completion of the study if they were assigned to 
AFM Pool Care but add local pharmacist management, thus diverging from their assigned 
group. Paxlovid e-visits from such a service area or facility will be censored as of the dropout 
date (the date that local pharmacist management began).  
 
Tests of significance will be two-tailed with an alpha of .05. No Bonferroni or other formal 
correction for multiple comparisons will be used, but results will be interpreted with the multiple 
testing issue in mind.    
 
4.8. Sample size and power. To estimate the available sample, we extrapolated from April 2023. 
This was a month when COVID-19 incidence was low to medium; 4900 e-visits for Paxlovid 
were made across the 15 KPNC service areas. Based on this, an average of 1089 e-visits occur 
per week during low to medium COVID-19 incidence periods. However, e-visits that need to be 
answered on weekend days will not be eligible for pharmacist management; all of these will be 
excluded from the study. If 5 out of 7 e-visits occur on weekdays, an estimated 778 e-visits 
occur on weekdays during times of low to medium COVID-19 incidence.  
 
This study will include 17 facilities that serve 872,200 adult members, or 25% of the 3.55 million 
adult members in the region. Each week, the estimated number of e-visits eligible for study is 
25% of 778, or 195. Assuming an 8-week evaluation period, the estimated sample size is 1,560, 
or 92 patients in each of the 17 clusters.  
 
Given the group-randomization study design, calculation of sample sizes and minimum 
detectable treatment effects must account for the expected intraclass correlation (ICC) of 
observations within facilities.  The ICCs in outcomes of interest are expected to be quite small, 
and therefore we present minimum detectable effects for an expected ICC of .01.   
 
For Aim 1, the primary outcome is the % of patients with common potential DDIs who received 
counseling. We conservatively estimate that 70% of patients in the Pharmacist Care group will 
have received DDI counseling. A clinically meaningful difference between the two groups would 
be a relative risk (RR) of 0.8, which corresponds to a small effect size. To attain 80% power to 
identify such an effect with a two-tailed alpha of 0.05 and an ICC of .01, we need a sample size 
of 250 in each group and will plan our review of text extracts from the EHR accordingly.  
 
For Aim 2, the example hypothesis is that the cost of management per e-visit for Paxlovid will be 
lower in the Pharmacist Care group than in the AFM Pool Care group. Given a two-tailed alpha 
of .05, the sample size of this study has 80% power to identify an effect size of 0.20 standard 
deviation units, which is a relatively small effect size. In other words, this study will have good 
power to find a statistically significant difference between the Pharmacist Care and AFM Pool 
Care groups even if the effect is relatively small.  
 
For the patient survey in Aim 3, the sample size is selected to balance power and feasibility. We 
aim to collect completed surveys from 200 patients in each of the two study groups. We 

10 
 
estimate a response rate of 60%. To obtain 200 completed surveys in each group, we estimate 
needing to attempt surveys with 333 patients in each group. The completed sample size of 200 
in each group will give us good power to identify relatively small differences in satisfaction 
scores, which are measured on a continuous scale. We will be able to detect effect sizes of .31 
relative to a standard deviation of 1. 
 
The purpose of the pharmacist and AFM surveys is process assessment and performance 
improvement. These surveys will focus on qualitative, open-ended responses; thus, sample size 
and power are not estimated here. We will survey all pharmacists on the regional team 
providing Paxlovid e-visits (estimated n=10). We also will survey the AFMs who are Paxlovid e-
visit site leads (estimated n=12). 
 
4.9 Timeline 
 
Date 
Operational team 
Research team 
Sep 25 
Pilot care delivery to the service 
areas to receive Pharmacist Care; 
run-in period 
Not included in study 
Oct 9  
Continue care delivery at steady 
state 
Start of study period 
- 
Conduct review of text extracts from 
the EHR (random sample 
throughout period) 
- 
Conduct patient surveys (random 
sample throughout period) 
- 
Conduct pharmacist and AFM 
surveys (mid-Nov. to early Dec.) 
Dec 11  
End study – pharmacists cover all 
service areas 
End of study period 
Dec 11 
– Jan 12 
 
Analyze computerized data of outcomes for 
Aims 1 and 2  
Complete data collection with clinicians and 
patients  
Jan 12 – 
Feb 2 
 
Finish cleaning and analysis of data from 
clinicians and patients 
Feb 2 –     
Create reports for internal use 
Share findings with other TPMG and KFH 
leaders 
Draft manuscript for journal submission 
 
 
4. Projected Impact 
 
This study’s findings quality of care, cost, and patient and clinician satisfaction will inform 
decisions about how to deploy pharmacist effort for KPNC. The study leaders include regional 
leaders for Virtual Medicine and Pharmacy who are well-positioned to make implementation 
decisions based on the results we generate.  
 
5. Deliverables and Dissemination Plan 
 

11 
 
Our deliverables and dissemination plan have two purposes: To provide information for 
operational decisions by TPMG and KFH, and to establish our reputation as leaders in Desktop 
Medicine and Pharmacy Services through publication. Specific deliverables and plans include: 
 
• 
The PI will prepare a presentation prior to study initiation to support conversations with 
leaders of service areas that will be included.  
• 
Soon after the analysis of computerized data is completed, we will create an internal 
report for Drs. Lee and DeLaunay to share via presentation to the appropriate chiefs and 
other operational leadership groups.  
• 
The analysis of clinician and patient data will be completed after this, and we will update 
the internal report to support a follow-up communication.  
• 
We will write a manuscript for submission to a peer-reviewed journal to share our 
findings with health care leaders in the U.S. and beyond.  
 
 
 
 

12 
 
References 
1. Akbar F, Mark G, Prausnitz S, et al. Physician Stress During Electronic Health Record Inbox 
Work: In Situ Measurement With Wearable Sensors. JMIR Med Inform. Apr 28 
2021;9(4):e24014. doi:10.2196/24014 
2. Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: Primary Care Physician 
Workload Assessment Using EHR Event Log Data and Time-Motion Observations. Ann Fam 
Med. Sep 2017;15(5):419-426. doi:10.1370/afm.2121 
3. Tai-Seale M, Dillon EC, Yang Y, et al. Physicians' Well-Being Linked To In-Basket 
Messages Generated By Algorithms In Electronic Health Records. Health Aff (Millwood). Jul 
2019;38(7):1073-1078. doi:10.1377/hlthaff.2018.05509 
4. Akbar F, Mark G, Warton EM, et al. Physicians' electronic inbox work patterns and factors 
associated with high inbox work duration. J Am Med Inform Assoc. Apr 23 2021;28(5):923-
930. doi:10.1093/jamia/ocaa229 
5. Lieu TA, Warton EM, East JA, et al. Evaluation of Attention Switching and Duration of 
Electronic Inbox Work Among Primary Care Physicians. JAMA Netw Open. Jan 4 
2021;4(1):e2031856. doi:10.1001/jamanetworkopen.2020.31856 
6. Lieu TA, Altschuler A, Weiner JZ, et al. Primary Care Physicians' Experiences With and 
Strategies for Managing Electronic Messages. JAMA Netw Open. Dec 2 
2019;2(12):e1918287. doi:10.1001/jamanetworkopen.2019.18287 
7. Moulton LH. Covariate-based constrained randomization of group-randomized trials. Clin 
Trials. 2004;1(3):297-305. doi:10.1191/1740774504cn024oa