Pharmacist Management of Attention Deficit Hyperactivity Disorder Medication Refill Requests: A Randomized Clinical Trial
Phase: NA
Conditions: Attention-deficit Hyperactivity
Sponsor: Kaiser Permanente

1 
 
 
 
 
 
 
Pharmacist Management of Attention Deficit Hyperactivity Disorder  
Medication Refill Requests 
 
Study Protocol and Statistical Analysis Plan 
 
Tracy Lieu, MD, MPH, Daniel Parry, MD, Andrew deLaunay, PharmD, E. Margaret Warton, 
MPH, Eric Smallberg, MA, Michelle Mancha, BA, Stephanie Prausnitz, MS, Charles 
Quesenberry, PhD, Mary Reed, DrPH, Kristine Lee, MD 
 
From The Permanente Medical Group (Division of Research, Virtual Care Team, Adult and 
Family Medicine regional leads, and TPMG Consulting Services) and Pharmacy Services, 
Kaiser Permanente Northern California, all in Oakland, CA  
 
 
Version date: August 27, 2024 
NCT06388694 
 
 
 

2 
 
Abstract 
 
Importance. Enhancing the management of messages from patients and providing high-quality, 
consistent care are top priorities for The Permanente Medical Group (TPMG) and Kaiser 
Foundation Hospitals/Health Plan (KFH/P). A special extension of the Ryan Haight Act currently 
enables providers to prescribe controlled substances via telehealth interactions but expires in 
December 2024. Information about the quality of care provided via telehealth prescribing of 
controlled substances by pharmacists and primary care physicians would be helpful to inform 
care delivery within Kaiser Permanente and beyond.  
 
Objective. To compare pharmacist management of refill authorization requests for attention 
deficit hyperactivity disorder (ADHD) medications with physician management regarding quality 
of care, efficiency of service, and parent satisfaction.  
 
Design, setting, and participants. This cluster randomized clinical trial will include parents of 
children in Kaiser Permanente Northern California who request ADHD medication refills via 
secure messages from April 29 to June 28, 2024. Of KPNC’s 63 facilities eligible for inclusion, 
we will assign 32 to Pharmacist Care and 31 to PCP Care.   
 
Intervention. In the intervention group, a regional team of pharmacists will manage electronic 
refill authorization requests for ADHD medications using a standard protocol. In the comparison 
group, pediatricians will manage these visits using a similar protocol.  
 
Main Outcomes and Measures. The primary outcome is whether a patient who did not have a 
weight recorded in the 6 months before the refill request was referred for a primary care follow-
up visit. Secondary outcomes are the days from the secure message request to the prescription 
order and medication fill, and parent satisfaction. 
 
Potential Results. We will test the hypotheses that Pharmacist Care compared with PCP Care 
will have higher quality of care, faster time to prescriptions and fills, and higher parent 
satisfaction.   
 
Potential Conclusions and Relevance. If pharmacist care for ADHD medication refill requests 
has better or similar outcomes compared with PCP care, this will provide evidence supporting 
continuation of this approach. This study’s findings will be useful for KPNC and to inform 
discussions about renewing the special extension of the Ryan Haight Act that allows this 
approach. 
 
1. Specific Aims and Hypotheses 
 
Enhancing message management and virtual urgent care options for patients are among TPMG 
and KFH’s top priorities. As team-based inbox management is further developed, evaluating its 
clinical effectiveness and outcomes in terms of patient care experience and costs will provide 
valuable information for operational planning. In addition, sharing what we learn with other 
health care systems through talks and publications will help highlight TPMG’s reputation as a 
group that innovates to support clinicians while increasing patient access through virtual care.  
 
This study will compare the clinical outcomes and costs of two alternative modes of 
management of parent requests for refills of medications for attention deficit hyperactivity 
disorder (ADHD): management by a regional team of pharmacists, vs. management by primary 
care physicians. Patients will be included if they are younger than 18 years and if their parent’s 

3 
 
secure message request was sent to a pediatrician or family medicine physician (but not to a 
psychiatrist). We will conduct a cluster randomized trial in 63 facilities in Kaiser Permanente 
Northern California to compare Pharmacist Care with Physician Care for patients with ADHD 
medication refill requests with regard to: 
 
Aim 1. Quality of care 
  
Example hypothesis: Patients without a weight recorded in the past 6 months will be 
more likely to receive referral for a follow-up weight check in the Pharmacist Care group 
than the Physician Care group.  
 
Aim 2. Timeliness of care  
 
Example hypothesis: Patients managed via Pharmacist Care will have a faster time to 
refill prescription than those managed via Physician Care. 
 
 
Aim 3. Parent satisfaction 
 
Example hypothesis: Parents in the Pharmacist Care group will report comparable 
satisfaction with the process compared with the Physician Care group. 
 
2. Background and Significance 
 
During the COVID-19 pandemic, the federal Drug Enforcement Agency (DEA) and Health and 
Human Services (HHS) created a temporary special flexibility (under the Ryan Haight Act) to 
allow clinicians to prescribe refills of controlled substances via telemedicine interactions. Prior to 
this, patients needing refills of controlled substances, including medications for attention deficit 
hyperactivity disorder (ADHD), had to make in-person doctor’s office visits for refill requests.  
 
Thanks to this special flexibility, Kaiser Permanente Northern California has been able to offer 
patients and parents the option of requesting medication refills via secure messages and 
structured refill authorization requests sent via the electronic health record (EHR) portal. To 
assist busy primary care physicians (PCPs) who would receive these requests in their inboxes, 
pharmacists have taken charge of replying to refill requests for all adults aged 18 years and 
older. Starting in April 2024, the regional pharmacist team will also start to taking charge of 
replying to refill requests sent to pediatricians and family medicine physicians for children and 
adolescents younger than 18 years starting in April 2024.  
 
The special flexibility granted by the DEA and HHS to allow refills of controlled substances via 
telemedicine is set to expire on December 31, 2024 if not renewed. The current study will 
evaluate the effectiveness of providing pharmacist care for ADHD medication refill authorization 
requests for children and adolescents. It will generate information that helps KPNC enhance our 
quality of care, and will also contribute to broader national evidence on the effectiveness of 
virtual care approaches for such needs.  
 
Appropriate management of ADHD medication refills is a quality measure in the Healthcare 
Effectiveness Data Information Set specified by the National Committee for Quality Assurance. 
Specifically, the HEDIS measure specifies that children between 6 and 12 years of age who 
have had a prescription for ADHD medication and have remained on the medication for at least 
210 should have at least 2 follow-up visits with a practitioner in 9 months. A key purpose of such 

4 
 
follow-up visits is to check the child’s weight to ensure that their weight gain is not falling behind 
associated with ADHD medication use.   
 
3. Innovation 
 
Pharmacist management of secure messages from patients is innovative in not having been 
adopted or evaluated at large scale. Preliminary anecdotal and unpublished information from 
within TPMG suggests that pharmacists’ scope of practice is well-suited to message 
management for many specific conditions based on treatment algorithms. At our KP San 
Francisco medical center, pharmacists have been able to take charge of messages for a range 
of issues, including refill authorization requests and requests to manage chronic conditions 
(hypertension, osteoporosis, attention deficit disorder, gout) and time-limited conditions 
(sexually transmitted infections, latent tuberculosis, opioid tapering, and assessment of Paxlovid 
eligibility).  
 
4. Approach  
 
4.1. Design, setting, and participants.  
 
This cluster randomized clinical trial will assign parent secure message requests to pediatricians 
or family medicine physicians for ADHD medication refills for children aged <18 years to either 
Pharmacist Care or Physician Care. The unit of randomization is the KPNC facility. KPNC is an 
integrated health care system with 4.5 million members served by more than 9700 physicians 
and 90,000 employees in 21 hospitals and 207 medical offices. It has a mature EHR 
(EPIC/HealthConnect, adopted circa 2008) and encourages members to use a wide array of 
options for outpatient care, including secure messages to physicians via the EHR portal, e-
visits, video, telephone, and in-person office visits.  
 
The target population is children and adolescents aged <18 years whose parents send a refill 
authorization request (RAR) for an ADHD medication to their child’s pediatrician. An RAR is a 
highly structured electronic communication that does not allow the requestor to add any free 
text. This structure makes it possible for either a pharmacist or physician to handle the RAR.  
 
The regional pharmacy team started to handle free-text secure message requests for refills from 
parents of patients with PCPs in pediatrics during a pilot period starting April 11, 2024. Secure 
messages are different from RARs because they contain free text. These were found to be 
difficult to flag and send to the regional pharmacy team because accurately identifying the 
relevant secure messages required manual handling. The rate of such messages that could be 
identified per day would not have reached the intended sample size for this study. Thus, this 
study was designed to focus only on RARs, which are much simpler to identify consistently. The 
regional desktop medicine leader sent an email to the pediatric department at every facility to 
ask that during the study period, they ignore RARs from facilities randomized to Pharmacist 
Care, but continue to handle RARs from facilities randomized to Physician Care.  
 
The formal study period will include patients with secure messages sent to the regional 
pharmacy team from May 6, 2024 to June 21, 2024 (7 weeks). The unit of analysis will be the 
RAR, allowing some patients to appear up to two times in the study cohort. 
 
4.2. Intervention.  
 

5 
 
All procedures will be conducted as part of standard of care. In the Pharmacist Care group, 
each RAR for an ADHD medication will be assigned to a pharmacist on the regional team. 
Pharmacists will review requests using a standard protocol. They will call parents via telephone 
to elicit additional information where needed, file ADHD medication prescriptions where 
appropriate, and send follow-up secure messages. 
 
In the PCP Care group, each ADHD medication RAR will be sent to the PCP. Physicians will 
review requests using the standard regional protocol (as well as any existing medical center-
level protocols). Following this protocol, physicians call patients via telephone to elicit additional 
information where needed, file ADHD medication refill prescriptions when appropriate, and send 
follow-up secure messages. 
 
Due to system technical constraints, during this study it will not be possible to withhold RARs 
assigned to the Pharmacist Care group from the inboxes of the patients’ PCPs. To ask 
physicians to not handle RARs assigned to the Pharmacist Care group, the regional director of 
desktop medicine sent an email to all pediatric chiefs at the start of the study period explaining 
the study and asking them to cascade the message to pediatricians to avoid handling RARs at 
the facilities selected for the Pharmacist Care group.  
 
4.3. Randomization. Of the facilities in KPNC, 63 are eligible to be included. One facility (South 
Sacramento, which does not use this virtual care approach) will be excluded. The eligible 
facilities were assigned at random to either the Pharmacist Care or PCP Care arm. The goal of 
the randomization assignment plan was to balance on these variables: 
• 
Number of estimated ADHD refill requests (based on ADHD med refill data) 
• 
Race and ethnicity 
• 
Sex 
• 
Neighborhood deprivation index 
• 
Age 
 
We used covariate-constrained randomization, a method to prevent imbalances on important 
variables that may be associated with the outcomes.7 We conducted preliminary analyses to 
characterize facilities with regard to the above variables. We have chosen these variables for 
use in the restricted randomization given the expected association with study outcomes, along 
with the expected level of between facility variability which would increase the likelihood of 
between-arm imbalance.  
 
Using these variables, we enumerated all possible outcomes of randomization and eliminated 
all allocations of the facilities to the two groups that do not meet specified between-group 
balance criteria.  Standard diagnostics were applied in selecting the covariate balance criteria, 
ensuring no threats to the validity of the study design, including extreme reductions in total 
possible allocations for the trial, and instances of cluster pairs never/rarely or always/often 
randomized to the same arm.  An allocation of facilities to the two arms for implementation was 
randomly selected from the set of allocations that meet the final covariate balance criteria.     
 
The balance criteria were: maximum 5% between arm difference in total membership, 30% 
difference in total Black, 20% for each of White, Asian and Hispanic race/ethnic groups, and 
0.25 standard deviation difference in mean NDI.  
 
4.4. Main outcomes and measures 
 

6 
 
4.4.1. Outcomes 
 
The primary outcome will be measured only among patients who have not had a weight 
recorded in the EHR during the 6 months (180 days) prior to the ADHD medication refill request. 
We will determine whether the prescribing clinician attempted to ensure that the child had a 
weight checked or recorded, counting the outcome as present if (a) the patient was referred for 
an in-person visit for a weight check (including a well child visit or a nurse visit for a weight 
check), or (b) had a parent-reported weight recorded in the EHR. This outcome will be assessed 
via analysis of EHR data and chart review. Referral for a visit or weight check (part a above)will 
include any of the following: 
(a) the EHR record or chart review indicates that an appropriate in-person appointment was 
booked with pediatrics, family medicine, or psychiatry after the pharmacist or PCP 
handled the message;  
(b) a chart note indicates that the pharmacist or physician handling the RAR instructed the 
parent to schedule a follow-up appointment for a weight check; 
(c) a note in the EHR indicates that a medical assistant or other staff member had 
attempted to book a follow-up appointment for a weight check 
 
Secondary outcomes will be the: 
 
a. Whether the child had a weight recorded within 30 days after the RAR, regardless of the 
reason for the weight being measured (this measure would count weights recorded at 
urgent visits)  
  
b. Hours from the secure message request to the prescription order being placed, among 
patients who had a prescription ordered 
 
c. Days from the secure message request to the prescription being filled (sold), among 
patients who had a prescription ordered 
 
d. Parent satisfaction with care, as measured by standardized survey items 
 
Exploratory outcomes will be: 
 
a. Whether the provider initially handling the request was able to fully resolve it without 
referring it to another provider (first-touch resolution) 
 
b. Whether the patient had a refill prescribed (% of patients with refills prescribed) 
 
4.4.2. Predictors and covariates 
 
The primary predictor is study group, i.e. Pharmacist Care or PCP Care. Covariates are other 
variables that could potentially mediate, moderate, or confound the association between the 
primary predictor and the outcomes. In this study, covariates include patient age, sex, race, 
ethnicity, language spoken, and comorbidities, as well as the patient’s neighborhood-level SES.  
 
4.5. Data collection.  
 
The primary outcome will be measured via analysis of EHR records and chart review, as 
defined above.  
 

7 
 
For the secondary outcomes that involve medication prescription orders and fills, and for the 
exploratory outcomes, we will use data available from KPNC computerized sources, including 
HealthConnect (HC, aka EPIC) EHR data (from Clarity) and EPIC access logs. For each patient 
with a refill request, EHR data will be used to identify all ADHD medication prescriptions ordered 
and filled (sold) during the study period.  
 
Patient care experience will be evaluated by surveying a sample of parents selected at random 
in both the Pharmacist Care and PCP Care groups. We have designed a short survey that takes 
no more than 5 to 10 minutes to complete. The survey will be fielded by email, mail, and 
telephone. Due to the fact that we are asking about a specific care experience and patients’ 
recall fades quickly after 2 weeks, we will field the survey using all 3 modes in a relatively 
efficient sequence. A letter containing a description of the study, language regarding informed 
consent and confidentiality, and the survey will be sent by mail on day 7 after the secure 
message has been replied to, and an email with a link to the online version will be sent on day 
11. Telephone administration will be attempted (up to 4 attempts total) starting on day 14 for 
those who have not replied to the paper or online survey. Only the first RAR will be eligible to 
generate a survey request. A gift card valued at $10 to thank patients for considering 
participation will be included in the mailing. Responses will be treated as confidential and data 
will be aggregated, with no individual identifiers associated with data presented in reports.  
 
Parent-reported outcomes will be reported on a 5-point scale and will include:  
a. Perceived effectiveness of the initial reply in addressing the request 
b. Perceived quality of care for the request overall 
 
We will also ask parents to indicate the mode by which they would prefer to make future ADHD 
medication refill requests, e.g. secure message or a telephone, video, or in-person appointment.  
 
4.6. Statistical analysis plan. 
 
The primary analysis will treat each patient as part of the group to which their facility was 
assigned (intention-to-treat). However, some patients in the Pharmacist Care group might have 
unintentional crossover and be handled by physicians instead of pharmacists. For this reason, 
we will conduct a secondary analysis in which patients are grouped based on the initial 
management approach they actually received (pharmacist or physician, based on which type of 
clinician first contacted the patient), similar to a per-protocol analysis. An RAR first accessed by 
a pharmacist and later accessed by physician will be counted in the pharmacist group.  
 
The primary outcome is binary, i.e., whether or not a patient who had not had a weight check in 
the 6 months before the refill request is referred for an appropriate follow-up visit. This will be 
compared between study groups using mixed effects logistic regression providing point and 
interval estimates of outcome odds ratios associated with study group. This analysis technique 
accounts for the within-facility and within-patient correlation to obtain valid estimates of 
treatment effects and associated standard errors. The pre-specified set of covariates for 
inclusion in regression models include: study group and covariates used in the randomization 
procedure (i.e. patient socioeconomic status (using the Neighborhood Deprivation Index), 
patient age, sex, race and ethnicity, and comorbidity score). Study group and other covariates 
will be treated as fixed effects and facility will be handled as a random effect.  
 
In addition, exploratory/sensitivity analyses will examine the impact of including additional 
covariates in our estimation of treatment effect, with focus on variables with chance imbalance 
in distributions by treatment group, as determined in preliminary analyses. Analyses resulting in 

8 
 
an appreciable change in the estimate of treatment differences or increased precision of the 
treatment effect will be noted.   
 
For the secondary outcomes, we will compare the days from the secure message request to 
prescription order or medication fill between the Pharmacist Care and PCP Care groups using 
linear mixed regression models, providing point and interval estimates of between-group 
differences in mean outcomes. If the numbers of days have a skewed distribution, log 
transformation will be used. Analyses of other continuous secondary outcomes will be similar. 
The approach to inclusion of covariates and sensitivity/exploratory analyses is as described for 
Aim 1.   
 
For patient satisfaction, care experience scores will be compared between the study groups 
using linear mixed effects regression with log transformation if needed for skewed data, with 
analyses paralleling that described for the secondary outcomes. For analyses, categorical 
responses will be combined into a binary variable (Excellent or Very Good vs. Good, Fair, or 
Poor).  
 
For all aims, secondary analyses will assess variation in the estimates of treatment effect by 
age, sex, socioeconomic status (using the Neighborhood Deprivation Index), and race and 
ethnicity, via stratification by level of the potential effect modifier.  These stratified analyses will 
inform the fitting of mixed effects regression models with appropriate cross-product terms 
between the potential effect modifier and study group indicator variable, allowing for a more 
formal assessment of heterogeneity in treatment effect.  
 
A facility may drop out before completion of the study if they were assigned to PCP Care but 
add local pharmacist management, thus diverging from their assigned group. ADHD medication 
refill requests to such a facility will be censored as of the dropout date (the date that local 
pharmacist management began).  
 
Tests of significance will be two-tailed with an alpha of .05. No Bonferroni or other formal 
correction for multiple comparisons will be used, but results will be interpreted with the multiple 
testing issue in mind.    
 
4.8. Sample size and power. To estimate the available sample, we extrapolated from data 
analyzed during the pilot period in April 2024.  
 
Based on initial data in the pilot period, we estimate that the message pool handlers will send 
the regional pharmacy team 7 secure messages for straightforward ADHD medication refill 
requests per weekday, i.e., 35 eligible messages per week. We estimate based on expert 
opinion that 30% of these patients don’t have a weight check in the previous 6 months. 
Assuming a 9-week evaluation period, the estimated sample size is 315 patients, or an average 
of 5 patients in each of the 63 clusters.  
 
Given the group-randomization study design, calculation of sample sizes and minimum 
detectable treatment effects must account for the expected intraclass correlation (ICC) of 
observations within facilities.  The ICCs in outcomes of interest are expected to be small, and 
therefore we present minimum detectable effects for an expected ICC of .03.   
 
For Aim 1, the primary outcome is the % of patients without a weight check in the past 6 months 
who received referral for an appropriate follow-up visit. The following estimates assume that 
30% of the 315 patients with secure messages will have not had a weight check in the past 6 

9 
 
months, for a sample size of 95 (63 clusters with 1.5 patients each). We also assume a desired 
power of 80%, a two-tailed alpha of 0.05 and an ICC of .03. If 50% of patients in the PCP Care 
group will receive an appropriate referral, the least detectable difference will 77%. If 80% of the 
patients in the PCP Care group receive an appropriate referral, the least detectable difference 
will be 98%. Thus, this sample size yields good power to identify only large differences between 
the two arms in the primary outcome.  
 
For Aim 2, the outcome is days from the secure message to the prescription order. The 
following estimates assume a sample size of 315 (63 clusters with 5 patients each), a desired 
power of 80%, a two-tailed alpha of 0.05 and an ICC of .03. The least detectable effect size for 
a comparison of means will be 0.334. In other words, this study will have good power to find a 
statistically significant difference between the Pharmacist Care and PCP Care groups for a 
small to medium effect size. 
 
For Aim 3, the outcome is perceived quality of care on the parent survey, measured on a 5-point 
Likert scale. We will attempt to survey all 315 parents in each of the two study groups. We 
estimate a response rate of 60%, for a completed sample of 189 across groups. The outcomes, 
effectiveness and quality scores, are measured on a continuous 5-point scale. This sample size 
will yield 80% power to detect effect sizes of .0.37 (relative to a standard deviation of 1). Thus, 
the survey will have good power to identify medium sized differences in perceived effectiveness 
and quality of care. 
 
4. Projected Impact 
 
This study’s findings quality and timeliness of care and patient satisfaction will inform decisions 
about how to deploy pharmacist effort for KPNC. The study leaders include regional leaders for 
Virtual Medicine and Pharmacy who are well-positioned to make implementation decisions 
based on the results we generate.  
 
5. Deliverables and Dissemination Plan 
 
Our deliverables and dissemination plan have two purposes: To provide information for 
operational decisions by TPMG and KFH, and to establish our reputation as leaders in Desktop 
Medicine and Pharmacy Services through publication. Specific deliverables and plans include: 
 
• 
The PI will prepare a presentation prior to study initiation to support conversations with 
leaders of service areas that will be included.  
• 
Soon after the analysis of computerized data is completed, we will create an internal 
report for Drs. Lee and DeLaunay to share via presentation to the appropriate chiefs and 
other operational leadership groups.  
• 
The analysis of clinician and patient data will be completed after this, and we will update 
the internal report to support a follow-up communication.  
• 
We will write a manuscript for submission to a peer-reviewed journal to share our 
findings with health care leaders in the U.S. and beyond.  
 
 

10 
 
Appendix A. Power calculations 
 
Est. # eligible, per week (7 per day to pediatrics) 
35 
Est. # eligible over: 
9 weeks 
315 
# of facilities (clusters) 
63 
Cluster size 
5 
Least detectable 
differences 
Assumptions: Power = 0.80, alpha = 0.05 
For proportions, P0=0.50 
ICC = 0.01 
0.657 
ICC = 0.03 
0.663 
ICC = 0.05 
0.679 
For proportions, P0 = 0.80 
ICC = 0.01 
0.912 
Surveys  
  
ICC = 0.03 
0.915 
Response rate  
0.6 
ICC = 0.05 
0.919 
N 
189 
N per cluster  
3 
For means,  
  
  
ICC = 0.01 
0.322 
  
0.412 
ICC = 0.03 
0.334 
  
0.42 
ICC = 0.05 
0.346 
  
0.427 
 
 
Ref: University of Aberdeen cluster sample size calculator