Evaluating a Patient-Centered Tool to Help Medicare Beneficiaries Choose Prescription Drug Plans
Phase: NA
Conditions: Prescription Drug Insurance Decision Making
Sponsor: Stanford University

1 
 
 
 
 
 
 
Evaluating a Patient-Centered Tool to Help 
Medicare Beneficiaries Choose Prescription Drug 
Plans (CHOICE) 
NCT02895295 
September 23, 2016 
M. Kate Bundorf, Principal Investigator 
Stanford University 
Stanford, California 94305 
 
 

2 
 
1. Protocol 
A. Abstract 
To obtain publicly subsidized insurance coverage for prescription drugs, Medicare beneficiaries must 
choose among the private plans offering coverage in their geographic area. While Part D has reduced 
beneficiary spending on and lowered prices for prescription drugs, Medicare beneficiaries consistently 
report that choosing a plan is difficult, time consuming and unpleasant. 
In this project, we test the effects of a patient-centered Medicare Part D prescription drug decision tool 
that is scalable and patient-centered and builds on research indicating that many people would like 
expert assistance in making this choice.  Our main objectives are to determine whether providing 
Medicare beneficiaries with a patient-centered decision tool and whether providing an expert 
recommendation of a particular plan improve decision outcomes relative to providing them with access 
to publicly available tools.  
 
B. Specific Aims 
Aim 1:  To determine whether providing Medicare beneficiaries with a patient-centered decision tool to 
help them choose among plans improves outcomes for patients including a greater likelihood of 
changing a plan, better coverage for prescribed drugs, lower decisional conflict when choosing plans, 
and greater satisfaction with the choice process relative to current practice. 
 
Aim 2:  To determine whether providing people with personalized expert plan recommendations 
improves outcomes for patients including a greater likelihood of changing a plan, better coverage for 
prescribed drugs, lower decisional conflict when choosing plans, and greater satisfaction with the choice 
process relative to current practice and relative to a similar tool without such recommendations. 
Aim 3:  To determine what types of older adults are likely to use an internet-based tool for health plan 
choice. 
C. Study Overview 
In this study, we will test the effectiveness of two versions of a web-based tool to help people choose 
among Part D plans (Treatments A and B) relative to each other and relative to standard care (Control).  
Both treatment arms will incorporate simplified design and automated importation of an individual’s 
drugs relative to standard care. The treatment arms will vary based on whether they provide expert 
guidance on recommended plans. In the control arm, study participants will be directed to the existing, 
publicly available Medicare.gov website and will receive instructions on how to download their drugs 
from the Palo Alto Medical Foundation (PAMF) patient-facing EMR (myhealthonline). Our study sample 
will include PAMF patients who were enrolled in Part D plans during the 2016 enrollment period.  Prior 
to the 2017 open enrollment period (October 15 to December 7, 2016), we will invite a subset of PAMF 
patients enrolled in Part D plan to participate in a study examining the effectiveness of decision tools 
that provide personalized information on the financial implications of enrolling in different Part D plans.  
For aims 1 and 2, the primary study outcomes will be: 
1) Plan switching 
2) Change in generosity of coverage of prescription drugs 

3 
 
3) Satisfaction with the choice process 
4) Decisional conflict 
Secondary study outcomes for aims 1 and 2 
5) Enrollment in an expert-recommended plan 
6) Difficulty choosing a plan 
7) Confidence in choosing a plan 
8) How much do you like the plan you chose 
9) Satisfaction with chosen plan (people found this difficult) 
10) Knowledge of the Medicare Part D prescription drug benefit? 
 
For aim 3, we will analyze how the following patient characteristics influence enrollment in the trial and 
take-up of the treatment among those randomized to the treatment arm: 
1) Prescription drug profile 
2) Age 
3) Use of myhealthonline 
We will measure the primary study outcomes using a combination of administrative data and a post 
open enrollment survey administered to all study participants.  We will also collect information on 
individual characteristics at the time of enrollment in the study and implement a survey examining use 
of the intervention tool to assess patient experience at the time of use. 
 
D. Background 
Medicare beneficiaries consistently report that choosing a plan is difficult, time consuming and 
unpleasant. In our preliminary qualitative work, older adults said that these difficulties cause them to 
seek assistance from trusted authorities like brokers and counselors and sometimes even to avoid 
changing their plan altogether. 
Research on plan choice is consistent with these observations.  People are often not enrolled in the plan 
that would cover their drugs at the lowest price and rarely change plans even when there are 
alternatives that provide coverage at a lower cost.  In addition, brands have strong effects on individual 
choices, and it is not clear whether particular brands provide real value for consumers or simply serve as 
a decision short-cut. 
Research has also shown, however, that personalized information on the financial implications of 
enrolling in different plans can influence plan choice. Using a relatively simple, paper-based 
intervention, Kling et al. demonstrate that, when people received a letter with personalized information 
about the cost of enrolling in different plans, they were more likely to switch plans than those who 
received information about the availability of this type of information on the Centers for Medicaid and 
Medicare Services (CMS) website.  While this trial shows potential for decision tools to improve 
outcomes in this area, many questions remain on how best to provide this type of information to 
consumers.  Although publicly available tools are available to Medicare beneficiaries to help them make 
these types of decisions, evidence suggests that they rarely use this source of information. 

4 
 
One potential barrier is that the currently available tools are difficult for beneficiaries to use.  In our 
ethnographic research observing Medicare beneficiaries using the Medicare.gov website, we identified 
two limitations of the existing tool:  1) it is difficult for beneficiaries to customize the information with 
their personal drug histories, and 2) the most commonly used tool (Medicare.gov plan calculator) is 
difficult for people to navigate and use.  We have developed a tool that addresses these barriers by 
importing beneficiaries’ information about drug use directly into the tool and by creating a patient 
interface that incorporates modern design concepts and patient input to make the tool accessible and 
effective for beneficiary decision making. 
In our ethnographic study, older adults also expressed interest in having additional support, often 
indicating they would like the help of an expert, such as an insurance broker or other type of 
knowledgeable individuals.   
Our study is based on the hypothesis that these barriers create switching costs for beneficiaries and that 
removing these barriers would lower switching costs making beneficiaries more likely to evaluate their 
alternatives and to enroll in plans that provide better coverage for the drugs they are likely to use.  We 
also hypothesize that reducing switching costs, particularly by providing expert recommendations, will 
reduce decisional conflict and create greater satisfaction with the choice process.   
While the tool we have designed in partnership with patient stakeholders is easier to use than the 
Medicare.gov website, it provides less detail on how the different plans cover particular types of drugs.  
In our qualitative work examining the use of the first version of our tool, we found that its simplicity was 
more attractive to some people than others. Thus, we will examine particular characteristics of 
individuals including numeracy, “do-it-yourselfness”, and satisficer/maximizer behavior influence how 
people evaluate the decision process in the different tools. 
A final issue in this context is whether simply providing access to web-based decision support tools is 
likely to assist the types of people who experience the greatest difficulty in decision-making.  We will 
examine the take-up of decision support in this setting to determine how well targeted it is by analyzing 
who chooses to enroll in our trial and among those who enroll, who uses the study intervention. 
 
E. Participants 
Our study population will include PAMF patients aged 66 to 85 who were enrolled in Part D coverage 
during 2016. We plan to have a final study sample of at least 915 people with complete data on study 
outcomes. 
F. Screening 
We will prescreen patients using administrative to identify potential study participants based on 
whether they are enrolled in Medicare, their age (66-85), whether they live in the primary PAMF service 
area (county=Santa Clara, San Mateo, Santa Cruz, and Alameda), and whether they have active 
medication orders in the PAMF EMR.  We will also exclude patients enrolled in MediCal or a Medicare 
Advantage plan in 2016.  In preliminary analyses, we have identified 46,525 PAMF patients who meet 
these criteria.   

5 
 
An important sampling issue for our study is whether people are married or cohabitating.  Randomizing 
people who are cohabitating to different treatment arms could lead to cross-arm contamination, 
potentially diluting the treatment effects.  We will prescreen patients for cohabitation/marital status 
based on patient residence and phone number.  We will then randomly choose one member of each 
household to participate in the intervention.  In our preliminary analysis, we excluded an additional 
7,659 people using this method.  Because this represents fewer people than we would have expected 
based on national rates of marriage among older adults, we will also screen for cohabitation when 
recruiting.  Again, based on our preliminary analysis, there will be 38,866 remaining patients eligible for 
recruitment. 
G. Recruitment 
We will recruit a subset of those patients eligible for recruitment with a goal of obtaining complete 
study data from 915 patients.  We will identify a set of patients for recruitment by randomly choosing a 
subset of approximately 10,500 patients from those who are eligible for recruitment. For the patients 
targeted for recruited (recruitment sample), we will send them a letter in August introducing the study 
and inviting them to participate.  The letter will inform the respondent that they have been selected to 
participate in a study of the effects of providing personalized information on Medicare Part D 
prescription drug plans and that they will be randomized to different arms representing different ways 
of providing that information, that participation in the intervention will allow them to easily access 
information from PAMF on the prescription drugs they are currently using.  The e-mail will also inform 
the patients that they will receive a $50 electronic Target gift certificate at the end of the open 
enrollment period for participating in the study following the completion of a questionnaire. The letter 
will direct participants to a website (enrollment portal) where they can check if they meet the eligibility 
criteria and, if so, enroll in the study and provide informed consent.   
The initial invitation will include a study ID that the participant will use to enter the enrollment portal.  
This ID will be used as their study ID throughout the study and will be used to link data from their 
medical records to data gathered as part of the study.  The enrollment portal will have 4 functions: 
1. Eligibility verification - When patients log into the application, they will verify their name and 
address as well as their eligibility to participate in the study based on age, part d enrollment, 
and single representative from their household. (We will not exclude people from accessing the 
tool based on the single representative but will exclude them from data analysis). They will also 
identify the Part D plan in which they are currently enrolled.   
2. Informed consent 
3. Collect contact information -  Consented participants will provide their e-mail address, using 
double-entry verification, and will be informed that further communication regarding the study 
will be by e-mail rather than by mail. 
4. Pre-enrollment survey (Survey 1)  - We will ask a few questions intended to provide information 
on individual characteristics of interest for the purpose of data analysis. 
We will consider people who consent through the enrollment app as the “enrolled” population.  The 
enrolled population will be randomly assigned to one of three arms (Treatment A, Treatment B or 
Control). Within 2 days after subjects enroll, we will send them by e-mail a letter that includes their 
study ID and information on how to access the tool, including the location of the website, when it will be 
available and log-in instructions.  They will use the study ID to log into the website when open 

6 
 
enrollment begins.  Shortly before and close to the end of open enrollment, we will send them another 
reminder with the same information.   
We will provide support to study participants in the form of an e-mail address and a phone number they 
can call.  This information will be available in our study tool, the control arm screen and the consent 
form.  We will indicate that they will receive a response within 48 business hours. 
Table 1:  Recruitment Objectives 
Sample Name 
Category 
Assumptions 
Number of 
Study Subjects 
Eligible for Recruitment 
Eligible for 
recruitment 
38,866 
Recruited Sample 
Chosen for mail 
recruitment 
Derived from assumptions below 
10,417 
Check for Eligibility 
20% of those who contacted by mail 
check their eligibility 
2,083 
Enrolled Population 
Enroll in Study 
Assume 40% attrition among those 
who access enrollment app 
1,250 
Study Sample 
Complete final 
survey 
Assume 80% of those enrolling 
complete final survey 
1,000 
Objective based on sample 
size calculation 
based on 3 arms of 305 each 
915 
Source:  trial protocol tables google sheet – recruitment objectives 
We will identify a set of patients for recruitment by randomly choosing a subset of approximately 10,500 
patients from those who are eligible for recruitment.  We estimate that 20% of those contacted will 
choose to access the enrollment portal.  Among those who access the enrollment portal, we estimate 
30% attrition based on ineligibility or lack of consent.  Finally, we estimate that 60% of those who enroll 
will complete the final survey in January.  The table above identifies the effects of each these 
assumptions on our final sample size.   
We estimate the size of the final study sample will be 1,000 people with 333 assigned to each arm.  Our 
sample size calculations indicate that we need 305 people per arm to have adequate sample size (see 
detail under section “assessments”). 
H. Randomization 
We will randomize those who enroll in the study into two treatment arms and one control arm in which 
they will be directed to different on-line tools.  Participants will be randomized at the time they enroll by 
simply assigning them sequentially to the three arms as they consent. 
 
I. 
Retention 
While our study period is relatively short, only lasting throughout the open enrollment period, a key 
issue in retention in encouraging people to participate in a survey at the end of the open enrollment 

7 
 
period.  Study participants will receive their $50 gift certificate from Target after completing the post 
open enrollment survey in January. 
 
J. Intervention 
The study has two types of interventions.  The primary intervention is access to a decision tool (Aim 1) 
and the second is access to a decision tool with expert recommendations (Aim 2).  People who enroll in 
the study will be randomized to one of three arms – two treatment and one control arm.  People 
randomized to the treatment arms will receive a one of two versions of an on-line decision tool that 
automatically imports information on their prescription drug use from an electronic medical record into 
a decision tool developed as part of the study.  In Treatment Arm A (expert recommendations), the tool 
will provide people with a score for each plan using an algorithm for generating expert 
recommendations based on the individual’s prescription drug usage and the characteristics of the 
different plans.  In Treatment Arm B (individual analysis), we will provide information on each of the 
characteristics underlying the expert recommendation but not the recommendation itself.  In the 
control arm, people will receive information on how to download their prescription drug information 
from the electronic medical record and how to access the Medicare.gov Plan Finder tool. 
The calculations underlying the calculator as well as the expert recommendations will be powered by 
software we have purchased from a private third party, 
.  
 produces software that allows a 
user to enter electronically a set of prescription drugs and returns information on associated out-of-
pocket spending and the premium for each Part D plan available to the beneficiary. 
 also provides 
a “score” which represents a weighted average of the person’s estimated out-of-pocket spending and 
premium and the plan’s consumer satisfaction rating.  Our user interface will, in turn, present this 
information to study participants in a patient-centered way and will manipulate the information in the 
two study arms as described above. 
Participants will not actually enroll through our calculator.  Instead we will direct participants to the 
appropriate company website or to the Medicare website where they can enroll. 
The decision tool will be available throughout the open enrollment period. 
K. Assessments   
The study outcomes will be based on both primary data collection and administrative data sources.  The 
sample size was chosen to adequately power the primary study outcomes. 
Table 2:  Summary of Outcome Measures for Comparative Effectiveness Evaluation 
Note:  Primary study outcomes are italicized and bold.   
Outcome Type 
 
Measures 
Data Source 
Patient Satisfaction 
Aims 1 
and 2 
• 
Satisfaction with choice process 
 
 
Post-enrollment 
survey (primary)  
Plan Choice 
Aims 1 
and 2  
• 
Change in Part D plan chosen 
 
2016 plan from 
enrollment app and 
2017 plan from post 

8 
 
open enrollment 
survey. 
Also question in 
post enrollment 
survey about “did 
they change their 
plan?” 
Aims 1,  
and 2 
• 
Indicator of enrollment in lowest cost 
plan* 
• 
Indicator of enrollment in expert chosen 
plan* 
 
2017 plan from post 
open enrollment 
survey and 
information on OOP 
cost from 
 
collected in admin 
panel.  
Spending on 
prescription drugs 
Aims 1, 
and 2. 
• 
Difference in spending between lowest 
cost plan and chosen plan* 
 information 
for all plans and 
indicator of which 
plan was chosen 
from post open 
enrollment survey. 
Aims 1 
and 2 
• 
Difference in spending between 2016 
plan and 2017 plan based on initial drug 
list.* 
 
Note: this address whether they switched to 
a lower cost plan 
 
 information 
for all plans and 
indicator of which 
plan was chosen 
from post open 
enrollment survey 
and which plan they 
were initially 
enrolled in from 
enrollment app. 
Choice Experience 
Aims 1 
and 2 
• 
Decisional Conflict 
• 
Knowledge about Part D program  
• 
Sources of information used  
• 
Time spent making decision  
Post-enrollment 
survey 
 
• 
How satisfied were you with the process 
of choosing a plan? 
• 
Decision confidence 
• 
Decision conflict 
• 
Choice knowledge – understanding of 
features of chosen plan 
• 
Right amount of information 
• 
Define user monitoring outcomes 
Intervention survey 
* Note that calculating these variables requires a drug list for each person.  For all people, we can 
construct an initial drug list based on the data in the EMR.  For a subset of people who we observe using 
the tool, we can construct a revised drug list based on their activity in the tool.  All these outcomes are 
based on the initial drug list.  
 

9 
 
L. Data Management 
See data flow chart document. 
 
M. Study Timeline 
May 31 – Large dataset of test data 
June 30 – UI changes implemented  
July 15 – refreshed study data available to developer 
July 31 – administrative console implemented 
August 1 – Invitations mailed to participant 
August 1 – October 14 – recruitment and enrollment 
August 1 – website available with “coming soon” information 
August 1-August 30 – complete testing and list of changes to developer 
September 30 – all changes implemented 
October 1 – generate dataset of all 
 information for everyone chosen for recruitment. 
October 1 – October 12 – final verification of functionality 
October 15 – e-mail reminder (exact date depends on when tool is refreshed with 2016 open enrollment 
information 
October 15 – December 7 – Open enrollment – tool available 
December 8 – e-mail follow up survey 
January 4 – e-mail reminder about survey 
December 8 – January 15 – follow up survey open 
January 16 – e-mail gift cards 
 
2. Statistical Plan 
Sample size calculations: 
Hypothesis 
Outcome 
Relative to 
control, 
treatment 
groups will 
have: 
Baseline 
rate 
SD 
Change Power 
N per 
group Notes: 

10 
 
Plan 
switching 
Higher rates 
of plan 
switching 
13.6 
8.704 
80% 
305 
Kling et al find 28% of people in 
treatment group relative to 17% 
in control. Ketcham et al 2013 
find relative high rates of 
switching among new enrollees - 
33% between 2009 and 2010 
and that switching rates 
increased over time. KFF (2103) 
finds 13.6 among all ages in 
PDP plans. Change is Kling 
effect of 65% change applied to 
KFF baseline rate. 
Satisfaction 
with choice 
process 
Greater 
satisfaction 
with the 
choice 
process 
2.6 
1.03 
0.6 
80% 
50 
Scale of 1-4 in preliminary study 
with 1 as very satisfied. Data 
represent means of medicare 
versus all vendus arms where 
medicare is the baseline. 
Decisional 
conflict 
Less 
decisional 
conflict 
Coverage 
enrollment in 
plans that 
coverage 
current drugs 
more 
generously 
368 
286 
74 
80% 
236 Based on Kling et al 
Coverage 
(alternative) 
Increase in 
coverage 
generosity 
from last 
years to this 
years plan 
-129 588.42 
-543 
80% 
20 Based on Fall 2015 Results 
 
Analyses: 
For aims 1 and 2, we will compare outcomes between each treatment group and the control group, 
defining the study population as those who enroll in the study through the enrollment portal and are 
randomly assigned to a study arm.  The main focus of our analyses will be the comparison of the Expert 
arm relative to the control arm.  We will calculate differences in means and proportions using t-tests 
and chi-squared tests based on the type of study outcome.  We will also test the expert arm relative to 
the treatment arm without expert recommendations to identify the independent effect of expert 
recommendations.  We will develop unadjusted estimates of the treatment effect as well as estimates 
adjusting for individual characteristics from survey and administrative data. 
For aim 3, we will analyze how individual characteristics affect whether people choose to participate in 
the trial (take up of our invitation to participate).  The key study variables will include age, use of 
myhealthonline, and prescription drug profile.