Survey of VHA Clinical Reminde

Emily S. Patterson, PhD

VIReC Clinical Informatics Seminar

November 12, 2009
Acknowledgements

 Steve Asch—PI Emil

y Patterson—PI

 Connie Fung—First author on survey journal artic Jerry Tsai—Statistician and Primary programmer Armine Lulejian Armine Lulejian

—Former project leader Former project leader

 Peter Glassman—Consultant Brad Doebbeling—Consultant Barbara Simon Barbara Simon

—Survey consultant Survey consultant

 Mingming Wang—Programmer Joya Golden—Technical support J Sl ason 

S

aleem—L d h b ti l 

Lea

d researc

her on o

bservational 

 Laura Militello-Human Factors Consultant VA HSR&D Grant #TRX 02-216 (Asch/Patterson)
Goals for Our Time Toge



To describe results of a national survey national survey

 To discuss 

experience with method integration
Background

 Computerized clinical reminders

 Core VHA tool (~100% penetration) Users: Primary care providers, intake  Automated reminders of tasks during  ‘Cheap’ alternative to External Peer Re

Program (EPRP) performance measur

 No representative national physicia
Patterson ES, Nguyen AD, Halloran JP, Asch SM: Human factors barriers to the effectiv

HIV clinical reminders. Journal of the American Medical Informatics Association 2004, 11(1):50.
Patterson ES, Nguyen AD, Halloran JP, Asch SM: Human factors barriers to the 

of ten HIV clinical reminders. Journal of the American Medical Informatics Association 11(1):50.
Survey Questions

1. What are VHA primary care 

physicians’ perceptions of clinreminders?

2. What are physician and facilit

predictors of a more favorablglobal assessment of reminde
Conceptual Framework

Design Factors:

Ease of Ease of

use/learning

Efficiency

Function

Organizational Factors:

Leaders as champions

Training

Support for maintenance

Interacti

with 

other too

Redunda

Support Depende

for maintenance

Economic constraints

Time constraints

Workload

Depende

Other intera

Overall Usefulness, Usability, & 

Satisfaction With Reminders

Team Factors:

Coordination

Role design Individual 

Contextual Factors:

Situation-specific

Patient-specific Factors:

Expertise

Attitudes

Patient

Other Physician and Facility-

Adapted from Patterson ES, Nguyen AD, Halloran JP, Asch SM: Human factors barriers to the HIV clinical reminders. Journal of the American Medical Informatics Association 2004, 11(1):50.
Methods

 Cross-sectional national survey  March 2005 through October 2

March 2005 through October 2

 Samp g lin frame: “Personnel an

Accounting Integrated Data” d

 Stratified random sample Stratified random sample

 4 sites over-sampled: GLA, Cincinn

I di li d Mi li Indianapolis, and Minneapolis

 Other sites: random sampling fract
Methods

 Data collection: 3 waves 

 Web (n=403, 71% of respondents) Paper (n Paper (n 98 = 17% of respondents) , 17% of respondents) Telephone (n=69, 12% of respond

 Eli ibilit i i 

Eligibility: primary care specia

> half-day clinic, have used re

 Weighted response rate=69% 

 Four sites 66%; other VHA sites 69

Four sites 66%; other VHA sites 69
Scales constructed from su

 Design/interface Inte

gration with workload/work

 Clinical/situational specificity Clinical/situational specificity Self-efficacy P id l erceive

d role

 Sources of trainin

g

 VA management of reminders Global assessment Global assessment
Physician-Level Categorical Variables Weighted Freque

1) Length of VHA service● <5 

years 47%

● 5 to 9 years 23%● 10 to 14 years 11%● >15 years 18%● Missing 0%2) Specialty● Internal medicine 82%● Geriatrics 7%● Family practice 11%3) Male 59%● Missing 0%4) Has academic appointment 4) Has academic appointment 55%● Missing 3% 
Physician-Level Categorical Variables Weighted Freque

5) Self-reported use of reminders● Al i d ways use remin

ders 73%

● Sometimes 18%● O i ll l Occasionally or rarely 9%● Never 5%Physician-Level Continuous Variables Median (IQR)6) Number of hal

f-da

y p s of direct 

patient care 9 

(5-10

)

● Missing 0.43%7) Years since medical school graduation 19 (11-27)● Missing 0.21%
Facility-Level Categorical Variables Frequency (%) N

7) Academic affiliation● Yes 61%● Missing 21%8) Located in metropolitan area 81%● Missing 0%9) Facilities with 3 or fewer physicians represented in 

sample

87%

Facility-Level Continuous Variables Median (IQR)10

) py () Number of 

primar

y care visits 

(FY'04

) 148, (, 000 

(65,000 - 29

● Missing 1.02%
Global assessment ratings are only in the mid-r

Variable Name n Mean SD MediGlobal assessment 458 11.5 4.5 12.0 

● Overall satisfied with reminders 460 3.9 1.8 4.0 ● Overall reminders are effective 460 4.3 1.7 4.0 ● Overall reminders are not more 460 3.2 1.8 3.0 

useful in principle than they are in practice

•Item response ranges from 1 to 7, where 1=”strongly disagree” and 7=”•Scale response range 0-21 
Integration with workload/workflow only in m

Variable Name n Mean SD MedIntegration with workload/work flow 460 9.5 4.2 10.

● Enough time to complete 

reminders under typical clinical 

460 3.0 1.8 3.0 

workload

● Reminders do not unnecessarily 

duplicate information in my

461 3.1 1.9 3.0 

duplicate information in my

progress notes

● Total number of reminders is not 

t l

460 3.4 1.8 3.0 

•Each item res

ponse ran

ges from 1 to 7, where 1=”stron

gly disa

gree” an

too 

lar

g

e

p g , gy g 

•Scale response range 0-21
Poor clinical/situational specificity

Variable Name n Mean SD MediaClinical/situational specificity 452 11.3 4.4 11.0 

● Reminder dialo

g boxes provide 

appropriate options for MD to resolve reminder

457 3.3 1.6 3.0 

● Most reminders apply to MD’s 

patients

460 3.5 1.7 3.0 

● Adding

“Not Applicable Not Applicable

” would 458 2 0 1 4 1 5 

not improve use and effectiveness of reminders

458

2.0

1.4

1.5

● Adding “Pending” o ld not Adding “Pending” w

o

uld not 455 2 5 1 7 2 0 

improve use and effectiveness of reminders

455

2.5

1.7

2.0

•Each item response ranges from 1 to 7, where 1=”strongly disagree” and •Scale response range 0-28 
Pretty good self-efficacy; confident about compute

Variable Name n Mean SD MeSelf-efficacy 446 45.1 8.4 4

● Reminders help MD provide care Reminders help MD provide care 459 4 7. 1 8. 5 ● Feels comfortable using reminders 457 5.3 1.5 6● R i d k MD d ti Remin

ders ma

ke MD more pro

ductive 459 4 1. 2 0. 4 

● Recovers quickly when makes mistake 

using reminders

455 4.4 1.8 4

● Enough workstations are available 461 5.8 1.5 6● Computer speed sufficient to use 460 4.1 2.0 4

reminders

● Has proficient computer skills to use 

reminders

460 6.0 1.5 7

● Prefers to use computer while with 

patient

461 5.2 1.9 6

● Makes no notes on paper to use later to 460 5.5 1.8 6

•Each item response ranges from 1 to 7, where 1=”strongly disagree” and 7=”stron•Scale res

ponse ran

ge 0-63

● Makes no notes on paper to use later to

complete reminders
PCPs know who does the reminders—they 

Variable Name n Mean SD MePerceived role in reminder use 459 9.7 3.3 1

● Knows exactly which reminders Knows exactly which reminders 461 4 9 1 9 6 

responsible for completing

461

4.9

1.9

6

● Views reminders as part of core 

work activity

459 4.8 1.8 5

•E h i f 1 7 h 1 ” l di ” d 

Eac

h item response ranges 

from 

1 to 

7, w

here 

1

=

”strongly disagree

” an

d

•Scale response range 0-14
Many sources of training that are helpfu

Many sources of training that are helpfu

Variable Name n Mean SD MedSources of training help MD learn 

i d

451 16.3 5.6 16

rem

i

n

ders

● Training sessions 457 4.2 1.9 4● Online documentation 453 3.7 1.8 4● Performance feedback 456 4.2 1.8 4● Other clinical staff 457 4.2 1.8 4

•Each item response ranges from 1 to 7, where 1=strongly disagree and 7=•Scale response range 0-28 
VHA plays active role in increasing reminder 

Variable Name Variable Name

n Mean SD Media

Management role 460 4.6 1.8 5.0 ● VHA managing of reminders 

increases my completion of reminders

460 4.6 1.8 5.0 

* Each item response ranges from 1 to 7, where 1=strongly disagree and 
Design/interface slightly above mid range

Design/interface slightly above mid

-range

Variable Name n Mean SD MeD i /i f Desig

n/interface 448 25 1. 7 2. 2

● Easy to use most reminders Easy to use most reminders 459 3.9 1.9 4● Easy to learn how to use reminders 461 5.1 1.6 5

● Expected functions and capabilities are 

available 459 3.5 1.8 3

● Formats easy to use 458 4.4 1.6 4● N t i d b ti f N

o

t surprise

d by actions o

f some 

reminders 452 4.1 1.5 4

● Information on reminder screen is 

•Each item response ranges from 1 to 7, where 1=strongly disagree and 7=strongly •Scale response range 0-42

presented pleasantly 457 4.1 1.4 4
Study Question 1: What are 

Study Question 1: What are 

perceptions of reminders?

● Global assessment ratings are only in the mid-range ● I t ti ith kl d/ kfl l i id I

n

tegration with workload/workflow only in mid-range

● Poor clinical/situational specificity● P d lf retty goo

d self-effi fid b kill fficacy; confident a

bout computer skill

s

● PCPs know who does the reminders—they do●

Sources of ii hlfl f training are 

h

elp

ful

● VHA plays active role in increasing reminder use● Desi /i f li h l b id ign/inter

face sli

g

h

tly a

bove mid-range 
Study Question 2: What are physician 

facility-level predictors of a more favoglobal assessment of reminders?

 overall satisfaction, perceived 

effectiveness, and perceived use

MD Characteristics ++ 

Facility characteristics MD perceptions (scales)

+ So

Academically- affiliated West

Self-efficacy Integration with workflow/load Training p<0.05Design/interface 
Survey Limitations

 Facility/clinic-level variation (but model a

for clustering) for clustering)

 Assessed overall reminder process Cli i l l i f h PCP’ i Cli

nical complexity o

f eac

h PCP’s practice

 Staff physicians; no nurses, NPs, PAs, or  Informatics infrastructure, performance m

culture, incentive structure

Ælimited gene

beyond VHA beyond VHA
Goals for Our Time Toge



To describe results of a national survey national survey

 To discuss 

experience with method integration
Survey Reviewer Comm

 “Use of an unvalidated self-report meas “A lit ti t d thi t i i ht 

“A qualitative s

t

u

dy on this 

topic might 

deeper understanding of the underlying phenomena”

 “No data on actual use of CCRs were us

analysis; only limited data on self-reporwas included was included

”

 “It is unclear...how these findings can b

to im

prove the use of reminders”

 “Were there any qualitative data collect

suggestions for how the system could bi d?” mproved?”
‘Qualitative’ Method Myt

 Identical methods can be used at multiple  Findings from a few sites generalizes acro Can get reliable frequency data on system  Findings directly inform how to improve de Can predict how much a design change or Can predict how much

a design change or

intervention will impact performance



Anal

yses can include both micro & macro 

 Studying a site where a system is in use c

all im

plementation hurdles at another site
Integration Lessons Lea

 Timing not a barrier in survey design Issues of sub Issues of sub

-populations not worth populations not worth ‘real e

real e

(But...we had no open-ended responses) 

 Phys c a s o y (a d o es de ts) o de 

icia

n

s

onl

y (a

n

d n

o reside

nts) in 

o

rde

nationally representative sample

 Over-sampling 4 observational sites had lo Self-report not appropriate for perception 

maybe undesirable behavior), but reasonaadoption/usefulness/usability/satisfaction/

 Too difficult to ask about desirability of de

desi

g g n chan

ges or other interventions
Triangulated Findings: B

Barrier to CR Use Survey: 

National VHA 

(Physicians)

Observations: 

8 HIV sites 

(Physicians)

Observations: 

4 Outpatient 

Sites (Mixed)

Survey: 

Camp CPR

(Mixed)

Workload Yes Yes Yes Yes Integration with workflow

Yes Yes Yes N/A 

Ease of use Somewhat Yes Yes Somewhat Learnability/Training

Somewhat Yes Somewhat Yes 

Clinical/situational specificity

Yes Somewhat Somewhat Somewhat 

PCP-Patient Relationship

N/A Somewhat Somewhat N/A 

Unclear responsibility

No No Yes N/A 

Adoption No No No Reminderdependent 
References

 Fung CH, Tsai JS, Lulejian A, Glassman P, Patterson ES

BN, Asch SM. Computerised Clinical Reminders Use In Healthcare System. The Journal on Information Techn

Healthcare; 2009;7(3):144-159.

 Fung CH, Tsai JS, Lulejian A, Glassman P, Patterson ES

BN, Asch SM. (2008). An Evaluation of the Veterans Administration Administration s’ Clinical Reminders System: Clinical Reminders System: A Nationa

A Nationa

Generalists. Journal of General Internal Medicine, HeaTechnology supplement, 23(4): 392-398.
Questions?