USH Award for Outstanding Achievement in Health Services 

Research 

April 28, 2010 

Alan M. Garber, A.B., A.M., Ph.D., M.D., Introduction of Mary K. Goldstein:

It’s a real pleasure and honor to introduce Mary as she receives the VA's highest honor for 

health services research. Her impressive academic accomplishments are undoubtedly the 

chief reason for this award, but to my mind, she's an inspirational figure for many other 

reasons as well. And as all who know her in Palo Alto can testify she's a dedicated 

mentor, a superb clinician, a beloved colleague, and she has made deep and valued 

contributions locally in Palo Alto, also to Stanford and to the nation as a whole. 

One of the things that people may not really appreciate about Mary is that she had been 

headed in this general direction for a long time but she could hardly be said to have 

pursued a linear path. After graduating from medical school, she did residency in family 

practice and went directly into a faculty position, family practice at UCSF and subsequently 

moved to Stanford. And she was a rising star with a number of academic 

accomplishments. Not long thereafter in fact, she became a member of the board of 

directors to the American Board of Family Practice as well as their vice president. And she 

also became certified in geriatrics and was elected to the board of directors of the 

American Geriatrics Society which she served on from 1996 to 2002. That in fact 

overlapped with her service on the family practice board which I really suspect was 

unprecedented. 

But to her great credit, although Mary was clearly headed in a direction toward a 

tremendous academic success, she came to think that she really needed to reboot in the 

sense that she needed to acquire greater research skills. She had developed a strong 

interest in End-of-Life Care and in the use, or rather under-use of advanced directives and 

had done really good work but she fell limited by her methological tools. That is her 

methological training. And so, she approached me for advice about how she might 

acquire deeper research skills, as a very junior faculty member. She then applied for a 

fellowship program in healthcare and policy research based at Stanford. She enrolled in 

the fellowship program, pursued a master’s degree in health services research while 

staying clinically active and, by the way, also raising the two daughters at the same time. 

I was Mary's formal adviser when she took this step, and needless to say I was always 

impressed with her dedication and her ability to learn quickly and to apply what she 

learned. 

And everything that she did was leavened by her deep clinical knowledge, clinical skills, 

and clinical practice. And she ended up evolving her research and as Seth mentioned she 

is someone who has worked for a long time in healthcare IT issues--and her first 

experience was developing multimedia computer-based preference assessment tools. 

And this lead later to research on decision support systems for clinical care leading up to 

Athena, the expert system to improve hypertension management; and that as many of you 

know is incorporated into CPRS. After her training and throughout this period, Mary 

formally entered faculty positions at Stanford and at the VA, rising within Stanford to 

become a professor of medicine and health research and policy. At VA, she's had a 

number of positions including chief of the section of general internal medicine. And later 

on, she became head of the geriatrics fellowship program and most recently, she took on 
the leadership of the Palo Alto GRECC, the geriatrics research and education and clinical 

center. 

Throughout all this time, Mary had served as a mentor to many of our residents, 

undergrads, grad students, post docs, trainees at all levels in other words. And she has 

helped every one of them in ways that many, many have come to me to tell me about and 

to sing her praises. She's been a great colleague and I have to tell you that over the years 

the only advice that I have tried to give her, rather unsuccessfully I might add, is that she 

needs to do a little bit less. She is generous to at fault and contributes greatly for every 

kind of situation that calls for leadership insight and perseverance. She's the person who 

can be called on to lead. I won't try to summarize her research because I know that that's 

what she will be talking about. But let me just say that Mary's integrity and generosity 

have been assets to the VA as well as to her local colleagues. I'm extremely pleased to 

introduce the 2010 winner of the undersecretary’s award for outstanding achievement in 

health services research, Mary Goldstein. Thank you.

Mary K. Goldstein, MD, MS, Palo Alto VA Medical Center Physician: Award acceptance 

speech and presentation

Wow. Thank you so much. I just--thank you so much Alan and Seth. And also let me 

thank Gerry McGlynn and Molly for setting this up and I just feel very honored and very 

humbled by this. So let me just launch in here. The first thing I wanted to say is that I--

well, I guess, is that we're doing this as a live meeting rather than with a live audience 

because there's no HSR&D meeting this year. But we do have a small studio audience 

here. So this, what I think we have is a Radio Days performance. And there, folks are 

here with us in Palo Alto which I'm very thankful for. 

I also wanted to say that I have a close friend, Emily, who at this moment is extremely ill. 

And her husband Tom is with her every moment and close friends are coming frequently. I 

was able to see her recently and I wanted to dedicate this talk to Emily who is much in my 

mind these days. 

So I have so many people to thank but first I'd want to mention mentors and Alan, who you 

just heard from, who has a number of roles that I won't repeat, has been someone who 

started me out by taking a chance on me. When I was a clinician educator in Geriatrics, 

who had a new found interest in developing research skills plus some school-age children 

as he mentioned, he offered me a research fellowship position and he has supported me 

as a mentor ever since then. Alan created an environment in which I was able to thrive as

well as many others I've seen around me thriving in the environment he created. There 

was the course work in HSR, of course, but also topnotch faculty actively participating in 

the research and progress seminars to hone our skills, and an atmosphere of people who 

are superb investigators, who shared their knowledge and skills and advice with us. He's 

been my division chief since I came to the faculty and also a mentor and collaborator. He 

helped me prepare for and carry out a Career Development Award and in enumerable 

other ways, helped launch and maintain my career, and I thank him very much for that. 

The next person I wanted to thank is another mentor, Brian Hoffman. Brian is a molecular 

pharmacologist and an unusual mentor for health services research person, but Brian's an 

unusual person in many ways. In addition to being a serious basic scientist, he's also 

someone who has a very serious overall interest in mentoring and encouraging junior 

investigators and clinician educators. He's an outstanding clinician, as I know from 

working with him in the hypertension clinic and from discussing many patient cases in 
detail over the years. He's devoted to the veterans. Brian encouraged me early on to 

think about ways to improve care for hypertension and he collaborated closely on 

development of the knowledge base for BP (ph) in a hypertension system I'll discuss later. 

Brian also taught me, in a typical Brian way, that in research as in hockey, you don't want 

to be where the puck is now. You want to be where the puck is going to be. And I think 

that has a lot say about how we ended up starting some research in health informatics 

when most people we talked to about it couldn’t quite understand what we were talking 

about and ending up with 10 years of work at a point where now there's a lot of interest in 

it. 

So--also I want to give thanks to many collaborators, mentors and colleagues who are 

listed on this slide. And I don't have time to describe all of them, I wish I could. Phil Lavori

for a lot of time and statistical support; Terry Blaschke, Ted Shortliffe for informatics; 

colleagues and collaborators, Michael Gould, Paul Heidenreich, Laura Carstensen, Doug 

Owens, Joel Tsevat, Yuval Shahar, informaticist; Lorene Nelson, Kate Lorig, Ingram Olkin

who worked with us on our group visits project; Judy Walsh, you'll hear mentioned later as 

well as Carmen Peralta; site PIs for multi-site studies and many of the staff at the Center 

for Healthcare Evaluation which is the HSR&D center of excellence at VA Palo Alto, 

including Ruth Cronkite, Denise Daniels, John Finney, Susan Frayne, Rudy Moos, Jodie

Trafton and others. 

And then a number of trainees I've had the real privilege to work with, some of whom are 

listed here will be mentioned during this talk and others, too many to be able to list all of 

them. I'd also like to thank VA Office of Information and Technology, especially Jeff 

Shyshka and Eric Raffin and Doug Wirthgen, VA HSR&D and VA Palo Alto Health Care 

System for supporting our work. And then our research team members for specific 

projects and especially Samson Tu, knowledge modeler par excellence and wonderful 

person, Mark Musen who heads Stanford Biomedical Informatics Researcher; Dena 

Bravata, who I met during her first week of residency at Stanford and came to know over a 

period of years, including her fellowship, and she'll be mentioned later in some of the work; 

colleagues at Durham and Duke, Durham VA, Eugene Oddone and Hayden Bosworth, 

and then some of our research team pictured here and some here in this room, and I thank 

them all so much. 

So, it's customary in these talks to have a little bit that's--about something about where it 

all began. And none of us really knows how we got to be who we are, but we construct 

some story about that from bits and pieces of memory. And here's a picture of my parents 

on the day they became engaged. My father was a captain in the Army in World War II, 

within North Africa and Italy--through a number of campaigns--worked as a surgeon at a 

forward field hospital and after he came back, my parents married and my father set up his 

practice. My mother has been a strong role model. Before marriage, she had an 

executive position, but in keeping with the times and my father's wishes, she stopped work 

outside the home after marriage to take on what I think was something like a CEO position 

within our family, raising seven children. Here I am with my brothers and sisters. And 

being the sole person managing my father's practice, and no matter how hard I work--and I 

do work very hard--I often think of how my mother, who may be listening on the phone to 

this, set an example of how hard she worked in support of her family. So, it's a wonderful 

family, I'm close to my brothers and sisters. And growing up in a large family, I learned a 

lot about teamwork, negotiation and collaboration, as you might imagine. And I thank 

them all for that. 
So, moving along to how this all began, I grew up in 1950s America. So I was part of a 

culture and a time when successful women were expected to be wives and mothers. My 

father and grandfather--that's my mother's father--were both surgeons and medicine was a 

part of our life, with the phone ringing at night all the time as my father rushed out to do an 

appendectomy or pin a hip, but it was also very much a man's world and not something 

open to women. I went through a personal transformation along with much of the rest of 

the country during the social upheavals of the 1960s and 70s. This had not yet started 

when I started college at age 16 and I was very interested in Math, but could not imagine it 

being feminine to study Science or Math and ended up becoming a Philosophy major 

which seemed like the right mix of formal logic with its relationships to Math and plus 

textual exegesis as Literature, plus Moral Philosophy and Ethics. 

I worked at the computer center at Columbia University during a break from college and I 

started there in the very feminine position of editor and writer. But in that position, I had 

the opportunity to work with a lot of people who were the technical people providing all of 

the--what was a very cutting edge computer system at that time and my job was 

understanding what they were doing so I could write it clearly for the people who wanted to 

use the system. And I became something of a translator and in the course of doing that, I 

was able to learn programming and learn a lot about system architecture. I'm showing 

here the cover of a report from the project abstracts back in 1970 to 71 which was 

produced with Wilbur and Cribby (ph) that were cutting edge things at that time and very 

fun to work with.

I learned from that that in order for computers to be effective for the actual projects, they 

needed to have both the technical excellence but also, there needed to be some ability to 

translate, to talk across the disciplines to bridge the gaps from one site to another. Using 

my newfound computer skills, I continued some work which helped support me during 

medical school with School of Public Health at Columbia in a center called something like 

Center for Alternative Healthcare Delivery Systems, which was really a forerunner of what 

we might think of as implementation today for new ways of bringing what's known to be 

good for healthcare out to people to put it into actual practice. Now, I don't usually speak 

about my background or these kinds of personal things. And so, in thinking about this to 

put it together for the talk today, it was a bit like tasting the maudlin (ph) and opening the 

floodgate of memories in a Prussian richese de pomperdue (ph); however, I will assure 

you I will not go on for 1,100 pages on this and we'll quickly move to more comfortable 

territory of describing some research. 

So, as Alan had mentioned, my first work in the--which I started during the fellowship with 

Alan's help and continued in my first Career Development Award had to do with eliciting 

patients' preferences in order to capture the benefits to patients of improving functional 

status appropriate to geriatrics as well as other areas. So that the patients' preferences 

could be incorporated into individual decision-making to fully value intervention that have a 

major impact on functional status and also to incorporate into a comparative cost 

effectiveness analysis or comparative effectiveness. Early programs we developed were 

in multimedia utility elicitation using the systems available at that time like this. We built 

this ADL, activities of daily living, index builder using super card and little pictures and 

voices. And then moved on to develop a system we called FLAIR, Functional Limitation 

and Independence Rating, and we used that to interview 400 older adults with an average 

age of 72 and who were computer inexperienced. And we collected a lot of information 

from them about their experience in using the computer to improve it as well as their 

ratings for the health state. And some of the findings included ways that we could improve 

utility elicitation. A study we did with Dena Bravata, it looked at our health state 
descriptions had nested health state of combination of impairments in activities of daily 

living. And this structure of nested health states allowed for exploration of consistency of 

the responses from one state to another. So we learned a lot about how to improve that. 

And we also looked at what was the difference between simply finding out how many ADL 

dependencies someone had compared to the importance to the individuals of which ones 

were important and which combinations were important. And working with Tamara Sims 

and Tyson Holmes, we developed simple counts of ADL dependencies do not adequately 

reflect older adults' preferences towards states of functional impairment. You have to find 

out not only what they can do but also what's important to them. 

We then went on to develop a later version of FLAIR that was more colorful and elaborate, 

and we used for interviewing another 600 adults. So I then added additional area of work, 

clinical decision support, and I'm going to describe some of the considerations in 

addressing the topic of implementing clinical practice guidelines. Working with colleague 

Judy Walsh at UCSF, and also with Kathy McDonald, Doug Owens and others from the 

Stanford UCSF Evidenced-based Practice Center. We did a systematic review of quality 

improvement strategies for hypertension and found importantly that strategy that included 

organizational team such as team change, had a great impact, and then it might be 

important to stop thinking only in terms of the position patient dyad but ways of bringing 

other team members into the care of patients. 

In thinking about next steps, we began to notice in the late 90's that there was an increase 

in the complexity of medical care. Patients often had multiple co-existing chronic 

diseases. Sometimes each disease requiring several medications so that the overall 

number of medication is high, the amount of clinical data generated within the electronic 

systems can be overwhelming, especially when patients have several diagnosis, and the 

medical literature is growing all the time. So we wanted to work to link information 

technology with clinical care and develop systems to have information highly tailored to the 

patient being seen and presented quickly to the clinicians within the work flow. 

So we thought about how things like blood pressure control are achieved, what are the 

things that clinician does, what are the things the patient does. And there are many 

factors that affect clinician prescribing and we wanted to add in to that mix of influences 

shown on the slide from marketing guidelines, journals, peers, et cetera to add something 

of clinical decision support based on evidence. But whatever the clinician prescribed, and 

it is a necessary condition to have correct prescribing so that the patient has the 

opportunity to follow the regimen, it is still the patient who then elects whether to follow that 

regimen. And there are many factors that affect the treated patients' blood pressure, 

including such things as their individual response to therapy competing comorbidities, 

lifestyle choices and medication adherence. And to address this side of the patient side, 

and I do believe it’s important to have both provider-directed and patient-directed 

intervention in order to optimally achieve evidence based practice, we did a project 

involving group medical visits with patients; which I won't have time to discuss any further 

but I did want to acknowledge the importance of addressing both sides of this issue. 

In focusing back on clinical decision support, we thought about what are the things that 

computers can do well that could help people. One of them is data visualization and there 

seemed to be a need for better data visualization in clinical care. Managing patients 

requires a collection and interpretation of large amounts of time-oriented data which 

require exploration at logical computation by the viewer and also searches for information. 

So it seems to target for potential help from data visualization. And one of the studies we 

did, and Susanna Martin, who's here, we studied a system that had been developed by 
Yuval Shahar on knowledge based navigation of abstraction for visualization and 

exploration and had clinicians answer queries, clinical queries, using this system 

compared to an excel spreadsheet, availability of the same data or a paper-based version 

and measured the time and the correctness of their answers. And we found that the 

clinician answered the complex queries faster and more accurately using the data 

visualization tool. So, from that we did move on to developing some visualization that 

could be shown within the clinical decisions support. One example is on the screen here. 

We also wondered if we're going to be putting something new, are we potentially 

interfering with the clinical practice in a way that might even be harmful, and are there risks 

when health information technology is put in to the clinical setting? New technology can 

introduce new opportunities for error or distraction. Some examples I've shown on this 

slide. So, we thought that we should address this by doing some cognitive task analyses 

of closely observing health professionals in the clinical setting, using new health 

information technology. And in one study with Chris Johnson and Roni Zeiger, grad 

students looked at the mismatch of the way in which physicians think and act while writing 

orders as compared to computerized decision order entry computer systems. And another 

study with Carol Ching, who is now Carol Kane, did a close observation of the initial 

introduction of computerized decision order entry in an intensive care unit and found, 

among other things, that the conceptualization of the workflow that was used by the CPOE

developers, was quite different from the actual workflow. And so, as shown here, the 

conceptualization was very linear--physician writes order, pharmacist verifies, unit serf 

delivers it, nurse administers it--while the reality in an intensive care unit is not linear. 

There are many more eddies and flows of the sequence in which things happen, and 

failure to recognize this in development of a system can lead to problems. "It's one thing 

to know what should be done; it's another to actually do it." I like this quote from the 

Merchant of Venice. "If to do were as easy as to know what we're good to do, chapels had 

been churches and poor men's cottages, princess palaces." And implementation research 

is all about doing as well as knowing. 

So, we moved on to try to design clinical decision support that could be used for doing an 

actual implementation with design goals of linking information technology with clinical care, 

providing tools for visualization, giving quick displays that would happen within the 

workflow, give evidence-based advice, account for clinical complexity--if the information is 

too simplistic, it's not meaningful for many patients--and then to monitor the clinician 

feedback to the system. So, we developed the system called ATHENA clinical decision 

support system. We started with hypertension because it's highly prevalent, has 

significant clinical impact, and there's a large evidence base for best practices. And we 

designed a model system with a plan for extension to other clinical domains if it was 

successful. We built our initial system in collaboration with Stanford University, built as a 

platform independent system for integration with different electronic health records later. 

We represented the clinical knowledge in computer interpretable format using Protégé, 

and information on this slide presents what Protégé is and how to access it. There isn't 

time to give a lot of detail about this encoding process. But to take a little bit of a look 

under the hood, this is something that is not intended for the end-users, but is rather part 

of what the developers will use, we the developers, use in building the system. And so the 

Protégé system allows us to build a knowledge base, a computer interpretable knowledge 

that is, nevertheless, easy to read, view and see in what looks like medical terms, and we 

can encode classes that define the concepts in the clinical domain and slots that define the 

attributes and relationship between the different classes or concepts. We can develop 

clinical scenarios in the pink squares that are scenarios that patients might be in clinically 
and then define a number of different alternatives from which the patient might go next, 

and then eventually potentially leading to medication or other recommendations. 

As a little example to show the level of clinical detail that can be represented this way 

which goes far beyond what's available in typical simple clinical reminders, we can get to a 

level of detail about does the patient have sinoatrial node dysfunction and does the patient 

have or not have a pacemaker in place, and can set up the message which is very easily 

changed in wording if there's a wish to change what will be said about it. In order to then 

take this encoded computer interpretable medical knowledge and move it to linkage with 

patient data, we need to map high-level concepts such as in guidelines like diabetes to 

specific patient data which is represented in patient standards such as ICD-9 codes or test 

names. And there are data sources now that define which of the many different potential 

data standards have been identified for use by HHF in federal healthcare systems and 

websites shown there. So we put the whole thing together by combining the clinical 

patient information with the encoded knowledge, as in the knowledge base I showed you, 

using the special program to process these together to generate an information display for 

clinicians about an individual patient which then, voila, can pop up on top of the electronic 

health record when the clinician accesses that patient's record. For those who are more 

interested, we could talk separately about the technical architecture and our newer user 

interface designed with a design company and a group of physicians around the country to 

whom I expressed appreciation. 

We learned in all of this the importance of working with stakeholders. What Marc Berg of 

medical sociology talks about as a socio-technical aspect in which you may have a highly 

technically well-designed system that can be a failure in implementation if there's not 

sufficient attention to all of the different stakeholders and the social aspects. We were 

able to implement this system in two different large studies and earlier three-site studies 

shown with black dots, generated displays to more than 50 providers at almost 10,000 

clinic visits. And a recent study in New England, represented by the red dots there, for 

which data analysis is now ongoing, an additional 11,000 clinic visits. So one issue was 

will clinicians use it? And in many other systems where clinical decision support is 

available, it's used only a tiny percent of the time; for example, in one study, 0.8 percent of 

visit opportunities. So we took a look at how much clinicians were using the system both 

by observing actual use through log data and through questionnaire and we're happy to 

find that there was extensive use. Some of which is shown by the frequent rate of entering 

a new blood pressure and clicking to update.

Mary, I do apologize for interrupting. I would like to let you know that our scheduled time 

limit for the session is coming to a close. However, if you are still available, we do have 

the advanced call line for the next 20 minutes and I would be pleased if you could stay 

around and finish your presentation.

Thank you, Molly. I will take a few more minutes to finish up here, since that’s okay with 

you and I thank you for that.

Great.

We have had some additional uses of the system since our single implementations with 

primary care providers. And this slide shows something Hayden Bosworth and Gina 

Downey (ph) have done, using it as part of a nurse protocol system for managing blood 

pressure adurem. We also can envision potential uses in the patient center medical home. 

And there are--I'm going to skip ahead here. So at each deficit, we do evaluation. There's 
offline testing prior to deployment, validation of data extracted at each site, testing in the 

clinics and so on. And we also recognize challenges for the future with patients having 

multiple problems and while each individual recommendation may have a good evidence 

base, much less is known about the affect of the large number of potentially interacting 

meds and there's a need for both coordination of guidelines and prioritization of 

recommendations. 

So what do we dream about here in clinical decision support world? We dream about 

implementing additional knowledge bases that we have already under development or in 

some cases, completed, including diabetes, chronic kidney disease, heart failure, chronic 

pain, which is work done with Denise Daniels, Jodie Trafton and others. And we love to 

create a library of encoded guidelines and to make this available online for lots of people 

to work on together collaboratively, shared both throughout the VA and potentially with 

other healthcare systems. We'd like to add clinical information from the free-text portions 

of the electronic record. Doctor Eisen spoke earlier about the VA CHIR and Da Vinci, 

we've been working with them and we look forward to be enabled to incorporate more of 

that type of information as it advances. We'd like to see development of one coordinated 

set of recommendation for each patient and finally, we'd like to make all this available 

directly to patients. You can use the same knowledge bases and processing techniques 

but with a different user interface suitable to patients. We’d love for--there are many 

patients who'd be interested in being able to access this type of information themselves. 

So, coming to the end of the time, I did want to thank my family. My husband, who, in a 

rare occurrence--I think that was the second time in his life he ever had a tux on. I had to 

grab a picture. And my daughters Kiera and Gavi and my family has just been so 

wonderfully supportive. I can't begin to say what I'd like to say about them other than 

thanks. And finally, to just thank all the veterans who participated in all of the projects and 

it's been a wonderful thing to be a part of the VA. So thank you very much, that’s the end. 

 

END