Former New York Yankees’ catcher Yogi 

Berra, famous for his amusing observations 

about life, once reputedly said, “You’ve got 

to be careful if you don’t know where you’re 

going—because you might not get there.” As 

nonsensical as Yogi’s comment may seem, it 

underscores an important point about providing veterans with the best possible health 

care: in order to know where we are going, 

we first need to know where we are. 

That’s what makes VA’s performance management system so important. The system 

helps us understand our successes, identify 

gaps in our care or management processes, 

and build improvements to enable us to provide veterans with “the best care anywhere.” 

Turning Point for Measurement 

System

Although the current performance measurement system has served VA well, we are 

at a turning point. The Executive Career 

Field (ECF) plan for 2008 is the beginning 

of a new use of performance measurement. 

In 1995, when measures became a formalized part of the ECF plan by which all 

Senior Executives in the Veterans Health 

Administration are evaluated, all VA performance measures were simultaneously part 

of the ECF plan. Measures flowed through 

a Performance Management Work Group 

where vote counts determined their fate. 

Those votes were sometimes swayed by 

views of the ideal number of measures, not 

just the intrinsic properties of the new measure which was up for consideration.

Of VA’s four missions: patient care, education, research, and emergency management, 

I believe that patient care is by far the most 

important. Therefore, it is critical we assess 

our performance in delivering patient care by 

collecting data on broad and comprehensive 

measures covering the core areas of care 

we provide. We have turned a key corner 

in measurement with the development, this 

year, of both a comprehensive set of Health 

Systems Indicators and an ECF plan that is 

a subset of Health Systems Indicators. The 

Health Systems Indicators are a library of 

measures that cover, with breadth and depth, 

the care that we provide to our veterans. We 

are liberated from the “too few, too many” 

dilemma that existed when all performance 

measures were included in the ECF plan. 

Now we can ask the only performance question that really matters: How are we doing in 

taking care of our veterans? 

Patients’ perceptions, as measured by 

the Survey of Healthcare Experiences of 

Patients, allow us to add a rich patient-centered dimension to what we learn about the 

care we provide and also allow us to shape 

that care to meet the needs of patients. Both 

dimensions, measuring clinical care and asking our patients about their care, are critical. 

In some cases, there are disconnects (such 

as the fact that patients’ perceptions of the 

clinical quality of care delivered does not 

correlate with the clinical quality we measure). However, we learn and improve by 

addressing both key components—as well as 

the factors that lead to the lack of connection between the two. 

The 2008 ECF plan will address a more select 

and limited set of measures. It will include a 

new category that we refer to as transformational measures, an expanded set of measures 

that reflect the efficiency of the care we provide, and core competencies that we deem 

essential for our senior executives. We know 

our staff, clinicians and non-clinicians alike, 

VA’s Performance Measurement Makeover 

by Michael J. Kussman, M.D., M.S., MACP, Under Secretary for Health, Veterans Health Administration 

November 2007

Director’s Letter 2

Response to Commentary 3

Research Highlights

• Providing Optimal Care 4

Is Complex—And So Is 

Measuring It

• Building a Better VA 5

Performance Measurement 

System

• Performance Measurement: 6

Technical Aspects

VA’s Performance Management 

Work Group 7 

Contents

FORUM

Translating research into quality health care for veterans

A publication of the VA Office of Research & Development, 

Health Services Research & Development Service, 

Center for Information Dissemination and Education Resources, 

in conjunction with AcademyHealth
want to do the right thing for our veterans. 

We do not believe that a measure must be in 

the ECF plan in order for us to improve our 

care. But we do know that presence in the plan 

is a powerful driver. This led to the development of Mission Critical Measures (measures 

designated as “must achieve” for this system of 

care) as part of the ECF plan.

Many new items are part of our new measurement system. The breadth and depth 

of our measures, and continual monitoring 

of them through a library of Health System 

Indicators, allow us to maintain continual 

readiness. If we see a key measure trending 

in the wrong direction, we act immediately: 

first through alerts to the field; then by 

adding the measure back to the plan; and, 

finally, by developing quality improvement initiatives around the measure. 

Transformational measures provide us with 

stretch goals in key areas where we have 

opportunities for improvement.

An External Benchmarking 

Opportunity

This year, we have formalized the concept of 

external comparisons. While we desire to provide great care because of our deep and passionate commitment to achieving excellence, 

the perception of VHA’s quality is important 

to our very existence. In the past, one of the 

challenges to external perceptions of quality 

was the charge by detractors that VA both 

developed our own measures and compared 

ourselves to ourselves—resulting in a lack of 

credibility. The statute that established the 

National Quality Forum (NQF) required federal entities such as ours to use NQF-endorsed 

measures. An August 2006 Executive Order 

further mandated use of measures developed 

by external entities, such as the Ambulatory 

Quality Alliance and the National Committee 

for Quality Assurance (NCQA). 

This year, the quality measures in the ECF plan 

will be those of the Healthcare Effectiveness 

Data and Information Set (HEDIS), and 

ORYX measures that have been developed 

and given face validity as a result of the work 

of multiple bodies, including the NCQA, 

the American Medical Association, the Joint 

Commission, the Centers for Medicare and 

Medicaid Services, and the final arbiter, the 

NQF. Therefore, it will no longer be possible 

to challenge that we do well on measures only 

when we can create them ourselves. We gladly 

accept the challenge of comparing ourselves 

on measures that have been externally developed and widely used. This also gives VHA a 

valuable external benchmarking opportunity, 

and we do well when given that opportunity.

These measures and all measures are population-based. None are intended to guide 

treatments of individual patients. Making 

sure that this message is clear for clinicians 

is the ongoing challenge of measurement, 

not just for the VA, but for all who develop 

and use health care performance measures. 

No VA measures will be dropped; instead 

they will remain in the library of Health 

Systems Indicators so that our clinicians 

continue to receive a refined and granular 

look at the care they provide to patients.

As our knowledge about how to measure 

clinical performance expands, and our 

library of measures grows, we also face the 

challenge of narrowing our focus to areas of 

critical importance, and the ability to shift 

that focus in response to new knowledge 

and events. As Yogi also supposedly said, 

“You can observe a lot by watching.” 

FORUM — Translating research into quality health care for veterans 2 

VA Office of Research & Development, Health Services Research & Development Service November 2007

Fiscal year 2007 has been a year of many accomplishments for 

HSR&D. Highlights include recognition of the innovative work 

done by HSR&D investigators through a number of prestigious 

awards. Dr. Rudy Moos received the 2006 Dan Anderson Research Award; Dr. Doug Owens earned the VA Under Secretary 

for Health Award; Drs. Rachael Werner, Louise Walter, and Paul 

Haidet received distinguished awards at the Society of General 

Internal Medicine Annual Meeting; Dr. Joseph Hanlon was honored by the American Geriatrics Society; and Dr. Kevin Volpp received the new investigator award at 

the AcademyHealth Annual Research Meeting, just to name a few. 

This year also brought enhanced training and learning opportunities. Our Resource 

Centers have developed and implemented 67 cyber seminars to train investigators in health economics and data and information systems. Both managers and 

researchers benefited from the results of the new Evidence-Based Synthesis Pilot 

Project, with several synthesis reports now available on the HSR&D website (visit 

www.hsrd.research.va.gov/publications/esp/). Most importantly, the investigator 

initiated research and career development programs continue to thrive. HSR&D reviewed 112 research proposals at our recent Scientific Merit Review Board meeting, 

and we expect to fund 27 of them. In addition, we hope to fund nine career development awards of the 31 reviewed. 

It has been a challenging year, particularly as we worked to improve data security, confidentiality, and privacy protection. Personally, in my first year as Director 

of HSR&D, I have learned more about the organization from the Central Office 

perspective. Moreover, I have gained an immense appreciation for the hard work 

and dedication of our investigators and of the HSR&D staff in both Central Office 

and the field, who work diligently to improve the health and care of our veterans. I 

look forward to working with all of you in fiscal year 2008! 

Seth A. Eisen, M.D., M.Sc.

Director, HSR&D

Director’s Letter
Health services researchers in VA have 

the benefit of working within a health care 

organization that is engaged in one of the 

boldest, large scale initiatives regarding 

performance measurement. Dr. Kussman’s 

article explains what the next generation 

of that performance measurement system 

will be, and how it relates to the Executive 

Career Field plan. This new initiative offers 

many exciting opportunities for health 

services researchers interested in quality 

measurement and improvement.

Refining Individual Measures

First, there is the traditional role of health 

services research in understanding and 

refining the individual measures themselves. How reliable is measurement? How 

accurately does the measure assess what is 

considered “quality” for this condition? Is 

it useful for distinguishing between providers? We know, for example, from work 

done by Eve Kerr and colleagues that existing External Peer Review Process (EPRP) 

measures correlate significantly but modestly 

with two other ways of assessing quality—

using a larger and more comprehensive set 

of process measures, and the implicit review 

of medical records by peers.1

The finding by Kerr et al. suggests that 

each of these measurement systems provides a valid assessment of quality but each 

is also only a part of the picture. What are 

the implications of this finding? How can 

the measurement be improved? Health services research has a long history of doing 

this kind of “basic science” on the properties of quality measures. As new measures 

get proposed for inclusion in VA’s measurement set, and as we learn more about 

what processes influence patient outcomes, 

there is—literally—an endless stream of 

this kind of work that needs doing.

Performance Measurement as 

Intervention

I would like to suggest that health services 

researchers get more involved in another kind 

of research, which looks at performance measurement as an intervention. An intervention 

that is complex, which includes many components, such as the process that sets what the 

measurement criteria are, activities to achieve 

buy-in by people in the field, mechanisms for 

providing feedback, benchmarking, the use 

of a modest financial incentive to stimulate 

performance, etc. As an intervention, VA’s 

performance measurement is likely influenced 

by many factors—at the VISN level, at the 

facility level, even perhaps at the individual 

level. And with 21 VISNs and hundreds of 

facilities, these factors are going to have a 

natural degree of variation throughout VA. 

What kinds of components in this multi-component intervention, and what organizational 

characteristics, are associated with varying 

“success”? Can we identify specific organizational features and characteristics of individual 

intervention components that lead to greater 

“success”? Does performance measurement 

produce unintended consequences, and how 

can these be minimized? Do some measures 

or implementation methods work better than 

others to promote improvement and minimize unintended consequences? Research to 

date has only begun to generate the needed 

insights regarding how best to implement 

large policy interventions such as performance 

measurement and feedback. 

The needed research is much messier than 

many researchers like—it is rarely feasible to 

study these policies experimentally, meaning 

that inference is going to have to come from 

observational studies. In most circumstances, 

it is not even possible to study policy implementation in a controlled fashion. I think it 

unlikely that policymakers would accept on 

a regular basis the implementation of performance measurement components on some 

facilities and not others. But just because it is 

more difficult does not mean we should shy 

away from the task. Indeed, VA health services researchers should strive to be at the leading edge of developing the methods by which 

generalizable knowledge can result from the 

study of real world policy initiatives. 

There are already resources within VA to 

help us. The VA’s QUERI program has been 

generating useful lessons about the difficulties 

in studying the implementation of complex 

interventions. Implementation Science, co-edited 

by Brian Mittman of the HSR&D Center 

at Sepulveda, publishes research in this field 

(visit www.implementationscience.com). We 

have an exciting task in front of us. It’s going 

to be a challenge. But it is also going to offer 

great rewards, in terms of understanding how 

better to improve the delivery of care by VA, 

and by extension, other health systems. 

References

1. Kerr E, et al. Quality by Any Other Name: A 

Comparison of Three Profiling Systems for Assessing 

Health Care Quality, Health Services Research, in press.

FORUM — Translating research into quality health care for veterans 3 

Response to Commentary

The Role of Health Services Research 

in Performance Measurement

Paul W. Shekelle, M.D., Ph.D., VA Greater Los Angeles Healthcare System

Mark your calendars! 

HSR&D’s National Meeting is scheduled for February 13-15, 2008 at the Marriott Waterfront in Baltimore, Md. Implementation across the Nation: From 

Bedside and Clinic to Community and Home is the theme for the 2008 meeting. 

This theme was selected to emphasize HSR&D’s commitment to examining 

best practices in implementing state of the art health care for veterans. Together with workshops, oral presentations, and poster sessions, meeting highlights 

include a keynote address by Dr. Carolyn Clancy, Director of AHRQ, presentation of the Under Secretary’s Award for Outstanding Achievement in Health 

Services Research, and a presentation by a young veteran injured in combat in 

Afghanistan. Visit www.hsrd.research.va.gov/meetings/2008/ for more inforVA Office of Research & Development, Health Services Research & Development Service November 2007
FORUM — Translating research into quality health care for veterans 4 

VA Office of Research & Development, Health Services Research & Development Service November 2007

Performance measurement is a powerful 

policy tool for promoting efficient highquality care. However, it is often underestimated just how harmful performance 

measures can be when they are poorly 

constructed.1-3 Perhaps the hardest lesson is 

that developing performance measures that 

promote optimal care usually requires clinically detailed data and complex measures, 

and that simplistic or naïve “good care” 

measures can have strong perverse incentives. 

This push for simple measures is strongest 

in the community where the resources 

and will to invest in electronic medical 

records and chart-based review have been 

lacking.3 However, since there is considerable political pressure for the VA to adopt 

those measures used in the community 

(so as to allow benchmarking), the recent 

push for all-or-none “good care” measures 

is of considerable concern (e.g., A1c < 7 

percent, blood pressure < 130/80, LDL < 

100mg/dl, etc). Such measures are likely 

to be very imprecise measures of efficient 

high-quality care, and are also prone to 

perverse incentives, such as promoting 

treatment irrespective of how small the 

potential benefit and how great the patient 

burden or risks.1-3 

Diabetes Measures

Take annual diabetes eye exams for example. Although well-timed photocoagulation 

for early diabetic retinopathy is one of the 

most beneficial treatments in all of medicine, research conducted by VA HSR&D 

suggests that almost all visual impairment 

that is preventable by early detection will 

be captured by: 1) screening those with no 

known eye disease every two to three years, 

and 2) close individualized surveillance 

(every 4 to 12 months) after early retinopathy has been detected.4

 Therefore, annual 

exams are not of high importance for the 

vast majority of low-risk patients, and are 

too infrequent for the highest risk patients 

who will account for most preventable 

complications. As a result, the conventional 

performance measure (annual examinations 

for diabetic patients), which provides a 

strong incentive to focus resources on getting everyone into clinic every 13 months, 

provides no incentive to develop an effective system to optimize care. 

In fact, a health system that schedules 

exams every 10 to 11 months and devotes 

its scarce administrative resources to trying 

to get anyone who misses this appointment into the clinic as soon as possible, 

is likely to improve its performance rating 

while doing almost nothing to improve 

outcomes. In contrast, a health system that 

uses its administrative resources to aggressively reschedule those needing close follow up (because of known retinopathy) and 

to have low-risk patients seen at least every 

two years (those whose last retinal exam 

was normal on their last visit), may make 

its performance rating much worse while 

substantially improving patient outcomes. 

It should not be surprising therefore that 

when we tried to develop an effective system to improve eye screening and followup for diabetic patients, that the prevailing 

annual eye exam performance measure was 

one of the biggest barriers to implementing 

the more effective system. Although many 

may criticize clinic leaders for not doing 

the “right” thing, these clinics and providers have huge demands on their time and 

attention. As has happened with so many 

important quality improvement initiatives that we’ve consulted on in the past 

10 years, the clinicians and administrators 

eventually said, “If the problem you want 

us to address is really that important, then 

get the performance measure changed. We 

are struggling to meet dozens of demands 

and we just do not have the time and personnel to electively take on more things.”

Measuring “Good Care”

It is rare that “good care” can be measured simply. Although the new NCQA 

diabetes measures of A1c < 7 percent and 

BP <130/80 may seem straightforward 

enough, they actually provide strong incentives for speculative, costly, and potentially 

dangerous polypharmacy. In addition, these 

are unadjusted outcome measures that are 

likely to be inaccurate. For example, the 

A1c measure more strongly rewards adding or increasing the dose of a glitazone, 

which has high costs and limited data on 

long-term safety, in someone with an A1c 

of 7.5 percent than it does for doing so in 

someone with an A1c of 8.5 percent, even 

though a risk of an A1c of 7.5 percent is 

trivial compared to the risk of an A1c of 

8.5 percent. 

Similarly, although adding up to three to 

four medications at moderate doses in 

pursuit of good blood pressure control in 

a high CV-risk patient has been shown to 

be highly beneficial, the benefits of pursuing the 130/80 targets using more than 

three to four medications is pure speculation. Furthermore, there is ample reason to 

be concerned about harmful effects from 

polypharmacy or excessively lowering diastolic blood pressure. 

Trying to measure a complex clinical 

scenario using simplistic performance 

measures and wishful thinking is increasingly promoted by disease advocates and 

industry-sponsored experts in the community. NCQA’s new “good control” diabetes 

measures and its resistance to revising its 

eye care measure despite the strong advice 

Research Highlights

Providing Optimal Care Is Complex—

And So Is Measuring It

Rodney A. Hayward, M.D., HSR&D Center for Practice Management 

and Outcomes Research, VA Ann Arbor Healthcare System 

continued on page 8
FORUM — Translating research into quality health care for veterans 5 

Why measure quality of care? The answer 

to this question might seem obvious, but in 

thinking about how we can build a better 

quality measurement system, we must first 

clarify the purpose of this sometimes labor 

intensive activity. The most obvious answer 

is that measuring performance is an essential step in guiding quality improvement. 

But no measurement set is comprehensive, 

and there is always a danger that tracking 

performance for selected indicators will lead 

to improvement in those limited areas at 

the expense of equally important areas not 

targeted by the performance measurement 

system. So, a second purpose to any 

quality measurement system must also be to 

represent the broader and underlying quality 

provided in areas beyond the system. With 

luck, a quality measurement system will lead 

to improvement in those areas as well. 

The Evidence for Performance 

Measurement

The VHA performance measurement system is among the most widely implemented 

in the world. Managers receive regular 

reports as to how their regions or facilities 

are doing on selected measures of essential 

processes of care and intermediate outcomes. They pass these incentives down 

to line providers. What is the evidence that 

performance measurement has improved 

care? While it is difficult to disentangle the 

effects of performance measurement from 

other components of VHA reorganization 

that began in the 1990s, it is clear that 

VHA performance on the tracked items has 

surpassed competing systems. For example, 

Jha et al. compared the VHA to Medicare 

and found better performance for 12 of 13 

measures. But was this teaching to the test? 

How much of the improvement was confined to the tracked areas? 

“To paraphrase Kevin Costner 

in Field of Dreams, this is evidence 

that if you build it (the performance 

measurement system, that is), they 

will come.”

A few years ago, Eve Kerr, Beth McGlynn, 

myself, and others investigated this question 

using a very broad range of quality indicators 

included in RAND’s QATools measurement set. We confirmed that the VHA outperformed a community sample overall on 

basic process measures. In the tracked areas, 

the difference was marked. VHA patients 

received 66 percent of recommended care as 

compared to 43 percent, a 23 point advantage. In unrelated measures of quality, the 

VHA also did slightly better, but the difference was much smaller, on the order of 

5 percent. To paraphrase Kevin Costner in 

Field of Dreams, this is evidence that if you 

build it (the performance measurement system, that is), they will come.

Building a Better 

Measurement Set

So far, it seems that performance measurement is bearing fruit in its primary purpose 

of guiding quality improvement to measured 

areas, but not for the secondary purpose 

of engineering more widespread improvement. Interestingly, however, we found that 

the VHA performed better on measures 

that were related, but not the same as, the 

targeted areas. A related measure might be 

an aspect of diabetes that was not in the 

performance set, or an immunization that 

was not directly tracked like influenza vaccination. The VHA advantage here was real, 

12 percent. We hypothesize that this is due 

to a chain reaction effect in the minds of 

the providers, who now think about 

diabetes care more because of performance 

measurement, but do so more holistically 

than the performance measurement set 

would require. Likewise, clinic managers 

have adjusted their thinking by building 

systems to make it easier to vaccinate for 

influenza and thus ease other vaccinations 

as well. So, if you build it, they will come, 

and they might stay at your neighbor’s 

house, too.

“...we cannot afford to ignore important broad areas of care, because if 

we do, we may fail to spark quality 

improvement in related areas.” 

What are the implications for building a 

better performance measurement set? First, 

we cannot afford to ignore important broad 

areas of care, because if we do, we may fail 

to spark quality improvement in related 

areas. For example, the VHA performance 

measurement system had ignored acute care 

in the past, and this is now being remedied. 

Second, we should empirically test how 

well leading indicator systems represent 

broader concepts of quality of care to help 

us choose leading indicators more wisely. 

The VHA system is widely admired, and 

considerations such as these will keep it at 

the forefront. 

Research Highlights

Building a Better VA Performance 

Measurement System

Steven M. Asch, M.D., M.P.H., HSR&D Center for the Study of Healthcare 

Provider Behavior, VA Greater Los Angeles Healthcare System 

VA Office of Research & Development, Health Services Research & Development Service November 2007
FORUM — Translating research into quality health care for veterans 6 

VA Office of Research & Development, Health Services Research & Development Service November 2007

This article will briefly review some general 

concepts related to the technical aspects of 

performance measurement that managers 

should be aware of in creating and using 

performance measurement systems. These 

concepts all relate to the following basic 

principle of performance measurement:

Specific performance indicators represent a sample of all 

the possible processes and behaviors that need to happen so that the patients receive high overall quality care. 

Sampling and Scoring

While the result of an indicator relating to 

colon cancer screening has some intrinsic 

interest, when managers offer substantial 

monetary incentives and assess performance 

through the use of such indicators the explicit 

assumption is that they measure a broader 

construct, such as a clinic’s or hospital’s overall quality, or perhaps the state of primary 

care or preventive care at a facility. Like the 

more familiar sampling of people to estimate 

a population characteristic, the indicators 

must be sampled in such a way as to represent the entire target population of indicated 

processes and behaviors. Furthermore, the 

sampling of indicators to estimate a construct 

such as quality introduces another source of 

uncertainty into the estimates. Standard performance measurement approaches do not 

adequately consider these issues.

First and most obviously, a performance 

measurement system is irretrievably flawed 

if the sampled measures do not adequately 

represent the range of behaviors that are actually important and that we should encourage. 

A non-representative sample provides a distorted measure of a broader construct such as 

“primary care quality.” A non-representative 

sample also provides perverse incentives for 

providers to abandon important processes of 

care and concentrate on the incidental processes that are over represented in the performance measures. 

“While seeking the holy grail of comprehensive automated performance 

measurement, other established but 

often underemphasized management 

tools—such as an active and effective emphasis on the perennial challenges of human resources and staff 

morale—remain critically important to 

maintaining and improving quality.”

Second, the usual aggregate scores such as 

average pass rates—on all measured preventive care or chronic care indicators for 

example—do not adequately reflect the sampling variability inherent in the choice of a few 

indicators to represent a broader construct of 

quality. The result is that these aggregate measures may have a much higher noise to signal 

ratio than is suspected and may not track well 

any changes in practice by a provider or clinic. 

This leads to cynicism and demoralizes those 

profiled. One way to mitigate this problem is 

to use a random effects or multilevel analysis, 

which by explicitly modeling and removing 

some of the measurement error, results in a 

more precise quality score. 

Third, in designing a performance measurement system, it is important to find the 

organizational level at which the variation 

is located, and at which, a response should 

occur. If a process varies across facilities 

but not across providers within the facility, and if the best approach to fixing the 

problem is an organizational rather than an 

individual one, then what is the point of 

constructing provider level profiles? 

A corollary of this last point is that if there 

is not much variation, then there may not be 

much point in measuring other than at the 

population level. So, for example, if only 50 

percent of patients get a recommended process of care across a health care network and 

there is little variation across providers relative 

to this huge absolute gap (from 50 percent to 

100 percent), then a network-wide remedy is 

needed. Furthermore, to assess the remedy a 

simple measurement based on a modest, network-wide sample is all that is needed to see 

how the rate changes. This task is much simpler than implementing a performance measurement system that must draw samples, calculate rates, and educate individual providers. 

The Halo Effect

Finally, managers should be relieved to know 

that there is data to suggest that measurement, feedback, and incentives using well 

designed performance indicators, such as 

some of the VA External Peer Review 

Process (EPRP) indicators, do appear to have 

a “halo” effect on indicators that are not part 

of the active measurement and feedback set.1

However, this halo appears to extend only 

across the same clinical condition and not 

to unrelated clinical areas that are not part 

of the current EPRP system. The implication of this finding is not to start using poor 

measures for clinical areas that we do not yet 

monitor, but to recognize that performance 

improvement using clinical indicators may 

only be able to cover a finite amount of 

the waterfront. While seeking the holy grail 

of comprehensive automated performance 

measurement, other established but often 

underemphasized management tools—such 

as an active and effective emphasis on the 

perennial challenges of human resources and 

staff morale—remain critically important to 

maintaining and improving quality.

References

1. Asch SM, et al. Comparison of Quality of Care for 

Patients in the Veterans Health Administration and 

Patients in a National Sample. Annals of Internal Medicine

2004; 141(12):938-45.

Research Highlights

Performance Measurement: Technical 

Aspects

Timothy P. Hofer, M.D., Ms.C., HSR&D Center for Practice Management and 

Outcomes Research, VA Ann Arbor Healthcare System 
FORUM — Translating research into quality health care for veterans 7 

As Dr. Kussman points out in his lead 

commentary, the VA’s performance measurement system—with over 100 performance measures in the areas of access, 

satisfaction, cost, and quality—has served 

VA well. In 1995, VA first incorporated 

performance measures in the Executive 

Career Field (ECF) performance contract, 

an annual plan that establishes standards 

against which both central office and VISN 

leaders are evaluated. ECF contracts ensure 

accountability for both administrative and 

clinical performance measures and are 

developed collaboratively by central management and field leaders. 

These performance contracts form the 

basis for quarterly management reviews, 

and incorporate modest management 

incentives. Results are also reported broadly within the VA and externally to key 

stakeholders including Congress, the Office 

of Management and Budget, and advocacy 

groups. Linking performance measures 

to these contracts results in personal 

accountability throughout the system. The 

Performance Management Work Group is 

central to this process. The work group has 

four key responsibilities:

• Engage in regular and systematic planning for measurement;

• Perform annual review of existing network trends;

• Consider non-VA comparator system 

trends; and, 

• Oversee the development of the annual 

ECF performance plan.

This last responsibility involves a multistep process, which begins with the work 

group integrating the priorities it receives 

from the Under Secretary for Health and 

other central and local VA leaders. The 

work group then convenes to identify 

measurement priorities and to assure measurement initiatives are developed, and 

approves proof of concept proposals 

from subject matter experts. The work 

group must also maintain a clear set of 

criteria for measurement approval and 

retirement, and, ultimately, identify and 

incorporate proposed performance 

measures into the next year’s performance 

plan. The 2008 ECF plan emphasizes 

greater personal accountability by utilizing 

a streamlined set of performance measures, 

including new measures aimed at the 

efficiency of care provided to veterans, 

and also core competencies expected of 

senior leaders. 

The recent adoption of a new VA national 

performance measure for thiazide diuretic 

use illustrates the rigorous process by 

which new measures are vetted by the 

work group. Despite evidence-based guidelines which favor thiazide diuretics among 

patients with uncomplicated hypertension, 

use of these drugs remains low. National 

hypertension management guidelines such 

as the Joint National Committee (JNC) 7 

and the VA/DoD Hypertension Clinical 

Practice Guidelines recommend thiazide use for most patients in accordance 

with the results of the ALLHAT study. 

However, review of published pharmacy 

utilization data both in VA and in the private sector strongly suggests that hypertension guidelines are not followed for most 

patients. The work group approved inclusion of a pilot measure for diuretic use, 

which eventually led to adoption of the 

new performance measure for diuretic 

use in 2007. 

Comprised of central and field leaders, 

and including both administrators and 

clinicians, the work group is co-chaired by 

Barbara Fleming, M.D., Ph.D., and Jim 

Roseborough, FACHE. The efforts of the 

group result in performance contracts that 

are recommended to the Under Secretary 

for Health and represent the collaborative efforts of central and field leadership, 

and reflect both administrative and clinical 

priorities.

VA’s Performance Management Work 

Group

Tammy Czarnecki, M.S.O.L., M.S.N., R.N., Clinical Quality Program Specialist,

Office of Quality and Performance

VA Office of Research & Development, Health Services Research & Development Service November 2007

Introducing FORUM’s new look!

FORUM is a newsletter of VA’s Health 

Services Research & Development 

Service. This issue marks the 

debut of a new format that we hope 

will improve readability and the overall 

look and feel of the newsletter. We 

hope you enjoy the FORUM makeover 

and we especially welcome your 

comments and suggestions. Send 

comments and suggestions via email 

to hsrdweb.boston@va.gov

The HSR&D Cyber Seminar 

Program, A Great Educational 

Resource 

The HSR&D Cyber Seminar Program 

continues to grow with new courses 

being added monthly. The new and improved HSR&D Cyber Seminar Catalog 

now allows users to view, sort, and search 

the catalog of offerings according to 

user-defined criteria. Interested in costeffectiveness? Implementation research? 

Clinical informatics? A quick search of 

the catalog points you to an array of 

seminars on your topic of interest. Sign 

up to participate in a live session or 

download a recorded session for viewing 

on your computer at your convenience. 

Audio recordings, session slides, and 

handouts also are available for download 

from the catalog. Check out the new and 

improved catalog on the HSR&D 

website at www.hsrd.research.va.gov/

for_researchers/cyber_seminars/catalog.cfm 
FORUM — Translating research into quality health care for veterans 8 

of evidence-based medicine experts is just 

one example of this trend. This is not to 

suggest that we should limit ourselves to 

only measuring bad care, such as A1c > 9 

percent or retinal screening exams > two 

years. However, if we wish to measure 

“good care,” we will need more nuanced 

performance measures that consider the 

benefit of reaching these “optimal” treatment goals, as well as the risks, costs, and 

patient burden associated with treatments 

needed to reach these optimal goals. 

References

1. Hayward RA. Performance Measurement in Search 

of a Path. New England Journal of Medicine 2007; 

356(9):951-3.

2. McMahon LF, Hofer TP, Hayward RA. Physicianlevel P4P–DOA? Can Quality-based Payment Be 

Resuscitated? American Journal of Managed Care 2007; 

13(5):233-6.

3. Hayward RA. All-or-nothing Treatment Targets 

Make Bad Performance Measures. American Journal 

of Managed Care 2007; 13(3):126-8.

4. Hayward RA, et al. Causes of Preventable Visual 

Loss in Type 2 Diabetes Mellitus: An Evaluation of 

Suboptimally-timed Retinal Photocoagulation. Journal 

of General Internal Medicine 2005; 20:467-9.

Geraldine McGlynn, Editor-in-Chief

Margaret Trinity, Editor

Peter Almenoff, M.D., 

Director, VA Network #15, 

Kansas City, MO

Martin P. Charns, D.B.A., 

Director, VA HSR&D Center 

of Excellence, Boston, MA

Seth A. Eisen, M.D., M.Sc., 

Director, HSR&D

Joseph Francis, M.D., M.P.H., 

Deputy Chief Research and 

Development Officer, 

Associate Director, HSR&D 

QUERI, Central Office

Rodney A. Hayward, M.D., 

Director, VA HSR&D Center 

of Excellence, Ann Arbor, MI

Shirley Meehan, M.B.A., 

Ph.D., Deputy Director,

VA HSR&D, Central Office

Michael J. Miller, M.D., 

Ph.D., Chief Medical 

Officer, VA Network #1, 

Bedford, MA

Eugene Z. Oddone, M.D., 

M.H.Sc., Director,

VA HSR&D Center of 

Excellence, Durham, NC

Alan S. Perry, M.H.A., 

FACHE, Director, VA 

Central California Health 

Care System

Editorial Board

FORUM

continued from page 4