ABSTRACT

The increased availability of funding for effectiveness research provides a unique opportunity to evaluate

existing interventions as well as innovative approaches

to patient care. Studies of clinical problems that occur

frequently, that are costly to society, and for which there

is no consensus regarding the most effective treatment

are particularly attractive to funding agencies. For many

conditions, there is little agreement about the best course

of action and as a result, there is considerable variation

among providers in how they manage the condition. Determining which of several alternative approaches is most

effective is fundamental to efforts to improve the quality

of health care.

A wide variety of research designs and methods can

be used in outcomes research and the choice of a specific

method will depend on the state of knowledge about a

particular problem and the resources available to the investigator. Initially, literature syntheses and analyses of

existing databases may be most appropriate. However,

the literature is often unable to provide clear answers and

existing databases often lack clinical detail and information on outcomes. However, data useful for outcomes

studies might be obtained from patients’ medical records,

through questionnaires, or from cohort studies. While

these methods can provide valuable insights, the most

convincing scientific evidence for or against a specific

intervention will require experimental designs such as

randomized controlled trials.

Nurses involved in outcomes research will be faced

with several challenges. First, data relevant for studies

of the outcomes of nursing interventions will rarely be

obtainable from existing databases which are typically

designed to capture procedure and cost information. Second, education outcomes researchers outside of nursing

about the potential contribution of nurse interventions will

be necessary to build multidisciplinary research teams that

have the greatest potential to produce useful findings.

Finally, the introduction of innovative nurse interventions,

even if shown effective, may be difficult in a cost-conscious and conservative health care system. Nevertheless, the health care system is undergoing fundamental

change and the nursing perspective is likely to become

increasingly appreciated as outcome measures that have

depended only on physician observation and measurement are supplanted by patient-centered measures such

as functional status, quality of life, and satisfaction with

care.

I recently overheard my 5-year old daughter singing a children’s song that went like this:

Miss Suzy had a baby, his name was Tiny Tim

She put him in the bathtub to see if he could swim

He drank up all the water, he ate up all the soap

Methods And Measurement In Patient Outcomes Research:

Universal Issues

Dan Cherkin, PHD
He tried to eat the bathtub, but it wouldn’t go down

his throat.

Miss Suzy called the doctor

Miss Suzy called the nurse

Miss Suzy called the lady with the alligator purse

“Mumps” said the doctor

“Measles” said the nurse

“Nothing said the lady with the alligator purse

Miss Suzy punched the doctor

Miss Suzy kicked the nurse

Miss Suzy paid the lady with the alligator purse.

It is rare to hear a children’s song that relates so

closely to one’s professional interests. After singing it

over and over to myself it hit me: Of course! This song

was an allegory for the current crisis in American health

care! Tiny Tim obviously represents the insatiable consumer of health services whose easy access to medical

care had to gluttonous consumption and soaring costs.

Miss Suzy clearly represents the party who is footing the

bill for health care, probably Tiny’s employer or the federal government. The doctor, nurse and lady with the

alligator purse undoubtedly exemplify our pluralistic

health care system wherein people with a particular problem can receive care from a wide variety of experts. The

lack of agreement among the three experts consulted by

Miss Suzy illustrates that the diagnosis and treatment on

receives may depend more on which expert is consulted

than on the nature of the underlying problem. Miss Suzy,

as the payer, rejects the diagnoses of the two traditional

health care providers and then rewards the lady with the

alligator purse. (In reality, payers are becoming increasingly concerned about the costs, appropriateness and effectiveness of care and are starting to refuse to pay for

care that has been shown to be ineffective or inappropriate.) But who exactly is this enigmatic lady with the

mysterious alligator purse? Maybe she represents the ideal

low cost health professional of the future who payers

would love to see supplant the present day high cost providers. With diagnoses like “nothing wrong”, she is probably cheap—but is she effective? Is she cost-effective?

I believe that this children’s rhyme, or at least my

interpretation of it, illustrates why AHCPR’s medical effectiveness program is so important. Payers are reaching

their limits in terms of the amount of health services they

are willing and able to pay for and they need information

regarding which treatments are effective and which are

not. This is in large part why Congress has funded outcomes research.

In the next few minutes I would like to discuss some

of the broader methodological issues that confront those

of us interested in outcomes research. I will first comment on the types of clinical problems that funding agencies consider priority areas for investigation and will give

particular emphasis to the issue of practice variation. I

will then describe the range of research designs that my

be employed to address these types of problems. Finally,

I will list several challenges that those pursuing research

on outcomes of nursing practice may encounter. Since I

have been working with the Low Back Pain Patient Outcome Research Team for the past 2 years I will illustrate

my points with examples involving low back pain.

Clinical Problems Most Worth Studying

What clinical problems are most worth studying?

Clearly problems that occur frequently , that are most

costly to society, and for which there is no consensus

about diagnosis and treatment are in greatest need of outcomes research. Let me illustrate with low back pain

which, in the United States, is exceedingly common,

costly, and controversial. It is common in that it will
significantly affect almost 80 percent of us before we retire. It is the second most common symptom presented

to physicians in general and is the leading reason for visits to orthopedic and neurosurgeons. In the hospital, DRG

243 “Medical Back Problems” was the second most prevalent DRG exceeded only by normal deliveries in 1987.

While there are also numerous visits to physical and occupational therapists for back problems and possibly to

nurses, the number or content of such visits is not captured in any national database.

Back pain is also costly. A 1984 study estimated

the annual cost of back pain in the United States to be

almost 16 billion dollars, most of this in direct health care

costs. Among men, earnings and productivity losses due

to back problems exceeded those for respiratory conditions and ischemic heart disease, totalling $5 billion in

1984.

Finally, back pain is controversial. There is tremendous uncertainty about almost all aspects of back pain,

including its etiology, pathophysiology, assessment, and

treatment. For only 10-15 percent of people seeing a

physician for their back pain can a specific cause of the

pain be determined. There is disagreement about whether

back pain results from disorders of the muscles, joints, or

discs. Diagnostic x-rays are of little clinical value for

most patients but are still commonly used by some physicians. Probably the most controversial area, however,

has been therapeutic procedures. A whole array of therapies have been employed for back pain over the centuries

but there is little solid evidence that any of them is significantly better for most patients than reassurance and

allowing nature to take its course. And finally, it is worth

noting that the profession branded as an “unscientific cult”

by the American Medical Association, chiropractic, is

largely devoted to the management of back pain.

In view of this lack of consensus about the diagnosis and treatment of back pain, it should not be surprising

that there is enormous variation in how back pain is managed. The rate of back surgery in the United States is

double that in most Western countries and five times that

in England. Overall back surgery rates are twice as high

in the western United States than in the northeast and the

rate of spinal fusions is ten times as high. Hospitalizations for medical back problems is four times higher in

Boston than in the demographically similar community

of New Haven. Patients receiving care from health maintenance organizations are much less likely to be hospitalized for back problems than patients receiving care in the

fee-for-service sector. Although data are limited, it is

clear that there is also tremendous variation in the outpatient management of low back pain.

The underlying hypothesis that underpins much of

the effectiveness research, is that these practice variations

are more a function of provider practice style than of patient need. This raises a whole host of questions: Is the

amount of observed practice variation in fact more than

one would expect by chance? Is it the higher rate or the

lower rate the represents the better care? Is the procedure more effective (and more cost-effective) than available alternatives? What outcomes are most important to

measure? And finally, once answered to these questions

have been found, how can this information be translated

into improved clinical practice? It is the lack of the research necessary to adequately answer these questions

that makes outcomes research so important.

Research Design Issues

There is no single set of research designs or methods that are uniquely appropriate for outcome studies.

The design and methods used will depend on the state of

knowledge about a particular problem and the resources

available to the investigator and can range from analysis
of existing information to observational studies to experimental trials. As with any type of research, a logical initial step is to determine what might be learned from a

synthesis of the scientific literature. Literature syntheses

should go beyond merely summarizing the results of previous studies by providing an objective critical evaluation of the quality of the research methods employed in

each study. This approach makes it possible to assign

greater weight to studies that were more scientifically rigorous. In some cases, it is even possible to combine data

from several similar small studies and reanalyze the data

as if they came from a single large study. Unfortunately,

the literature often fails to provide clear answers. For

example, while we found a large number of studies documenting the effectiveness of surgery for spinal stenosis,

all of them suffered from serious methodological deficiencies.

The analysis of data from existing databases can

also provide a useful starting point. In fact, much of the

earliest outcomes research has relied on analyses of administrative databases such as the Medicare data. Jack

Wennberg’s use of available hospital discharge data documented dramatic geographic variation in the rates of common surgical procedures and is largely responsible for

stimulating the current interest in and funding of outcomes

research. Analyses of existing data can also provide useful estimates of the costs and complications associated

with specific procedures. In our back pain project, we

have made extensive use of the Medicare data as well as

data from the National Hospital Discharge Survey and

hospital discharge data for the state of Washington.

While existing databases can be very useful, they

generally lack clinical detail and key outcome data such

as functional status, disability, pain, and patient satisfaction. Hence, at some point it will be necessary to design

studies that involve the collection of new data. This might

be done through such mechanisms as chart reviews, questionnaires, or even cohort studies. For example, we were

intrigued by the fact that according to hospital discharge

data, over half a million Americans were hospitalized with

diagnoses such as herniated discs, lumbago, and back

strain and did not have surgery. We wondered why these

patients were being hospitalized. The hospital discharge

data indicated that about half of these patients had had an

imaging study, typically a myelogram, but provided no

information about the reasons for the admission and almost no information on the treatments these patients received. In order to obtain this type of information, we

abstracted data from the inpatient medical records of a

sample of back pain patients discharged from Washington state hospitals.

We have been involved in other activities designed

to gain an initial understanding of the reasons for nonsurgical hospitalizations for back pain and the potential

to shift some of the hospitalizations to the outpatient setting. These include a national survey of physicians concerning their policies for hospitalizing patients with nonsurgical back problems and interviews with patients recently hospitalized for non-surgical back pain to determine their perceptions of the rationale for and the benefits of having been hospitalized.

We are also conducting a prospective cohort study

of the outcomes of surgical and non-surgical care for patients with herniated discs and spinal stenosis. While this

study will provide valuable information about the outcomes of care associated with these two treatments, even

prospective cohort studies are unable to conclusively determine the relative effectiveness of different interventions since patients and physicians selecting one particular course of treatment may differ in prognostically important ways from those receiving alternative treatments.

Hence, the ultimate test of effectiveness is a well-designed
and well executed randomized and controlled trial. Randomized trials are not common in the back pain literature

and most suffer from serious defects. However, given

the lack of clearly superior back pain treatments for most

patients and the large number of back pain patients available for study, randomized trials should be quite feasible.

In fact, we are about to pilot test two primary care interventions for low back pain: one comparing spinal manipulation therapy with physical therapy and the other

evaluating the benefits of an educational intervention with

a specially trained nurse. In the latter study, we will determine if a 15 minute session with a nurse immediately

following a standard physician visit plus 3 follow-up

phone calls to the patient will improve patient outcomes.

This 15 minute nurse intervention will be designed to elicit

and address questions of greatest concern to the patients

and to convey a positive and caring attitude. We will

compare outcomes such as patient satisfaction, functional

status, and disability for this group with those for patients

receiving standard care alone.

Hence, there is a broad range of designs and methods that can be usefully employed in evaluating the effectiveness and cost-effectiveness of health care interventions.

Challenges for Nursing Outcomes Research

I would like to conclude with some observations

about challenges that may face nurses as they become

more involved with outcomes research. Some of these

challenges apply to outcomes researchers in general while

others apply more directly to nurse researchers. I offer

these observations from the perspective of someone who

primarily functions in the largely medical environment

and will therefore risk appearing naive concerning the

world of nursing.

Challenge 1: Obtaining Data Relevant to Study

the Effectiveness of Nursing Practice

Nurse researchers will be less able to find useful

existing data than those of us who have studied outcomes

for surgical procedures. In fact, there may be few if any

existing databases that collect the type of information that

is needed for studies of the effectiveness of nursing practice. Even medical records and nursing notes may lack

the kind of information that will be required for these

types of studies. As a result, most nursing effectiveness

studies will require the collection of data that has not normally been collected pertaining to both the process and

outcomes of care. Furthermore, it is my belief that much

of the potential for nursing practice to improve patient

outcomes will involve that evaluation of new models of

care. To the extent that this is true, researchers will not

be able to rely on observational studies of existing procedures and will therefore need to develop and evaluate new

interventions. This will require familiarity with experimental and quasi-experimental research designs.

Challenge 2: Becoming Involved in

Multidisciplinary Research Teams

Much of the current outcomes research has centered

around crude outcomes associated with surgical procedures and therefore has not included as much nurse involvement as might be desirable in other types of outcome studies. As outcomes research matures, I believe

there will be an increasing emphasis on such areas as

patient satisfaction, patient understanding of their problem, patient activation, and patient function—all areas in

which nurses already make significant contributions.

However, while nurses could pursue outcomes research

in isolation from other disciplines, becoming integrated

with a team of outcomes researchers from a broad range

of disciplines may ultimately prove more productive and

rewarding. One of the most promising and exciting aspects of our back pain project has been our ability to bring
experts in research methodology together with clinicians

who understand the clinical issues and are open to evaluating whether what they are doing in fact benefits patients. We have benefitted from having a doctoral candidate in nursing working on our project, and as I mentioned earlier, are planning to evaluate the impact of a

nurse delivered intervention on outcomes. Some of the

individuals currently conducting outcomes research may

not be aware of the contributions nurses may be able to

make. Hence, an alternative to independently embarking

on an outcomes research project is to become familiar

with existing outcomes research activities and to then

suggest ways in which that research may benefit from

the inclusion of a nursing dimension.

Challenge 3: Introducing Innovations in a CostConstrained and Conservative System

It has been my impression that the potential for

nurses to use their knowledge and skills to benefit patients has been severely limited by the way the health

care system is currently structured and the way in which

services are reimbursed. I believe that increased use of

the skills nurses have in the areas of patient education,

motivation, and behavior change have great potential for

improving patient outcomes. However, in this era of costcontainment, realizing this potential for improving patient outcomes. However, in this era of cost-containment,

realizing this potential will require not only proving

through research that specific interventions are effective,

but also that they are cost-effective. Even this may not

be enough since it often seems that it is easier to continue

performing expensive procedures which have never been

proven effective than to obtain reimbursement for relatively inexpensive interventions, even if there is good

evidence they are effective. Changing standard medical

practice, and I have chosen my words deliberately, is a

difficult and slow process.

While I would counsel patience, I believe there is

reason for optimism. The health care system is undergoing major changes and the limitations of the biomedical

model are becoming increasingly apparent. Outcome

measures that depend only on physician observation and

measurement are being supplanted by patient-centered

measures such as self-assessed function and health status, quality of life, and satisfaction with care. Articles

documenting the potential for the quality of the patientprovider interaction to affect outcomes of care are beginning to appear with increasing frequency. It is my impression, that because these changes are very much in

concert with the values of the nursing profession, the

outcomes research to come out of nursing will become

increasingly relevant and will have the potential for making a major contribution to improvements in the effectiveness of health care in this country.