ABSTRACT

This paper reports on the results of a Congressionally mandated conference that was held last April

to develop a research agenda for outcomes research

for the Agency for Health Care Policy and Research

(AHCPR) and for Congress. The conference was conducted by the Foundation for Health Services Research

(FHSR) and the Alpha Center, with funding from

AHCPR. The meeting involved about 200 individuals from around the country representing both users

and producers of outcomes and health services research

information.

The purpose of this report is to contribute some

of the observations and recommendations from this

conference to the development of an agenda for outcomes research in nursing.

The paper begins with three important distinguishing characteristics of outcomes research: (1) a

focus on conditions, and the alternative treatments

for addressing conditions, rather than on more narrowly defined individual treatments; (2) a concern not

just with clinical or physiological outcomes, but with

measures of health-related quality of life, including

physical and emotional functioning, general perceptions of health and well-being, and satisfaction with

the process of care; and (3) a recognition of the fact

that the effectiveness and outcome of any treatment

process is a function as much of non-clinical factors,

such as the socio-demographic characteristics of the

patient, the organization of the delivery of care, the

financial incentives facing both patients and providers, as it is of clinical factors, thus necessitating a

multi-disciplinary approach to the conduct of outcomes research.

The paper then presents the recommendations of

the April conference in seven major areas:

1. The scope of outcomes research efforts;

2. Issues of measurement;

3. Issues of research design and methods;

4. Issues of data;

5. Factors determining the effectiveness of

treatments and the outcomes of care;

6. The potential sites for outcomes research;

and finally

7. Issues relating to the dissemination and

implementation of the findings from outcomes research.

The paper also identifies five major obstacles in

the conduct of outcomes research that should be addressed by an agenda for outcomes research in nursing:

1. The time delay between intervention

and outcome, which makes attribution of cause and

effect difficult to ascertain;

A Research Agenda for Outcomes Research

Stephen C. Crane, PhD, MPH
2. The need to take account of nonmedical patient characteristics, such as health insurance coverage, income, and educational level, that

are important determinants of the effectiveness and

outcomes of health care;

3. The potential for multiple providers and multiple therapies in any given treatment

episode, which makes disaggregation of the impact and

effect of any single intervention very difficult;

4. The problem of a lack of good

baseline population measures of health status and

health-related quality of life, which makes analysis of

treatment effects and outcomes for individual patients

difficult to assess;

5. The autonomy of the patient in the

treatment process, which can further confound and

confuse the already difficult task of assessing treatment effectiveness and outcomes.

The paper concludes with the recommendation

to study patients as individuals and people, and not

simply as organs, diseases, conditions, or disabilities.

Introduction

The assignment that I was given for this session

was to talk about the effect of the interventions of different disciplines, or health professions, on the outcomes of care, or what have we learned about the conduct of outcomes research in other areas that will contribute to the formulation of a better focused and more

targeted agenda for outcomes resarch in nursing. To

this end, I would like to report briefly on the results of

a Congressionally-mandated conference, which was

held last April, that developed a research agenda for

outcomes research for the Agency for Health Care

Policy and Research (AHCPR) and the Congress. This

conference was conducted by the Foundation for

Health Services Research (FHSR) and the Alpha Center, with funding from AHCPR. The meeting involved

about 200 individuals from around the country representing both users (e.g., clinicians, public and private

policymakers, payers, etc.) and producers of outcomes

and health services research information.1 The meeting provided an excellent summary of what is known

about the conduct of outcomes research as of the

present moment, and therefore should be of use to this

meeting as it develops an agenda for outcomes research

in nursing.

I will highlight just a few of the major observations and recommendations of conference participants

in seven major areas:

1. The scope of outcomes research efforts;

2. Issues of measurement;

3. Issues of research design and methods;

4. Issues of data;

5. Factors determining the effectiveness of

treatments and the outcomes of care;

6. The potential sites for outcomes research;

and finally

7. Issues relating to the dissemination and

implementation of the findings from outcomes research.

In the interest of time, I will be relatively brief

in my listing of points in each area, and will leave for

later discussion any of the points I raise. Before beginning, however, I would like to say a few words

about the overall intent and focus of outcomes research.
Intent and focus of outcomes research

A good summary definition of outcomes research

is provided in a recent IOM report on conflict of interest in PORTs (Patient Outcomes Research Teams).

Let me read to you what it says:

Outcomes research examines the treatment of

clinical conditions rather than individual procedures

or treatments. It is the systematic assessment of clinical practice, encompassing both outcomes that are relevant to patients — mortality, morbidity, complications, symptom reduction, and functional status improvement — as well as physiologic or biologic indicators; it involves all reasonably held theories and alternative clinical practices.2

This is a useful definition because it emphasizes

two points that were given substantial emphasis in the

April conference:

1. First, the focus of outcomes research

is on conditions, and the alternative treatments for

addressing conditions, rather than on more narrowly

defined individual treatments. In effect, outcomes research asks the question “What works to achieve the

best results in terms of amelioration of a condition,”

rather than the more focused question of “How well

does a particular treatment work?”

2. Second, outcomes research is involved not just with clinical or physiological outcomes,

but with measures of health-related quality of life,

including physical and emotional functioning, general

perceptions of health and well-being, and satisfaction

with the process of care. I will say more about this in

a moment.

The April outcomes conference, however, also

emphasized a third very important point that is relevant to the proceedings of this conference, which is:

That the effectiveness and outcome of any

treatment process is a function as much of non-clinical factors, such as the socio-demographic characteristics of the patient, the organization of the delivery

of care, the financial incentives facing both patients

and providers, as it is of the actual clinical treatment

provided.

This is a critical point that bears emphasis as this

conference begins to develop an agenda for outcomes

research in nursing. This fact makes it abundantly

clear that an inter- and multi-disciplinary effort involving not only the clinical sciences, but the social,

behavioral and managerial sciences as well, is also

necessary. It also suggests that a broader definition

of outcomes research may be more helpful, such as

the following:

Outcomes research is any inquiry that is designed to measure — and ultimately improve — the

outcome of medical treatment and, thereby, the health

status of individuals and communities.

This definition would encompass efficacy, effectiveness as well as appropriateness research. This

definition is implicit in the recommendations of the

April conference that I now want to summarize for

you.

Let me turn my attention, now, to some of the

more detailed recommendations and comments from

the conference.
Scope of Outcomes Research

With respect to the scope of outcomes research,

the conference produced three major categories of recommendation.

First, the scope of outcomes research, particularly the research currently being conducted in PORTs,

should be broadened to include:

1. Chronic conditions and issues of prevention, as well as acute conditions. For instance, hypertension, congestive heart failure, diabetes, and arthritis, as well as early screening for problems such as

breast cancer and cervical cancer, and preventive programs such as smoking cessation and weight control,

should be evaluated. Nursing should be particularly

concerned with these issues.

2. This broadening should also include

issues of mental health treatment, and

3. should take account of a wider range

of age and cultural groups than is now the case. One

group singled out for particular attention by the April

conference was children, but other groups also mentioned were: blacks, hispanics, the very old, the institutionalized, rural populations; and the economically

disadvantaged.

Second, greater emphasis should be given to

identifying procedures that are ineffective, or perhaps

too costly to use, as well as searching out and reporting on procedures that are deemed to be “cost effective.”

Finally, a recommendation was made that more

family-oriented health outcomes should be studied. I

think that this recommendation has special relevance

for a nursing-oriented agenda for outcomes research.

Issues of measurement

With respect to issues of measurement, there

were five major categories of recommendation.

1. First, there is a need to continue to improve measures of both health status and health-related quality of life Health status measures include

indices of clinical, biological and physiological status, such as gross measures of morbidity and mortality, and more specific measures such as blood pressure, blood sugar, hemoglobin, temperature, etc.

Health-related quality of life measures include:

- physical functioning (for instance, Activities

of Daily Living [ADLs], Instrumental Activities of

Daily Living [IADL], mobility, etc.)

- emotional and psychological functioning and

well-being

- social functioning and support

- role functioning

- general health perceptions

- pain

- vitality (energy/fatigue)

- cognitive functioning

2. Second, and as a corollary to the first

point, there is a need to continue investments in the

development of measures of consumer preferences for

different outcomes and treatments. a) For instance,

we need to know more about the natural history and

course of events for particular diseases and conditions

over time. b) We also need to know more about the

health status and health-related quality of life levels
in broad population groups to improve the basis for

analysis of the outcomes and effectiveness of individual treatments.

3. Third, we need to improve our ability

to define episodes of care during which effectiveness

and outcomes are to be measured.

This is a difficult but most

fundamental task, because there is often considerable

uncertainty over when an “illness” or “condition” starts

or ends, or over what specific period of an illness or

condition, measurements of the effectiveness and outcomes of care should be made.

4. Fourth, we need to develop more sophisticated measures of the factors affecting outcomes

and effectiveness in three particular areas: (1) those

that relate to the socio-demographic and economic

characteristics of patients and providers; (2) those that

relate to the organizational characteristics of different

delivery systems; and (3) those that relate to the role

and influence of co-morbidity factors, such as the

present of other medical problems, the mental health

status of a patient, history of substance abuse, etc.

• Wherever possible, measures should be used

that reflect the “full scale of effect” of a variable influencing effectiveness of treatments and outcomes of

health care. Simple dichotomous measures, such as

yes/no, 0/1, etc., often do not provide sufficient precision to give a good indication of the actual effect of

different variables.

• Also, researchers need to be alert to the possibility of “threshold effects,” rather than simply presuming more simple models of continuous, linear effects.

5. Fifth, there is a need to separate generic from disease-specific measures of health status

and health-related quality of life to improve measurement of each, and to promote the appropriate use of

each measure in outcomes studies.

6. Sixth, we need to develop more precise measurements of the differences between and

changes in health-related quality of life measures with

respect to:

- the socio-demographic characteristics of patients (for instance, age, income, etc.);

- the geographic location of patients;

- the ethnic and cultural characteristics of patients; and, most importantly

- the severity of illness.

7. Finally, there is a need to develop better information on how measures of health status and

health-related quality of life vary over time. For instance:

- What are the optimal points to measure health

status and health-related quality of life?

- What are the trajectories of health and illness,

and how are “blips” in illness (or health) to be captured and interpreted meaningfully?

- How can life cycle effects

be taken into account? Specifically, how do people’s

values and preferences over the course of different

stages of their life and illness systematically affect or

bias their response to health status questions?
Issues of research design and methods

With respect to issues of research design and

methods, conference participants identified four areas

for further work:

1. First, there is a need to find ways to

increase the generalizability, or applicability, of the

results of outcomes and effectiveness research. For

instance, is there a way to redesign the research currently being done by PORTs to address additional

population groups beyond the elderly? How can we

get the most mileage from outcomes research to be

undertaken in the future?

2. Second, more studies of the health and

functional status of populations should be conducted

to complement the “micro-level” clinical studies of

individual patients. These studies would include both

populations enrolled and not enrolled in clinical test

protocols. Such studies would improve the understanding of how health status and health-related quality of

life measures vary across the population at large.

 Some participants went so far as to suggest the creation of a national profile of the general health and

functional status of the population, analogous to the

general vital statistics profiles currently available.

3. Third, there is a need to develop ways

of handling the significant problem of the time lag that

occurs between the intervention or provision of care,

and the ultimate end result of interest. This is one of

the most troubling methodological problems in outcomes research today, and one that has yet to be resolved successfully in many instances.

4. Finally, we need to improve our capabilities for synthesizing existing information through

meta-analyses and other techniques for information

and data summarization and presentation. Achieving

this objective will help to stretch our limited research

dollars, and make most effective use of the work already undertaken and completed.

Issues related to data

The fourth major area dealt with issues related

to data.

1. Conference participants indicated that

there was a substantial need to incorporate outcomes

measures into existing and new health services research data sets to enable greater analysis of outcomes

and effectiveness in the future, as well as

2. to increase opportunities for data

sharing among existing research projects and research

investigators. In addition to these two broad recommendations, four more specific recommendations were

also made:

1. First, there is a need to identify standard data elements and forms to promote consistency of data collection across studies. The lack of

consistent definitions and measures is perhaps the

single most important factor limiting the combination

and use of existing outcomes studies.

2. Second, we need to develop and use

a common patient identifier not only to allow health

records from different sources or time periods to be

linked together more easily, but also to link data from

education, welfare, employment, and other social service systems to provide a more complete profile of a
patient’s situation and characteristics. This task is less

difficult in an integrated system of care, like a Kaiser

or Harvard Community Health Plan, but unfortunately

only a small fraction of the population get their care

from such systems.

3. Third, we need to develop and implement a common scheme for labelling health care

events, such as the reason for a provider visit, to provide for greater continuity across research studies, and

to promote greater comparability among research findings.

4. Finally, we need to create appropriate longitudinal data sets over adequate periods of

time to assess the long term as well as short term effects of interventions. On this point, conference participants agreed on the need to educate public and private funders about the importance of longitudinal data

and the need to invest in its collection and dissemination, even given its cost.

Factors determining the effectiveness of treatments and the outcomes of care

The fifth major area of recommendations dealt

with the factors that contribute to or determine the effectiveness of alternative treatments and the outcomes

of care.

1. Conference participants emphasized

the importance of analyzing the influence of socioeconomic and demographic characteristics of patients

and providers on effectiveness of treatments and outcomes care. For instance among many possible factors, researchers should consider the impact of income

and health insurance status on the timeliness of seeking care. The impact of cultural factors, such as religion and tradition on compliance with treatment protocols and drug regimens, should also be assessed.

2. Other equally important factors determining outcomes that need to be considered include:

the different ways in which health services are organized and delivered, the personal attributes and characteristics of care givers, the financial incentives facing both providers and patients, and the availability

of information about alternative treatments and their

consequences, among others.

In general, there is a need to recognize and appreciate the truly complex nature of the health care

system, including such issues as:

• How the interactions among the different components of the delivery system affect the effectiveness

of treatments and the outcomes of care;

• The barriers that exist within the delivery system that impede the efficient and effective use of the

system;

• The nature of internal power structures and

the impact of external environmental factors, such as

the economy and public and private regulation, on the

function and operation of health delivery systems.

3. Third, it is necessary to critically assess the contribution of all types of health care personnel to the treatment process, no matter whether the

principal focus of study is on nursing, medicine, or

some other health profession.

4. Fourth, researchers must take greater

account of the role of consumer preferences in the

treatment process and, as already stated, must develop

better measures of these preferences. The nature of
the interaction between patients and providers may also

be an important determinant of the effectiveness and

outcomes of health care.

5. Finally, research needs to be undertaken to identify the differences in effectiveness that

may exist between primary and specialty-type care in

the treatment of conditions at early, mid and late stages

of disease or illness development.

Issues related to the site of care

With respect to the potential sites for outcomes

research, conference participants made several recommendations.

1. Most importantly, participants recommended that more research be done outside academic

medical care settings. For many reasons, academic

medical centers have been the focus of outcomes research, but these centers are not typical of the care

provided to most people. A broader array of sites needs

to be used. For instance:

a) The concept of the “firm” needs to be further

developed. As used in a research sense, firms represent clusters of patients, within a larger group practice, who have been assigned randomly to a panel of

health care providers. Patients are then monitored both

within and across firms to identify variations in practice patterns and differences in effectiveness and outcomes of health care. This type of research offers a

rich source of comparative data and information.

b) “Practice networks” link otherwise independent providers together again for the purpose of comparative analysis of patterns of treatment and effectiveness of care provided. One major advantage in

using networks, or other consortia of health care providers, is that sample sizes for analyses are often increased, which improves the power of the analyses

conducted.

c) Finally, managed care settings represent ideal

research laboratories, given that they are “closed health

care systems.” Because most all sources of care in

these systems either are known or can be tracked, more

thorough analysis can be made of the impact of both

clinical and non-clinical factors on treatment effectiveness and care outcomes.

Issues related to dissemination and implementation

of the findings from outcomes research

Issues of the dissemination and implementation

of the findings of outcomes research are usually not

considered to be a part of mainstream research activities, but these are vitally important and crucial parts

of the whole outcomes effort. Dissemination is also

an area where we know relatively little, but where investments in research can return relatively large dividends. For instance:

1. We need to explore the theory and

practice of dissemination of information as it relates

to different audiences, including: various types of

health care providers; patients; payers; policymakers;

and health services managers. With respect to these

audiences, we need to ask the question: “What techniques work for which groups and how well?” Dissemination techniques that need to be considered include:

• guidelines and practice standards;

• research reports published in professional

journals;

• news articles presented in the lay media; and
• formal and informal professional communications.

2. Second, we need to examine different models for how consumer and provider behavior

can most effectively and efficiently be changed. Some

of the methods that need to be considered include:

• feedback on performance;

• education;

• peer pressure;

• administrative regulation;

• financial and non-financial incentives;

• financial and non-financial penalties.

Most importantly, we need to undertake basic

research into the methods of persuasion that can most

effectively influence the use of research results in both

clinical and policy settings.

3. Third, we need to develop mechanisms for resolving conflicts that may arise between

different guidelines or practice standards that may be

promulgated for the same condition. It is inevitable

that such conflicts will emerge, and we need to have

some way, in advance, for resolving these disagreements.

4. Fourth, we also need to create a strategy for monitoring trends in practice patterns and for

identifying changes in these patterns over time that

may be the result of outcomes research findings.

5. Finally, we need to determine how

feedback of information and information management

systems can positively (and negatively) affect patient

outcomes.

Special difficulties and obstacles in conducting outcomes research

Before concluding, I would like to highlight five

potential problems in the conduct of outcomes research

that have already been mentioned, but which are deserving of emphasis based on the April conference

recommendations. These are:

1. The time delay between intervention

and outcome, which makes attribution of cause and

effect (as in the case of treatment for multiple chronic

conditions) difficult to ascertain with any degree of

certainty;

This problem is particularly important in studies

of the effectiveness of health promotion activities for

conditions that may not reveal themselves as problems

for years.

2. The effect of non-medical patient

characteristics, such as health insurance coverage, income, and educational level, that influence the outcomes of health treatments;

Although these characteristics are difficult to measure, they ut often confound

the results of studies when they are not adequately

controlled.

3. The potential for multiple providers and multiple therapies in any given treatment

episode, which can make disaggregation of the impact

and effect of any single intervention very difficult;

4. The problem of a lack of good

baseline population measures of health status and

health-related quality of life, which makes analysis of

treatment effects and outcomes for individual patients

difficult to assess;
5. And finally, the autonomy of the

patient in the treatment process, which can further

confound and confuse the already difficult task of assessing treatment effectiveness and outcomes.

Conclusion

It is perhaps appropriate to conclude with a recommendation that had importance for many of the

April conference participants, and I know will have

great relevance for the field of nursing as well. That

recommendation is:

To study patients as individuals and people,

and not simply as organs, diseases, conditions, or disabilities.

One of the significant contributions of the growing field of functional health status within outcomes

research is to move us towards this goal, and it remains an important objective for you to meet as you

develop an outcomes research agenda for nursing.

Thank you.

 2 Molla Donaldson and Alexamder Capron,

eds. Patient Outcomes Research Teams: Managing

Conflict of Interest (Washington: National Academy

Press, 1991), p. 9.