Chapter 8. Health Services Research: 

Scope and Significance 

Donald M. Steinwachs, Ronda G. Hughes 

Background 

The provision of high-quality, affordable, health care services is an increasingly difficult 

challenge. Due to the complexities of health care services and systems, investigating and 

interpreting the use, costs, quality, accessibility, delivery, organization, financing, and outcomes 

of health care services is key to informing government officials, insurers, providers, consumers, 

and others making decisions about health-related issues. Health services researchers examine the 

access to care, health care costs and processes, and the outcomes of health services for 

individuals and populations. 

The field of health services research (HSR) is relied on by decisionmakers and the public to 

be the primary source of information on how well health systems in the United States and other 

countries are meeting this challenge. The “goal of HSR is to provide information that will 

eventually lead to improvements in the health of the citizenry.”1 Drawing on theories, 

knowledge, and methods from a range of disciplines,2 HSR is a multidisciplinary field that 

moves beyond basic and applied research, drawing on all the health professions and on many 

academic disciplines, including biostatistics, epidemiology, health economics, medicine, nursing, 

operations research, psychology, and sociology.3 

In 1979, the Institute of Medicine defined HSR as “inquiry to produce knowledge about the 

structure, processes, or effects of personal health services”4 (p. 14). This was expanded upon in 

2002 by AcademyHealth, the professional organization of the HSR field, with the following 

definition, which broadly describes the scope of HSR: 

Health services research is the multidisciplinary field of scientific investigation 

that studies how social factors, financing systems, organizational structures and 

processes, health technologies, and personal behaviors affect access to health 

care, the quality and cost of health care, and ultimately our health and well-being. 

Its research domains are individuals, families, organizations, institutions, 

communities, and populations.5

More specifically, HSR informs and evaluates innovations in health policy. These include 

changes in Medicare and Medicaid coverage, disparities in access and utilization of care, 

innovations in private health insurance (e.g., consumer-directed health plans), and trends among 

those without health insurance.6–10 The health care industry continues to change, and HSR 

examines the impact of organizational changes on access to care, quality, and efficiency (e.g., 

growth in for-profit hospital systems). As new diagnostic and treatment technologies are 

introduced, HSR examines their impact on patient outcomes of care and health care costs. 

The definition of HSR also highlights the importance of examining the contribution of 

services to the health of individuals and broader populations. HSR applied at the population level 

is particularly important in understanding health system performance and the impact of health 

policy on the public’s health. In the United States, the National Healthcare Quality Report,11

National Healthcare Disparities Report,12 and Healthy People Year 201013 exemplify our 

1 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

capacity for monitoring quality and assessing change. These reports tell us that the American 

quality of care is inconsistent and could be substantially improved. The associated cost of health 

care services is monitored by the Centers for Medicare & Medicaid Services (CMS). CMS 

reports tell us that American health care is the most expensive in the world, consuming 

approximately 16 percent of America’s gross domestic product.14

Beyond health policy, HSR examines the process of care and the interactions of patients and 

providers. For example, HSR methods have been developed to describe doctor-patient 

communication patterns and examine their impact on patient adherence, satisfaction, and 

outcomes of care.15–17 

Advances in HSR measurement methodologies have made possible policy innovations. 

Prospective payment of hospitals, nursing homes, and home health care by Medicare became 

possible with the development of robust case-mix measurement systems.18 CMS was able to 

initiate a pay-for-performance demonstration, rewarding hospitals with better quality 

performance, using valid and robust measures of quality.14 Innovations in health care policy are 

frequently made possible by advances in measurement of indicators of health system 

performance. 

History of Health Services Research 

The history of HSR is generally considered to have begun in the 1950s and 1960s with the 

first funding of grants for health services research focused on the impact of hospital 

organizations.19, 20 On the contrary, HSR began with Florence Nightingale when she collected 

and analyzed data as the basis for improving the quality of patient care and outcomes.21 Also 

significant in the history of HSR was the concern raised about the distribution, quality, and cost 

of care in the late 1920s that led to one of the first U.S. efforts to examine the need for medical 

services and their costs, undertaken in 1927 by the Committee on the Costs of Medical Care.22 

The committee published a series of 28 reports and recommendations that have had a significant 

impact on how medical care is organized and delivered in the United States.23 Other key reports 

of historical importance to HSR were, for example, the national health survey in 1935–1936 by 

the Public Health Service, the inventory of the nation’s hospitals by the American Hospital 

Association’s Commission on Hospital Care in 1944, and studies by the American Hospital 

Association’s Commission on Chronic Illness on the prevalence and prevention of chronic illness 

in the community.23

In 1968, the National Center for Health Services Research and Development was established 

as part of the U.S. Public Health Services to address concerns with access to health services, 

quality of care, and costs. The Center funded demonstration projects to measure quality and 

investigator-initiated research grants. In 1989, Congress created the Agency for Health Care 

Policy and Research and broadened its mission to focus attention on variations in medical 

practice, patient outcomes of care, and the dissemination of evidence-based guidelines for the 

treatment of common disorders. Later Congress reauthorized and renamed the agency, Agency 

for Healthcare Research and Quality (AHRQ). AHRQ provides Federal leadership for the field, 

investing in methods for quality measurement, development of patient safety methods, and health 

information technology (e.g., electronic health records and decision support systems). 

The Federal role in HSR has expanded over time, and investments in HSR are made by 

multiple Federal agencies. In addition to AHRQ, the U.S. Department of Veterans Affairs, 

Centers for Disease Control and Prevention, the National Institutes of Health, CMS, and other 

2 
Health Services Research

Federal agencies fund HSR. The diversification of funding comes, in part, from the recognition 

that HSR is important in managing health care systems, such as the Veterans Health 

Administration, and provides essential information on the translation of scientific discoveries 

into clinical practice in American communities, such as those funded by National Institutes of 

Health. It is estimated that total Federal funding of HSR was $1.5 billion in 2003, of which 

AHRQ was responsible for approximately 20 percent.24

Private funding of HSR has also grown over time. Funding by private foundations has a 

significant role and complements Federal funding. Among the many foundations funding HSR 

are the Robert Wood Johnson Foundation, Commonwealth Fund, Kaiser Family Foundation, 

Kellogg Foundation, and Hartford Foundation. Other private funding sources include the health 

care industry, for example, pharmaceutical companies, health insurers, and health care systems. 

Goals for Health Services and Patient Outcomes 

The goal of health services is to protect and improve the health of individuals and 

populations. In a landmark 2001 report, Crossing the Quality Chasm: A New Health System for 

the 21st Century,25 the Institute of Medicine (IOM) of the National Academy of Sciences 

proposed that the goals for health services should include six critical elements: 

1. Patient Safety: Patients should not be harmed by health care services that are intended to 

help them. The IOM report, To Err Is Human,

26 found that between 46,000 and 98,000 

Americans were dying in hospitals each year due to medical errors. Subsequent research has 

found medical errors common across all health care settings. The problem is not due to the 

lack of dedication to quality care by health professionals, but due to the lack of systems that 

prevent errors from occurring and/or prevent medical errors from reaching the patient. 

2. Effectiveness: Effective care is based on scientific evidence that treatment will increase the 

likelihood of desired health outcomes. Evidence comes from laboratory experiments, clinical 

research (usually randomized controlled trials), epidemiological studies, and outcomes 

research. The availability and strength of evidence varies by disorder and treatment. 

3. Timeliness: Seeking and receiving health care is frequently associated with delays in 

obtaining an appointment and waiting in emergency rooms and doctors’ offices. Failure to 

provide timely care can deny people critically needed services or allow health conditions to 

progress and outcomes to worsen. Health care needs to be organized to meet the needs of 

patients in a timely manner. 

4. Patient Centered: Patient-centered care recognizes that listening to the patient’s needs, 

values, and preferences is essential to providing high-quality care. Health care services 

should be personalized for each patient, care should be coordinated, family and friends on 

whom the patient relies should be involved, and care should provide physical comfort and 

emotional support. 

5. Efficiency: The U.S. health care system is the most expensive in the world, yet there is 

consistent evidence that the United States does not produce the best health outcomes27–30 or 

the highest levels of satisfaction.31 The goal is to continually identify waste and inefficiency 

in the provision of health care services and eliminate them. 

6. Equity: The health care system should benefit all people. The evidence is strong and 

convincing that the current system fails to accomplish this goal. The IOM report, Unequal 

Treatment,

32 documented pervasive differences in the care received by racial and ethnic 

minorities. The findings were that racial and ethnic minorities are receiving poorer quality of 

3 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

care than the majority population, even after accounting for differences in access to health 

services. 

Crossing the Quality Chasm concludes that for the American health care system to attain 

these goals, transformational changes are needed.25 The field of HSR provides the measurement 

tools by which progress toward these goals is assessed, as seen in the National Healthcare 

Quality Report.11 Equally important, health services researchers are developing and evaluating 

innovative approaches to improve quality of care, involving innovations in organization, 

financing, use of technology, and roles of health professionals. 

Evaluating the Quality of Health Care 

HSR evaluation of quality of care has proven to be an inexact science and complex, even 

though its definition is relatively simple: “Quality of care is the degree to which health services 

for individuals and populations increase the likelihood of desired health outcomes and are 

consistent with current professional knowledge.”1 This definition draws attention to the 

importance of the application of current professional knowledge in the diagnostic and treatment 

processes of health care. The goal of quality care is to increase the likelihood of achieving 

desired health outcomes, as expressed by the patient. 

The complexity in measuring quality comes from gaps in our knowledge regarding which 

services, for which patients, will actually improve the likelihood of desired health outcomes. 

Also, patients need not have the same desired health outcomes and therefore might not receive 

the same care for an identical health problem, further complicating the measurement of quality of 

care. Quality measurement has advanced substantially, but it remains early in its development. 

The conceptual framework widely applied in evaluating quality comes from years of research 

and the insightful analysis of Avedis Donabedian.33 He formalized the conceptual model for 

describing, analyzing, and evaluating the quality of care using three dimensions: (1) structure, (2) 

process, and (3) outcome. This model is applied in the evaluation of health services and the 

accreditation of health care providers and organizations. 

Seminal research about variation in the quality of care patients received brought to focus the 

need to monitor and improve the quality of health care. Wennberg and Gittelsohn34, 35 found 

wide variation in practice patterns among community physicians, surgical procedures, and 

hospitals. Brook and colleagues36 found that a small number of physicians were responsible for a 

large number of improperly administered injections. This was the precursor to research on the 

appropriateness of procedures and services under specific circumstances36, 37 as well as the 

development of practice guidelines and standards for quality care.38 Yet the challenge of research 

on variations in care is the implication of the inappropriateness of care. The challenge is 

determining whether there is a direct relationship between rates of utilization, variations in 

appropriateness, and quality of care.

One of the challenges in understanding quality, how to measure it, and how to improve it is 

the influence of physical, socioeconomic, and work environments. Income, race, and gender—as 

well as individuals within society and organizations—influence health and risks to health.40

Researchers have found that differences in internal factors, such as collaborative relationships 

with physicians, decentralized clinical decisionmaking, and positive administrative support, 

impact nurse and patient outcomes41, 42 and the quality and safety of care.43 Differences in 

external factors, such as insurance and geographic location, can influence access to available 

health care professionals and resources, what type of care is afforded patients, and the impact of 

4 
Health Services Research

care on patients. The structure, process, and outcome dimensions of quality are influenced by 

both internal and external factors. 

Structure of Health Care 

The structure of health care broadly includes the facilities (e.g., hospitals and clinics), 

personnel (e.g., number of nurses and physicians), and technology that create the capacity to 

provide health services. Structural characteristics are expected to influence the quality of health 

care services. One component in the accreditation of health care facilities (e.g., hospitals, nursing 

homes) is the review of the adequacy of structural characteristics, including staffing, on-call 

resources, technology, and support services (laboratory, pharmacy, radiology). The structural 

resources of health care facilities and organizations are the foundation upon which quality health 

care services are provided. 

Process of Care 

The interactions between the health care providers and patients over time comprise the 

process of health care. The process of care may be examined from multiple perspectives: the 

sequence of services received over time, the relationship of health services to a specific patient 

complaint or diagnosis, and the numbers and types of services received over time or for a 

specific health problem. Examining the time sequence of health care services provides insights 

into the timeliness of care, organizational responsiveness, and efficiency. Linking services to a 

specific patient complaint or diagnosis provides insights into the natural history of problem 

presentation and the subsequent processes of care, including diagnosis, treatment, management, 

and recovery. Examining the natural history of a presenting health complaint across patients will 

reveal variations in patterns of care. For example, presenting complaints for some patients never 

resolve into a specific diagnosis. An initial diagnosis may change as more information is 

obtained. Patients may suffer complications in the treatment process. Also, the process of care 

may provide insights into outcomes of care (e.g., return visit for complications). Generally it is 

not possible to examine the process of care and determine how fully the patient has recovered 

prior health status by the end of the episode of treatment. For this reason, special investigations 

are needed to assess outcomes of care. 

Evaluation of the process of care can be done by applying the six goals for health care 

quality.25 Was the patient’s safety protected (i.e., were there adverse events due to medical errors 

or errors of omission)? Was care timely and not delayed or denied? Were the diagnosis and 

treatments provided consistent with scientific evidence and best professional practice? Was the 

care patient centered? Were services provided efficiently? Was the care provided equitable? 

Answers to these questions can help us understand if the process of care needs improvement and 

where quality improvement efforts should be directed. 

Outcomes of Care 

The value of health care services lies in their capacity to improve health outcomes for 

individuals and populations. Health outcomes are broadly conceptualized to include clinical 

measures of disease progression, patient-reported health status or functional status, satisfaction 

with health status or quality of life, satisfaction with services, and the costs of health services. 

5 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

Historically, quality assessment has emphasized clinical outcomes, for example, disease-specific 

measures. However, disease-specific measures may not tell us much about how well the patient 

is able to function and whether or not desired health outcomes have been achieved. To 

understand the patients’ outcomes, it is necessary to ask patients about their outcomes, including 

health status, quality of life, and satisfaction with services. HSR has developed valid and robust 

standardized questionnaires to obtain patient-reported information on these dimensions of health 

outcomes. As these are more widely applied, we are learning about the extent to which health 

care services are improving health. 

Public Health Perspective on Health Services 

Another perspective on health care services comes from the field of public health in which 

preventive health services are conceptualized at three levels: primary, secondary, and tertiary 

prevention.44 Primary prevention includes immunizations, healthy lifestyles, and working and 

living in risk-free environments. Primary prevention seeks to prevent disease or delay its onset. 

Examples of primary prevention include immunizations against infectious disease; smoking 

prevention or cessation; and promotion of regular exercise, weight control, and a balanced diet. 

Secondary prevention includes the range of interventions that can reduce the impact of disease 

morbidity once it occurs and slow its progression. With the increasing burden of chronic 

diseases, much of the health care provided is directed at secondary prevention. Tertiary 

prevention is directed at rehabilitation for disabilities resulting from disease and injury. The goal 

of tertiary prevention is to return individuals to the highest state of functioning (physical, mental, 

and social) possible. The public health framework expands the structure, process, and outcome 

conceptual model by identifying the role and value of health services at three stages: prior to 

onset of disease, disease management, and disease recovery and rehabilitation. 

Methodologies and Data Sources Used in Health 

Services Research 

The interdisciplinary character of HSR draws on methods and data sources common to the 

many disciplines that form the intellectual underpinnings of the field. This section discusses the 

measurement of effectiveness and efficacy of health services and some of the methods and data 

sources used to understand effectiveness. Effectiveness is one of the six goals of health services. 

Effectiveness is interrelated with the other five goals, and some of these interrelationships are 

discussed. 

Efficacy and Effectiveness 

An important distinction is made between efficacy and effectiveness of health services. 

Efficacy is generally established using randomized controlled trial (RCT) methods to test 

whether or not clinical interventions make a difference in clinical outcomes. A good example is 

the series of studies required for Food and Drug Administration approval of a new drug before it 

is certified as safe and efficacious and allowed to be used in the United States. Efficacy research 

is generally done with highly select groups of patients where the impact of the drug can be 

validly measured and results are not confounded by the presence of comorbid conditions and 

6 
Health Services Research

their treatments. The efficacy question is: What impact does a clinical intervention have under 

ideal conditions? 

In contrast, effectiveness research is undertaken in community settings and generally 

includes the full range of individuals who would be prescribed the clinical intervention. Many of 

these individuals will have multiple health problems and be taking multiple medications, unlike 

those who were recruited to the RCT. Effectiveness research is seeking to answer the question: 

Who will benefit from the clinical intervention among all those people in the community who 

have a specific health problem(s)? 

Both efficacy and effectiveness questions are important. Logically, effectiveness research 

would be conducted after finding the clinical intervention to be efficacious. However, there are 

many treatments for which no efficacy information exists; the treatments are accepted as 

common practice, and it would not be ethical to withhold treatments from a control group in an 

RCT. As a result, effectiveness research may not have the benefit of efficacy findings. 

The routine use of an RCT to evaluate efficacy began in the 1960s and is the accepted 

procedure for evaluating new medications. However, this standard is not applied across all health 

care services and treatments. Most surgical procedures are not evaluated using an RCT. Intensive 

care units have never been evaluated using an RCT, nor are nurse staffing decisions in hospitals 

or the evaluation of many medical devices. We currently accept different standards of evidence 

depending on the treatment technology. As a result, the level of evidence guiding clinical and 

public health decisionmaking varies. 

Methods for Effectiveness Research 

A variety of methods are used to examine effectiveness of health services. RCT methods are 

not usually applied in effectiveness research because the intervention being studied has 

demonstrated efficacy or is acknowledged as accepted clinical practice. When this is true, it 

would be unethical to randomly assign individuals who would be expected to benefit from the 

intervention to a control group not receiving an efficacious treatment. We will discuss when 

RCT methods can be used to test effectiveness and provide several examples. More commonly, 

effectiveness research uses statistical methods for comparing treatments across nonequivalent 

groups. 

RCT and Policy Research 

RCT study methods can be used to compare the effectiveness and costs of services across 

randomly assigned representative population groups. In an RCT, study participants are randomly 

assigned to two or more groups to ensure comparability and avoid any selection bias. At least 

one group receives an intervention (clinical, organizational, and/or financial), and usually one 

group serves as a control group, receiving a current standard of care, sometimes referred to as 

“usual care.” Two examples of effectiveness research using an RCT methodology to answer 

policy questions are described. 

Health insurance experiment. Probably the first application of RCT methods in 

effectiveness research was undertaken in the 1970s as a health insurance experiment. The 

experiment was designed to test the impact on cost and health outcomes of different levels of 

insurance deductibles and copayment rates. A total of 3,958 people, ages 14–61, were 

randomized to a set of insurance plans and followed over 3 to 5 years.45

7 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

The economic impact of receiving free care in one plan versus being in a plan requiring 

payment out-of-pocket of deductibles and co-insurance had the expected impact on utilization. 

Those paying a share of their medical bills utilized approximately one-third fewer doctor visits 

and were hospitalized one-third less frequently. 

The impact on 10 health measures of free health insurance versus paying a portion of medical 

care costs out of pocket was evaluated. The findings were that there was largely no effect on 

health as measured by physical functioning, role functioning, mental health, social contacts, 

health perceptions, smoking, weight, serum cholesterol, diastolic blood pressure, vision, and risk 

of dying.46 The exceptions were that individuals with poor vision improved under free care, as 

did low-income persons with high blood pressure. 

Medicare preventive services experiment. A more recent example of RCT methods applied 

in HSR is the Baltimore Medicare Preventive Services Demonstration. The study evaluated the 

impact on cost and outcomes of offering a defined preventive services package to Medicare 

beneficiaries. This was compared to usual Medicare coverage, which paid for few preventive 

services. The preventive services coverage being evaluated included an annual preventive visit 

with screening tests and health counseling. The physician could request a preventive followup 

visit during the year, which would also be covered. Medicare beneficiaries (n = 4,195) were 

randomized to preventive services (the intervention group) or usual care (the control group). 

Sixty-three percent of those in the intervention group had at least one preventive visit. 

Significant differences were found in health outcomes between intervention and control groups. 

Among the 45 percent with declining health status, as measured by the Quality of Well-Being 

scale,47 the decline was significantly less in the group offered preventive services. Mortality was 

also significantly lower in the intervention group. There was no significant impact of preventive 

services on utilization and cost.48

Comparative Clinical Effectiveness and RCTs 

The passage of the Medicare Prescription Drug, Improvement, and Modernization Act of 

2003 (MMA) included provisions for the funding of comparative effectiveness studies. AHRQs’ 

Effective Health Care Program (authorized under MMA Section 1013) informs comparative 

clinical effectiveness efforts by conducting and supporting research and evidence syntheses on 

priority topics to CMS. 

Comparative effectiveness studies ask the question: Which of the alternative treatments 

available is best and for whom? Interest in this question reflects how advances in science have 

provided multiple treatment options for many conditions. Currently, there is no systematic 

process by which treatment options are compared and matched to the needs of different types of 

patients. Frequently, patients are started on one treatment and then may be prescribed alternative 

treatments if they cannot tolerate the treatment or if it is not as effective as expected. RCT 

methods can be used to evaluate comparative effectiveness of an intervention in treatment and 

control populations. This is ethical to do when there is no evidence that the treatments are not 

equivalent. 

An example of a comparative effectiveness study using RCT methods is the CATIE study, 

testing alternative antipsychotic medications in the treatment of schizophrenia. A study of 1,493 

persons with schizophrenia compared five of the newer antipsychotic medications (second 

generation) and also compared them against one of the first-generation antipsychotic 

medications.49, 50 The findings were surprising to many. The second-generation antipsychotics 

8 
Health Services Research

were no more effective in controlling psychotic symptoms than the first-generation drug. There 

was one exception, the drug Clozapine.51 Furthermore, second-generation medications showed 

significant side effects that can affect health outcomes. These included weight gain, metabolic 

changes, extrapyramidal symptoms, and sedation effects. Each medication showed a somewhat 

different side-effect risk profile. From a positive perspective, the findings indicated that the 

clinician and patient can choose any of these medications as first-line treatment except 

Clozapine, which is generally used for treatment-resistant cases due to more intensive clinical 

monitoring requirements. The ultimate choice of treatment will depend on the patient’s ability to 

tolerate side effects that vary by drug. 

The conduct of any RCT is resource intensive, requiring the recruitment of participants, and 

participants must give informed consent to be randomized. The rationale for making this 

investment may depend on the importance of the policy or practice issue. As shown, RCT 

methods can be applied to address policy and clinical care concerns with effectiveness. To the 

extent that the RCT includes a broad cross-section of people who would be affected by a policy 

or receive a clinical treatment, this methodology provides robust effectiveness findings. 

Comparing Effectiveness and Costs Across Nonequivalent Groups 

A range of statistical methods can be used to compare nonequivalent groups (i.e., groups 

receiving different treatments or exposures when there has been no random assignment to ensure 

comparability of group membership). It is not practical to review all the specific statistical 

approaches that can be applied. In general, the statistical methods seek to adjust for 

nonequivalent characteristics between groups that are expected to influence the outcome of 

interest (i.e., make the comparisons fair). Statistical adjustment for nonequivalent characteristics 

is referred to as “risk adjustment.” The foundations for risk adjustment come from multiple 

disciplines. Epidemiologic methods are routinely used to identify and estimate disease and 

outcomes risk factors. These methods are applicable in comparative effectiveness evaluations.52

Operations research uses methods for creating homogeneous groups predictive of cost or 

disease outcomes. These methods are used to make fair comparisons across provider practices 

and health plans and to control the cost of health care. They also have been used in designing 

payment systems, including diagnostically related groups used in Medicare’s Prospective 

Payment System to reimburse hospitals for care rendered to Medicare beneficiaries, and 

resource-based relative value scales used in Medicare’s physician payment system. 

Diagnostically related groups are used to standardize and rationalize patient care in hospitals—

provided largely by nurses and other health professionals—and resource-based relative value 

scales are used to standardize and rationalize patient care in outpatient settings—care provided 

largely by physicians and nurse practitioners. Other disciplines also contribute to our 

understanding of risk factors for the range of health outcomes, including mortality, health and 

functional status, quality of life, and rehabilitation and return to work 

The basic form of a nonequivalent group comparison includes adjusting the outcomes of each 

group for the risk factors that are known to affect the occurrence and/or severity of the outcomes 

being evaluated.53 For many disease outcomes, risk factors include demographic characteristics 

(age, gender), disease-specific risk factors (e.g., health behaviors, environmental exposures, and 

clinical indicators of risk), and indicators of health status (e.g., presence of comorbid conditions). 

After adjustment for risks factors, variations in access to care and quality of care (e.g., choice of 

treatment and adherence to treatment) would be expected to explain the remaining observed 

9 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

variation in outcomes. Ideally, the nonequivalent group comparison makes it possible to compare 

the effectiveness of alternative treatments and assess the impact of poor access to care. One 

limitation of this methodology is the limit of current knowledge regarding all relevant disease 

risk factors. Even when risk factors are known, limits on data availability and accuracy of risk 

factor measurement have to be considered. 

Risk adjustment methods are also used to make cost comparisons across health care 

providers to determine which providers are more efficient. Instead of adjusting for disease risk 

factors, adjustments are made for the costliness of the patient mix (case mix) and differences in 

costs of labor, space, and services in the local area. Comparisons may be made to assess 

efficiency of providing specific services (e.g., hospitalization, office visit, or laboratory test). 

These comparisons would use case-mix measures that adjust for the costliness of different mixes 

of hospital episodes.18 Comparisons of the total cost of care for insured populations would apply 

case-mix measures that adjust for disease and health factors that affect total cost of care.54 

Data Sources for Effectiveness Research 

A range of data sources is used in effectiveness research, including administrative and billing 

data, chart reviews and electronic health records, and survey questionnaires. The following 

discussion identifies major attributes of each category of data source. 

Medical records. Medical records document the patient’s presenting problem or condition, 

tests and physical exam findings, treatment, and followup care. The medical record is generally 

the most complete source of clinical information on the patient’s care. However, medical records 

are generally not structured to ensure the physician or other provider records all relevant 

information. The completeness of medical record information can vary considerably. If the 

patient does not return for followup care, the medical record may provide no information on 

outcomes of care. If a patient sees multiple providers during the course of treatment, each with 

its own separate medical record, complete information on treatment requires access to all the 

records. Lack of standardization of medical records also can make abstracting records for 

research very resource intensive. 

Administrative and billing data. Health care providers generally have administrative and 

billing data systems that capture a limited and consistent set of data on every patient and service 

provided. These systems uniquely identify the patient and link information on insurance 

coverage and billing. Each service received by the patient is linked to the patient using a unique 

patient identifier. Services are identified using accepted codes (e.g., ICD9-CM, CPT), together 

with date of service, provider identifier, and other relevant information for billing or 

management reporting. Administrative data make it possible to identify all individual patients 

seen by a provider and produce a profile of all services received by each patient over any defined 

time period. Administrative data are comprehensive and the data are generally complete (i.e., no 

problems with missing data). The primary limitation is the data set collected by administrative 

systems is very limited and lacks the detail of the medical record. 

Administrative data systems can provide some insights into quality and outcomes of care. 

AHRQ has developed software that provides quality indicators and patient safety measures using 

one administrative data set, hospital discharge abstracts.55, 56 Utilization-based indicators of 

outcome include rehospitalization, return to surgery during a hospitalization, and incidence of 

complications; some systems include information on death. Administrative data can efficiently 

provide quality and outcomes indicators for defined populations and for health systems. Other 

10 
Health Services Research

applications of administrative data include assessing efficiency, timeliness, and equity. The 

limitation is that there are many health conditions and health outcomes that cannot currently be 

measured using administrative data. 

Survey questionnaires. Neither the medical record nor the administrative data capture 

information on the patient’s experience in health or patient-reported outcomes of care. Survey 

questionnaires are routinely used to obtain information on patient satisfaction in health plans. A 

widely used example is the Consumer Assessment of Healthcare Providers and Systems or 

CAHPS.57 

Information on the impact of health conditions on health and functional status has to come 

from the patient. This may be obtained at the time of a visit or hospitalization. However, to 

assess patient outcomes of care, systematic followup of patients after the completion of treatment 

is generally required. This can be done using mail questionnaires, telephone interviews, or inperson interviews. The HSR field has developed health-status and quality-of-life measures that 

can be used no matter what health conditions the patient has.47, 58–60 Numerous condition-specific 

measures of outcome are also used.53 

Effectiveness research relies on a range of data sources. Some are routinely collected in the 

process of medical care and patient billing. Others may require special data collection, including 

medical record abstracts to obtain detailed clinical data and survey questionnaires to gain 

information on the patient’s perspective on treatment and outcomes. Efficient strategies for 

examining effectiveness may use administrative data to examine a limited set of data on all 

patients, and a statistically representative sample of patients for in-depth analysis using data from 

chart abstracts and survey questionnaires. 

Using HSR Methods To Improve Clinical Practice 

HSR research tools can be applied in clinical settings to improve clinical practice and patient 

outcomes. These tools are used as part of quality improvement programs in hospitals, clinics, and 

health plans. Two examples illustrate applications to improve quality-of-care performance. 

Evidence-based treatment. For many chronic medical conditions, clinical research has 

evaluated the efficacy of diagnostic methods and treatment interventions. As a result, evidencebased reviews of research literature can provide a basis for establishing quality-of-care criteria 

against which to judge current practice. In a national study of quality of medical care, it was 

found that only 55 percent of patients received evidence-based treatments for common disorders 

and preventive care.61 The researchers examined treatment for a range of health conditions, using 

a national sample of medical records abstracts. For each quality criterion, a classification was 

applied to determine if the quality-of-care deficiency was one of underuse, overuse, or misuse. 

Greater problems were found with underuse (46 percent) than with overuse (11 percent). Quality 

of care varied by condition: senile cataracts scored highest, 78 percent of recommended care 

received, and alcohol dependence scored lowest, 10 percent of recommended care received. 

Overall, only about half of recommended care was received, frequently due to underuse of 

services. 

Researchers have sought to identify why rates of conformance with evidence-based 

treatments are low. Frequently cited barriers to evidence-based practice include physician 

disagreement with the evidence, perception that patients will not accept treatment, low ratings of 

self-efficacy as a provider of the treatment, and difficulty of integrating the evidence-based 

11 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

treatment into existing practice.62 More needs to be learned how to assist health care providers to 

overcome barriers to the adoption of evidence-based practices. 

The described data sources and methods can be applied in clinical settings to assess 

conformance to evidence-based quality criteria and provide feedback to clinicians. If electronic 

health records are available, the feedback and reminders may be directly incorporated into the 

medical record and seen by the clinician at the time of a visit. Intermountain Health Care utilizes 

its electronic health records to monitor adherence to evidence-based quality standards and to 

provide decision support to clinicians when seeing patients. This strategy has contributed to 

substantial improvements in their quality performance.63

Outcomes management system. In 1988, Paul Ellwood proposed the adoption of outcomes 

management system (OMS) as a method to build clinical intelligence on “what treatments work, 

for whom, and under what circumstances.”64 OMS would require linking information on the 

patient’s experience with outcomes of care and information on diagnosis and treatment that 

would usually come from the medical record. 

In 1991, the Managed Health Care Association, an employer organization, brought together a 

group of employers and their health plan partners who were interested in testing the OMS 

concept in health plans.65 To do so would require a set of methods that could be widely applied 

across health plans with differing information systems. The methodology chosen was for each of 

16 health plans to identify all adult enrollees with at least two diagnoses of asthma over the 

previous 2 years. A stratified sample was chosen with half of the enrollees having more severe 

asthma (e.g., hospitalization or emergency room visit in the past 2 years) and the other enrollees 

having less severe asthma (outpatient visits only). Each adult received a questionnaire asking 

about their asthma treatment and health status. Followup surveys were done in each of 2 

successive years to track changes over time. 

The findings were compared to national treatment recommendations for adult asthma.66

Across the health plans, 26 percent of severe asthmatics did not have a corticosteroid inhaler, and 

42 percent used it daily, as recommended.67 Only 5 percent of patients reported monitoring their 

asthma using a home peak flow meter. Approximately half of adults with asthma reported having 

the information they needed to avoid asthma attacks, to take appropriate actions when an asthma 

flare-up occurs, and to adjust medications when their asthma gets worse. Health plans used the 

baseline findings to develop quality-improvement interventions, which varied across health 

plans. Followup surveys of the patient cohort provided feedback to health plans on their success 

in improving asthma treatment and outcomes over time. 

Conclusion 

This chapter has provided a definition and history of the field of health services research and 

discussed how this field is examining quality-of-care issues and seeking to improve quality of 

care. Comparisons of current practice to evidence-based standards with feedback to clinicians 

and the integration of patient-reported outcomes are two examples of how HSR tools can be used 

to provide quality-improvement information for health care organizations. These examples 

utilize multiple data sources, including medical records, patient surveys, and administrative data. 

The opportunities for nurse researchers to provide invaluable contributions to the growing field 

of health services research are innumerable. 

12 
Health Services Research

Author Affiliations 

Donald Steinwachs, Ph.D.; Director, Health Services Research and Development Center; 

Bloomberg School of Public Health; Johns Hopkins University; Baltimore, MD. E-mail: 

dsteinwa@jhsph.edu.

Ronda G. Hughes, Ph.D., M.H.S., R.N.; Senior Health Scientist Administrator; Center for 

Primary Care, Prevention, and Clinical Partnerships; Agency for Healthcare Research and 

Quality; Rockville, MD. E-mail: Ronda.Hughes@ahrq.hhs.gov. 

References

1. Institute of Medicine. Improving information services 

for health services researchers: a report to the 

National Library of Medicine. Washington, DC: 

National Academy Press; 1991. 

2. Ginzberg E, ed. Health services research: key to 

health policy. Cambridge, MA: Harvard University 

Press; 1991. 

3. Institute of Medicine. Health services research: 

training and workforce issues. Washington, DC: 

National Academy Press; 1995. 

4. Institute of Medicine. Report on health services 

research. Washington, DC: National Academy Press; 

1979. 

5. Lohr KN, Steinwachs DM. Health services research: 

an evolving definition of the field. Health Serv Res 

2002 Feb; 37(1):7-9.

6. Altman SH, Reinhardt UE, eds. Strategic choices for 

a changing health care system. Chicago: Health 

Administration Press; 1996. 

7. Brown LD. Knowledge and power: health services 

research as a political resource. In: Ginzberg E, ed. 

Health services research: key to health policy. 

Cambridge, MA: Harvard University Press; 1991. p. 

20-45. 

8. DeFriese GH, Ricketts III TC, Stein JS, eds. 

Methodological advances in health serivces research. 

Chicago: Health Administration Press; 1989. 

9. Shi L. Health services research methods. Albany, 

NY: Delmar Publishers, Inc.; 1997. 

10. Shortell SM, Reinhardt UE, eds. Improving health 

policy and management: nine critical research issues 

for the 1990s. Chicago: Health Administration Press; 

1992. 

11. Agency for Healthcare Research and Quality. 

National health care quality report. Washington, DC: 

Department of Health and Human Services; 2006. 

12. Agency for Healthcare Research and Quality. 

National health care disparities report. Washington, 

DC: Department of Health and Human Services; 

2006. 

13. U.S. Department of Health and Human Services. 

Healthy people 2010, 2nd ed., Vol 1 & 2. 

Washington, DC: U.S. Government Printing Office; 

November 2000. 

14. Centers for Medicare and Medicaid Services. 

Medicare demonstration shows hospital quality of 

care improves with payments tied to quality. CMS 

Office of Public Affairs. Available at: 

http://www.cms.gov/apps/media/press/release.asp?Co

unter=1729. Accessed November 14, 2005. 

15. Schmid Mast M, Hall JA, Roter DL. Disentangling 

physician sex and physician communication style: 

their effects on patient satisfaction in a virtual 

medical visit. Patient Educ Couns 2007 Sep;68(1):16-

22. Epub 2007 May 4. 

16. Beach MC, Roter DL, Wang NY, et al. Are 

physicians’ attitudes of respect accurately perceived 

by patients and associated with more positive 

communication behaviors? Patient Educ Couns 2006 

Sep;62(3):347-54. 

17. Johnson RL, Roter D, Powe NR, et al. Patient 

race/ethnicity and quality of patient-physician 

communication during medical visits. Am J Public 

Health 2004 Dec;94(12):2084-90. 

18. Averill RF, Goldfield N. All patient refined diagnosis 

related groups (APR-DRGs) version 20.0: 

methodology overview. Wallingford, CT: 3M Health 

Information Systems; 2003. 

13 
Patient Safety and Quality: An Evidence-Based Handbook for Nurses

19. Emery FE, Trist EL. The causal texture of 

organizational environments. Hum Relat 1965;18:21-

31. 

20. Terreberry S. The evolution of organizational 

environments. Adm Sci Q 1967;12:590-613. 

21. Gill G. Nightingales: the extraordinary upbringing 

and curious life of Miss Florence Nightingale. New 

York: Random House; 2004. 

22. Committee on the Costs of Medical Care. The fiveyear program of the committee on the costs of 

medical care. Washington, DC: The Committee 1928. 

(Publication 1). 

23. Starr P. The social transformation of American 

medicine: the rise of a sovereign profession and the 

making of a vast industry. New York: BasicBooks; 

1982. 

24. Coalition for Health Services Research. Federal 

funding for health services research. Washington, 

DC: AcademyHealth; 2004. 

25. Committee on Quality of Health Care in America. 

Crossing the quality chasm: a new health system for 

the 21st century. Institute of Medicine National 

Academy Press; Washington, DC, 2001. 

26. Kohn LT, Corrigan JM, Donaldson MS, eds. To err is 

human: building a safer health system. A report of 

the Committee on Quality of Health Care in America, 

Institute of Medicine. Washington, DC: National 

Academy Press; 2000. 

27. Anell A, Willis M. International comparison of health 

care systems using resource proviles. Bull World 

Health Organ 2000;78(6):770-8. 

28. Davis K. International health policy: common 

problems, alternative strategies. Health Aff 

1999;18(3):135-43. 

29. Sochalski J, Aiken LH. Accounting for variation in 

hospital outcomes: a cross-national study. Health Aff 

1999;18(3):256-9. 

30. Starfield B, Shi L. Policy relevant determinants of 

health: an international perspective. Health Policy 

2002;60(3):201-18. 

31. Schoen C, Osborn R, Doty MM, et al. Toward 

higher-performance health systems: adults’ health 

care experience in seven countries. Health Aff 

2007;26(6):w717-34. 

32. Smedley BD, Stith AY, Nelson AD, eds. Unequal 

treatment: confronting racial and ethnic disparities in 

healthcare, Institute of Medicine. Washington, DC: 

National Academy Press; 2003. 

33. Donabedian A. Evaluating the quality of medical 

care. Milbank Memorial Fund Quarterly 

1966;44:166-203. 

34. Wennberg JE, Gittelsohn A. Small area variations in 

health care delivery. Science 1973;192(4117):1102-8. 

35. Wennberg JE, Gittelsohn A. Variations in medical 

care among small areas. Scientific American 

1982;246(4):120-34. 

36. Brook RH, Chassin MR, Fink A, et al. A method for 

the detailed assessment of the appropriateness of 

medical technology. Int J Technol Assess 

1986;2(1):53-63. 

37. Kahn KL, Kosecoff J, Chassin MR, et al. Measuring 

the clinical appropriateness of the use of a procedure: 

can we do it? Med Care 1989;26(4):415-22. 

38. Brook RH. Practice guidelines and practicing 

medicine: are they compatible? JAMA 

1989;262(21):3027-30. 

39. Chassin MR, Kosecoff J, Park RE, et al. Does 

inappropriate use explain geographic variations in the 

use of health services? A study of three procedures. 

JAMA 1987;258(18):2533-7. 

40. Evans RG, Barer ML, Marmor TR, eds. Why are 

some people healthy and others not? The 

determinants of health of populations. New York: 

Aldine De Gruyter; 1994. 

41. Mark BA, Salyer J, Wan TTH. Professional nursing 

practice: impact on organizational and patient 

outcomes. J Nurs Adm 2003;33:224-34. 

42. Aiken LH, Clarke SP, Sloane DM. Hospital staffing, 

organization, and quality of care: cross-national 

findings. Nurs Outlook 2002;50:187-94. 

43. Clarke SP, Sloane DM, Aiken LH. Effects of hospital 

staffing and organizational climate on needlestick 

injuries to nurses. Am J Public Health 2002;92:1115-

9. 

44. Committee on Assuring the Health of the Public in 

the 21st Century. The future of the public’s health in 

the 21st century, Institute of Medicine. Washington, 

DC: National Academy Press; 2003. 

45. Newhouse JP, Manning WG, Morris CN, et al. Some 

interim results from a controlled trial of cost sharing 

health insurance. N Engl J Med 1981;305:1501-7. 

14 
Health Services Research

46. Brook RH, Ware JE, Rogers WH, et al. Does free 

care improve adult’s health? Results from a 

randomized controlled trial. N Engl J Med 

1983;309(23):1426-34. 

47. McDowell I, Newell C. Measuring health: a guide to 

rating scales and questionnaires. New York: Oxford 

University Press; 1987. 

48. Burton LC, Steinwachs DM, German PS, et al. 

Preventive services for the elderly: would coverage 

affect utilization and costs under Medicare? Am J 

Public Health 1995;85(3):387-91. 

49. Lieberman JA, Stroup TS, McEvoy JP, et al. 

Effectiveness of antipsychotic drugs in patients with 

antipsychotic schizophrenia. N Engl J Med 

2005;353:1209-23. 

50. Stroup TS, Lieberman JA, McEvoy JP, et al. 

Effectiveness of olanzapine, risperidone, and 

piprasidone in patients with chronic schizophrenia 

following discontinuation of a previous atypical 

antipsychotic. Am J Psychiatry 2006;163:611-22. 

51. McEvoy JP, Lieberman JA, Stroup TS, et al. 

Effectiveness of clozapine versus olanzapine, 

quetiapine, and risperidone in patients with chronic 

schizophrenia who did not respond to prior atypical 

antipsychotic treatment. Am J Psychiatry 

2006;163:600-10. 

52. Armenian HK, Shapiro S. eds. Epidemiology and 

health services. New York and Oxford: Oxford 

University Press; 1989. 

53. Iezzoni LI. Risk adjustment for measuring health care 

outcomes. Ann Arbor, MI: Health Administration 

Press; 1994. 

54. Weiner JP, Starfield BH, Steinwachs DM, et al. 

Development and application of a populationoriented measure of ambulatory care case-mix. Med 

Care 1991;29(5):452-72. 

55. McDonald K, Romano P, Geppert J, et al. Measures 

of patient safety based on hospital administrative 

data—the patient safety indicators. (Prepared by the 

University of California San Francisco-Stanford 

Evidence-based Practice Center under Contract No. 

290-97-0013). Rockville, MD; Agency for 

Healthcare Research and Quality. August 2002. 

Technical review 5. AHRQ Publication No. 02-0038. 

56. Davies SM, Geppert J, McClellan M, et al. Measures 

of patient safety based on hospital administrative 

data—the patient safety indicators. (Prepared by the 

University of California San Francisco-Stanford 

Evidence-based Practice Center under Contract No. 

290-97-0013). Rockville, MD: Agency for 

Healthcare Research and Quality; May 2001. 

Technical Review 4. AHRQ Publication No. 02-

0035. 

57. Hargraves JL, Hays RD, Cleary PD. Psychometric 

properties of the Consumer Assessment of Health 

Plans Study (CAHPSTM) 2.0 adult core survey. 

Health Serv Res 2003 Dec;38(6 Pt 1):1509-27. 

58. Bergner M, Bobbitt RA, Carter WB, et al. The 

sickness impact profile: development and final 

revision of a health status measure. Med Care 

1981;19(8):787-805. 

59. Shaw JW, Johnson JA, Coons SJ. US valuation of the 

EQ-5D health states: development and testing of the 

D1 valuation model. Med Care 2005 Mar;43(3):203-

20. 

60. Ware JE, Sherbourne CD. The MOS 36-item short 

form health status survey (SF-36). I. Conceptual 

framework and item selection. Med Care

1992;30:MS253-65. 

61. McGlynn EA, Asch SM, Adams J, et al. The quality 

of health care delivered to adults in the United States. 

N Engl J Med 2003;348(26):2635-45. 

62. Cabana MD, Rand CS, Pose NR, et al. Why don’t 

physicians follow clinical practice guidelines? A 

framework for improvement. JAMA 

1999;282(15):1458-65. 

63. James B. Quality improvement opportunities in 

health care—making it easy to do it right. J Manag 

Care Pharm 2002;8(5):394-9. 

64. Ellwood PM. Shattuck Lecture—outcomes 

management: a technology of patient experience. N 

Engl J Med 1988;318:1549-56. 

65. Steinwachs DM, Wu AW, Skinner EA. How will 

outcomes management work? Health Aff 

1994;13:153-62. 

66. National Heart, Lung and Blood Institute. National 

Asthma Education Program Expert Panel Report: 

Guidelines for Diagnosis and Management of 

Asthma. Bethesda MD: National Institutes of Health; 

1991. Publication No. 91-3042. 

67. Steinwachs DM, Wu AW, Skinner EA, et al. Asthma 

outcomes in managed health care: outcomes 

management and quality improvement. Final report 

submitted to the Managed Health Care Association 

Outcomes Management Consortium. Baltimore, MD: 

Johns Hopkins University; 1996. 

15