United States Government Accountability Office

GAO Report to the Ranking Member, 

Committee on Finance, U.S. Senate 

NURSING HOMES

Federal Actions 

Needed to Improve 

Targeting and 

Evaluation of 

Assistance by Quality 

Improvement 

Organizations 

May 2007 

 

GAO-07-373 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

May 2007

NURSING HOMES

Federal Actions Needed to Improve 

Targeting and Evaluation of Assistance 

by Quality Improvement Organizations 

Highlights of GAO-07-373, a report to the 

Ranking Member, Committee on Finance, 

U.S. Senate 

www.gao.gov/cgi-bin/getrpt?GAO-07-373.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Kathryn G. 

Allen, (202) 512-7118, allenk@gao.gov. 

In 2002, CMS contracted with

Quality Improvement Organizations 

(QIO) to help nursing homes 

address quality problems such as 

pressure ulcers, a deficiency

frequently identified during routine 

inspections conducted by state 

survey agencies. CMS awarded

$117 million over a 3-year period to 

the QIOs to assist all homes and to 

work intensively with a subset of 

homes in each state. Homes’ 

participation was voluntary. To 

evaluate QIO performance, CMS 

relied largely on changes in homes’ 

quality measures (QM), data based 

on resident assessments routinely 

conducted by homes. GAO

assessed QIO activities during the 

3-year contract starting in 2002, 

focusing on (1) characteristics of 

homes assisted intensively, 

(2) types of assistance provided, 

and (3) effect of assistance on the 

quality of nursing home care. GAO 

conducted a Web-based survey of

all 51 QIOs, visited QIOs and 

homes in five states, and

interviewed experts on using QMs

to evaluate QIOs. 

What GAO Recommends

GAO recommends that the CMS 

Administrator (1) further increase 

the number of low-performing 

homes that QIOs work with 

intensively, (2) improve monitoring

and evaluation of QIO activities, 

and (3) require QIOs to share with 

CMS the identity of homes assisted 

intensively in order to facilitate 

evaluation. CMS agreed with the 

first two recommendations, but did 

not specifically indicate if it agreed

with the third.

Although more homes volunteered to work with the QIOs than CMS 

expected them to assist intensively, QIOs typically did not target their 

assistance to the low-performing homes that volunteered. Most QIOs’ 

primary consideration in selecting homes was their commitment to working 

with the QIO. CMS did not specify selection criteria for intensive 

participants but contracted with a QIO that developed guidelines 

encouraging QIOs to select committed homes and exclude those with many 

survey deficiencies or QM scores that were too good to improve 

significantly. Consistent with the guidelines, few QIOs targeted homes with a

high level of survey deficiencies, and eight QIOs explicitly excluded these 

homes. GAO’s analysis of state survey data confirmed that selected homes 

were less likely than other homes to be low-performing in terms of identified 

deficiencies. Most state survey and nursing home trade association officials 

interviewed by GAO believed QIO resources should be targeted to lowperforming homes. 

QIOs were provided flexibility both in the QMs on which they focused their 

work with nursing homes and in the interventions they used. Most QIOs 

chose to work on chronic pain and pressure ulcers, and most used the same 

interventions⎯conferences and distribution of educational materials⎯to 

assist homes statewide. The interventions used to assist individual homes 

intensively varied and included on-site visits, conferences, and small group 

meetings. Just over half the QIOs reported that they relied most on on-site 

visits to assist intensive participants. Sixty-three percent said such visits

were their most effective intervention. Of the 15 QIOs that would have 

changed the interventions used, most would make on-site visits their primary 

intervention. Homes indicated that they were less satisfied with the program 

when their QIO experienced high staff turnover or when their QIO contact 

possessed insufficient expertise. 

Shortcomings in the QMs as measures of nursing home quality and other 

factors make it difficult to measure the overall impact of the QIOs on nursing 

home quality, although staff at most of the nursing homes GAO contacted 

attributed some improvements in the quality of resident care to their work 

with the QIOs. The extent to which changes in homes’ QM scores reflect 

improvements in the quality of care is questionable, given the concerns 

raised by GAO and others about the validity of the QMs and the reliability of 

the resident assessment data used to calculate them. In addition, quality 

improvements cannot be attributed solely to the QIOs, in part because the 

homes that volunteered and were selected for intensive assistance may have

differed from other homes in ways that would affect their scores; these 

homes may also have participated in other quality improvement initiatives. 

Ongoing CMS evaluation of QIO activities for the contract that began in 

August 2005 is being hampered by a 2005 Department of Health and Human 

Services decision that QIO program regulations prohibit QIOs from

providing to CMS the identities of homes being assisted intensively. 
Contents 

Letter 1

Results in Brief 4

Background 7

QIOs Generally Had a Choice among Homes That Volunteered but 

Did Not Target Assistance to Low-Performing Homes 19 

QIO Contract Flexibility Resulted in Variation in Assistance 

Provided to Intensive Participants 30 

QIOs’ Impact on Quality Is Not Clear, but Staff at Homes We 

Contacted Attributed Some Improvements to QIOs 39 

Conclusions 48

Recommendations for Executive Action 51 

Agency Comments and Our Evaluation 51 

Appendix I Scope and Methodology 55 

 

Appendix II Publicly Reported Quality Measures 62 

Appendix III Comments from the Centers for Medicare & 

Medicaid Services 63 

Appendix IV GAO Contact and Staff Acknowledgments 67 

Related GAO Products 68 

Tables 

Table 1: Quality Measures on Which QIOs Could Focus Their 

Quality Improvement Efforts in the 7th SOW 16 

Table 2: Examples of Other Categories of Homes Stakeholders 

Suggested QIOs Should Include as Intensive Participants 28 

Table 3: Examples of Resident Care Improvements Made by Homes 

as a Result of Intensive Assistance Provided by QIOs, 7th

SOW 48 

Table 4: QMs as of November 2002 and as of February 2007 62 

Page i GAO-07-373 Quality Improvement Organizations
Figures 

Figure 1: Timeline for 7th SOW Contract and Concurrent Special 

Studies by QIOs to Improve the Quality of Nursing Home 

Care 13

Figure 2: Levels of QIO Assistance and Nursing Home Participation 

in the 7th SOW 15 

Figure 3: QIO Contract Evaluation Scoring Methodology for the 7th

SOW 17

Figure 4: QIOs’ Considerations in Choosing among Homes That 

Volunteered for Intensive Assistance in the 7th SOW 23 

Figure 5: Comparison of Nonintensive and Intensive Participants’ 

Performance on State Surveys 26 

Figure 6: QMs Selected by QIOs for Statewide Interventions and 

QMs Selected by Nursing Homes for Intensive Assistance, 

7th SOW 31 

Figure 7: Statewide Interventions Most Relied on by QIOs, 7th SOW 33 

Figure 8: Intensive Interventions Most Relied on by QIOs and 

Frequency of Interventions (Range and Median Number) 

during the 7th SOW 35 

Page ii GAO-07-373 Quality Improvement Organizations
Abbreviations 

CMS Centers for Medicare & Medicaid Services 

FTE full-time-equivalent 

HHS Department of Health and Human Services 

IOM Institute of Medicine 

MDS minimum data set 

NQF National Quality Forum 

OSCAR On-Line Survey, Certification, and Reporting system 

PARTner Program Activity Reporting Tool 

PRO Peer Review Organization 

QIO Quality Improvement Organization 

QM quality measure 

SOW statement of work 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 

Page iii GAO-07-373 Quality Improvement Organizations
United States Government Accountability Office

Washington, DC 20548 

May 29, 2007

The Honorable Charles E. Grassley 

Ranking Member 

Committee on Finance 

United States Senate 

Dear Senator Grassley: 

The federal government plays a major role in the financing and oversight 

of nursing home care for individuals who are aged or disabled. Medicare 

and Medicaid payments for nursing home services totaled $67 billion in 

2004, including a $46 billion federal share.1

 The Centers for Medicare & 

Medicaid Services (CMS) defines quality standards that the nation’s 

approximately 16,400 nursing homes must meet to participate in the 

Medicare and Medicaid programs and contracts with state survey agencies

to assess homes’ compliance through routine inspections, known as 

standard surveys, and through complaint investigations. Under 3-year 

contracts beginning in August 2002 and referred to as the 7th statement of 

work (SOW), CMS directed Medicare Quality Improvement Organizations 

(QIO) to work with nursing homes to improve the quality of care provided

to residents in the 50 states, the District of Columbia, and the territories.2

 

 1

Medicare is the federal health care program for elderly and certain disabled individuals. 

Medicare may cover up to 100 days of skilled nursing home care following a hospital stay. 

Medicaid is the joint federal-state health care financing program for certain categories of 

low-income individuals. Medicaid also pays for long-term care services, including nursing 

home care. 

2

QIOs take a variety of forms. They can be for- or not-for-profit organizations and can be 

either sponsored by a significant number of actively practicing area physicians or have

available to them a sufficient number of these physicians to assure adequate peer review. 

In general, QIOs cannot be health care facilities. Prior to 1999, QIOs focused on quality 

improvement in the hospital setting. Beginning in 1999, CMS required QIOs to also work in

an alternative setting; about two-thirds selected nursing homes. The QIOs currently also 

work with physician offices, home health agencies, rural or underserved populations, and

Medicare Advantage organizations to improve Medicare beneficiaries’ quality of care. For 

the 7th SOW, the 53 QIO contracts, one for each state, the District of Columbia, and 

2 territories (Puerto Rico and the Virgin Islands) were held by 37 organizations. We 

excluded the 2 territories from our study because of substantial differences in health care

financing between the territories and the states. 

Page 1 GAO-07-373 Quality Improvement Organizations
As a condition of their contracts, QIOs were required to provide 

(1) information to all Medicare- or Medicaid-certified nursing homes in 

each state about systems-based approaches to improving resident care and 

clinical outcomes and (2) intensive assistance to a subset of each state’s 

homes, typically 10 to 15 percent, that were selected by the QIOs from 

among those homes that volunteered for assistance. 

In a series of congressionally requested studies undertaken since 1998, we 

have reported on the unacceptably high proportion of nursing homes 

providing poor care to residents.3

 Based in part on our recommendations, 

CMS has undertaken a number of enforcement initiatives to encourage 

nursing home compliance with federal quality standards, including 

improved oversight by both state survey agencies and CMS, and tougher 

enforcement measures to ensure that homes correct deficiencies and 

maintain compliance with federal standards. For example, CMS expanded 

its Special Focus Facility program in which state agencies survey selected

homes more frequently and terminate those that fail to improve 

significantly within 18 months. 

CMS’s decision to offer direct assistance to nursing homes that volunteer 

to work with QIOs represents a new strategy in the effort to help address 

long-standing quality problems in nursing homes. To evaluate QIO 

performance in improving nursing home care, CMS relied primarily on 

changes in nursing homes’ quality measures (QM) during the contract 

period. QMs are numeric measures derived from resident assessments—

known as the minimum data set (MDS)—that nursing homes routinely 

conduct and submit to CMS.4

 The QMs were developed to permit 

comparisons across nursing homes of the quality of care provided to 

residents and have been publicly reported on CMS’s Nursing Home 

Compare Web site since 2002.5

In 2005, CMS renewed the QIO contracts, including the nursing home 

component, for another 3-year period, with a budget of $96 million to 

 3

See Related GAO Products at the end of this report. 

4

The minimum data set (MDS) consists of data that are periodically collected to assess the 

care needs of residents in order to develop an appropriate plan of care. State surveyors use 

MDS data to help assess the quality of resident care, and Medicare and some state Medicaid 

programs also use MDS data to adjust nursing home payments. 

5

The Web site can be accessed at www.Medicare.gov/NHCompare/home.asp. 

Page 2 GAO-07-373 Quality Improvement Organizations
assist nursing homes.6

 Given the decision to continue the program, you 

asked us to assess QIOs’ work with nursing homes for the 7th SOW, 

covering the period August 2002 through January 2006. For this report, we 

assessed (1) characteristics of nursing homes the QIOs assisted

intensively, (2) the assistance the QIOs provided to nursing homes, and 

(3) the effect of QIOs’ assistance on the quality of nursing home care. 

To assess the characteristics of nursing homes that the QIOs selected to

assist intensively from among the homes that volunteered, we analyzed 

CMS data on deficiencies cited in standard surveys of nursing homes and

compared the results for homes assisted intensively by the QIOs with 

homes that were not assisted intensively.7

 To gather information about the 

QIOs’ criteria for selecting homes for intensive assistance, we fielded a 

Web-based survey to the 37 organizations that held the 51 QIO contracts in

the states and the District of Columbia, achieving a 100 percent response 

rate.8

 To determine the type of quality improvement assistance QIOs 

provided to nursing homes, our Web-based survey collected data on the 

types, frequency, and perceived effectiveness of specific interventions 

used to assist homes both statewide and in the group assisted intensively; 

interventions included activities such as on-site visits, mailings, and 

conferences. To gather more detailed information about QIOs’ work with

nursing homes, we conducted site visits to five states—Colorado, Florida,

Iowa, Maine, and New York—where we interviewed QIO personnel, staff 

from nursing homes that had received intensive assistance, and key 

 6

The QIO contract is divided into tasks and subtasks; the nursing home component is

subtask 1a. The amount budgeted for this component in the 8th SOW (the QIO contract

covering the period from 2005 through 2008) was approximately $10 million less than was 

budgeted in the 7th SOW. 

7

We ranked nursing homes as high-, moderately, or low-performing on the basis of the 

number, scope, and severity of the deficiencies for which they were cited (relative to other 

homes in their state) in three standard state surveys from 1999 through 2002. We based our 

classification of homes on their performance level relative to other homes in the state to 

take into account the inconsistency in how states conduct surveys, a problem we have

reported on since 1998. A limitation of our analysis is that we did not have information

about all of the homes that volunteered for intensive assistance, only those that were

selected by the QIOs, and therefore did not know the extent to which low-performing

homes volunteered for intensive assistance. 

8

Because a QIO is responsible for quality improvement activities in each state and the

District of Columbia, we refer to the 51 QIOs throughout this report. 

Page 3 GAO-07-373 Quality Improvement Organizations
stakeholders.

9

 The five states accounted for 15 percent of nursing home 

beds nationwide in 2002 and represented a range in terms of such

characteristics as number of nursing home beds, region of the country, 

and QIOs’ performance on the nursing home component in the 7th SOW. In

the five states, we interviewed staff from 28 nursing homes⎯4 to 8 per 

state; in addition, we interviewed staff from 4 homes in four other states 

for a total of 32 homes. We sought to select a group of homes that 

represented a range in terms of state survey deficiencies, improvement in 

QM scores during the 7th SOW, distance from the QIO, and urban versus 

rural location. However, the experiences of the 32 homes in our sample 

cannot be generalized to all homes that received intensive assistance from 

the QIOs nationwide. To assess the effect of QIOs’ assistance on nursing 

home quality, we reviewed performance requirements in the QIO contracts 

for both the 7th and the 8

th SOWs; reports on QIOs’ work with nursing 

homes, including the 2006 report on the QIO program by the Institute of 

Medicine (IOM);10 and other documents. We also conducted interviews 

with nursing homes, CMS officials, officials from state quality assurance 

programs and state MDS accuracy review programs, and experts on the 

nursing home QMs and the MDS data on which they are based. We 

conducted our review from October 2005 through May 2007 in accordance 

with generally accepted government auditing standards. (For a more 

detailed description of our scope and methodology, see app. I.) 

Although QIOs generally had a choice of homes to select for intensive 

assistance because more homes volunteered than CMS expected QIOs to 

assist, QIOs typically did not target the low-performing homes that 

volunteered. Most QIOs reported in our Web-based survey that their 

Results in Brief 

 9

To assist in the development of our site visit interview protocols, we also interviewed 

personnel from three other QIOs. On each of our five site visits, we interviewed officials 

from three stakeholder groups: (1) the state survey agency; (2) the local affiliate for the

American Health Care Association, which generally represents for-profit homes; and (3) the 

local affiliate for the American Association of Homes and Services for the Aging, which 

represents not-for-profit homes. 

10IOM of The National Academies, Committee on Redesigning Health Insurance 

Performance Measures, Payment, and Performance Improvement Programs, Board on

Health Care Services, Medicare’s Quality Improvement Organization Program: 

Maximizing Potential (Washington, D.C.: The National Academies Press, 2006). The 

Medicare Prescription Drug, Improvement, and Modernization Act of 2003, Pub. L. No. 108-

173, §109(d), 117 Stat. 2066, 2173-74, directed the Secretary of Health and Human Services

to ask the IOM to conduct an evaluation of the QIO program administered by CMS. In 2006,

the IOM issued a report that examined performance within the entire QIO program,

including the nursing home component, during the 7th SOW.

Page 4 GAO-07-373 Quality Improvement Organizations
primary consideration in selecting homes was their commitment to 

working with the QIO. CMS did not specify selection criteria for intensive 

participants but contracted with a QIO to develop guidelines, which 

encouraged QIOs to select homes that appeared committed to quality

improvement and to exclude homes with a high number of survey 

deficiencies, high management turnover, or QM scores that were too good

to improve significantly. Consistent with the guidelines, only 2 percent of

the QIOs that responded to our survey cited a high level of survey 

deficiencies among their top three considerations in choosing among 

homes that volunteered for assistance, and eight QIOs explicitly excluded

such homes. QIOs reasoned that these homes might be more focused on 

improving their survey results than on committing time and resources to 

quality improvement projects that might target other care areas. Our 

analysis of state survey data showed that, nationwide, intensive 

participants were less likely to be low-performing than other homes in 

their state in terms of the number, scope, and severity of deficiencies for 

which they were cited in standard surveys from 1999 through 2002. This 

result may reflect the nature of the homes that volunteered for assistance, 

the QIOs’ selection criteria, or a combination of the two. Most of the 

stakeholders we interviewed who expressed an opinion said that QIOs’ 

resources should be targeted to low-performing homes. CMS has directed 

a small share of QIO resources to low-performing homes in the current 8th

SOW. Specifically, each QIO is required to provide intensive assistance to 

up to three “persistently poor-performing homes” identified in

consultation with the state survey agency. 

The 7th SOW contracts allowed QIOs flexibility in the QMs they focused on 

and the interventions they used. While the majority of QIOs selected the 

same QMs and most used the same interventions to assist homes 

statewide, the interventions used to assist intensive participants and 

staffing to accomplish program goals varied. Of eight possible QMs, most 

QIOs and intensive participants worked on chronic pain and pressure 

ulcers.11 While intensive participants were supposed to have a choice of

QMs to focus on, some intensive participants told us that the QIO made 

the selection and that chronic pain and pressure ulcers were not 

necessarily their greatest quality-of-care challenges. The interventions 

 11A pressure ulcer is an area of damaged skin and tissue that results from constant pressure 

due to an individual’s impaired mobility. The pressure results in reduced blood flow and 

eventually causes cell death, skin breakdown, and the development of an open wound. 

Pressure ulcers can occur in individuals who are bed- or wheelchair-bound, sometimes

after only a few hours. 

Page 5 GAO-07-373 Quality Improvement Organizations
QIOs relied on most for homes statewide were conferences and the 

distribution of educational materials; for intensive participants, they relied 

most on on-site visits, conferences, and small group meetings.12 Although 

the interventions QIOs used with intensive participants varied, most QIOs 

(63 percent) considered on-site visits the most effective, and some would 

make on-site visits their primary intervention if they had the opportunity 

to change the interventions they used during the 7th SOW. Insufficient 

experience or expertise and high turnover among QIO personnel 

negatively affected homes’ satisfaction with the program and the extent of

their quality improvements. Turnover was particularly high at 24 of the 51 

QIOs, where one-quarter or more of the QIO personnel who assisted 

nursing homes worked less than half of the 36-month contract. One 

intensive participant home had four QIO principal contacts over the 

course of the 3-year contract. 

The impact of QIOs on the quality of nursing home care cannot be 

determined from available data, but at most nursing homes we contacted,

staff attributed some improvements in the quality of resident care to their

work with QIOs. Nursing homes’ QM scores generally improved enough 

for all of the QIOs to meet—and some to surpass widely⎯the modest 

targets set by CMS for improvement among homes both statewide and in 

the group assisted intensively. However, the overall impact of the QIOs on 

the quality of nursing home care cannot be determined from these data 

because of the shortcomings of the QMs as measures of nursing home 

quality and because confounding factors⎯including homes’ participation 

in other quality improvement efforts and any preexisting differences 

between homes that volunteered and were selected for intensive 

assistance and other homes⎯make it difficult to attribute quality 

improvements solely to the QIOs. Multiple long-term care professionals we 

interviewed stated that QMs should not be used in isolation to measure 

quality improvement, but combined with other indicators, such as state 

survey data. In addition, the effectiveness of the individual interventions 

QIOs used to assist homes cannot be evaluated with the limited data CMS 

collected from the QIOs. CMS planned to enhance evaluation of the 

program during the 8th SOW, but a determination by the Department of 

Health and Human Services (HHS) Office of General Counsel that the QIO 

program regulations prohibit QIOs from providing to CMS the identities of 

 12In our survey of the QIOs, we asked them to identify the interventions they relied on most 

and the interventions that were most effective in improving the quality of nursing home 

care; we allowed the QIOs to define these terms. 

Page 6 GAO-07-373 Quality Improvement Organizations
the homes they are assisting has hampered the agency’s efforts to collect 

the necessary data. Although we cannot determine the overall impact of 

the QIOs on the quality of nursing home care, over two-thirds of the 32

nursing homes we interviewed attributed some improvements in care to 

their work with the QIOs. 

We are recommending that the CMS Administrator increase the extent to

which QIOs target intensive assistance to low-performing homes and also 

direct QIOs to focus intensive assistance on the quality-of-care areas on 

which homes most need improvement. We are also recommending that the

CMS Administrator improve monitoring and evaluation of the QIO 

program by revising program regulations to require QIOs to provide to 

CMS the identities of the nursing homes they are assisting, collecting more 

complete and detailed data on QIO interventions, and identifying a broader 

spectrum of measures than QMs to evaluate changes in nursing home 

quality. In commenting on a draft of this report, CMS concurred with but 

did not indicate how it would implement our recommendations to increase

the number of homes that QIOs assist intensively and collect more 

complete and detailed data on the interventions QIOs use to assist homes. 

CMS did not specifically indicate if it agreed with our recommendation to 

revise program regulations to allow QIOs to reveal to CMS the identity of 

the nursing homes they are assisting, but did indicate that it continues to 

explore options which would allow access to such data in order to 

facilitate evaluation. CMS did not comment on the remaining two 

recommendations. 

Beginning in the late 1990s, CMS took steps to broaden the mechanisms in 

place intended to help ensure that nursing home residents receive quality

care. To augment the periodic assessment of homes’ compliance with 

federal quality requirements, CMS contracted for the development of QMs 

and tasked QIOs with providing assistance to homes to improve quality. 

CMS used QMs both to provide the public with information on nursing 

home quality of care and to help evaluate QIO efforts to address quality-ofcare issues, such as pressure ulcers. During the 7th SOW, organizations 

other than QIOs were also working with nursing homes to improve quality.

Two indicators used by CMS to assess the quality of care that nursing 

homes provide to residents are (1) deficiencies identified during standard 

surveys and complaint investigations and (2) QMs. Both indicators are 

publicly reported on CMS’s Nursing Home Compare Web site. 

Background 

Indicators of Nursing 

Home Quality 

Page 7 GAO-07-373 Quality Improvement Organizations
Under contract with CMS, state agencies conduct standard surveys to 

determine whether the care and services provided by nursing homes meet 

the assessed needs of residents and whether nursing homes are in 

compliance with federal quality standards.13 These standards include 

preventing avoidable pressure ulcers; avoiding unnecessary restraints, 

either physical or chemical; and averting a decline in a resident’s ability to

perform activities of daily living, such as toileting or walking.14 During a 

standard survey, a team that includes registered nurses spends several 

days at a home reviewing the quality of care provided to a sample of 

residents. States are also required to investigate complaints filed against 

nursing homes by residents, families, and others. Complaint investigations 

are less comprehensive than standard surveys because they generally 

target specific allegations raised by the complainants. 

Survey Deficiencies 

Any deficiencies identified during standard surveys or complaint 

investigations are classified according to the number of residents 

potentially or actually affected (isolated, pattern, or widespread) and their 

severity (potential for minimal harm, potential for more than minimal 

harm, actual harm, or immediate jeopardy). Deficiencies cited at the actual 

harm and immediate jeopardy level are considered serious and could 

trigger enforcement actions such as civil money penalties. We have

previously reported on the considerable interstate variation in the 

proportion of homes cited for serious care problems, which ranged during 

fiscal year 2005 from 4 percent of Florida’s 691 homes to 44 percent of 

Connecticut’s 247 homes.15 We reported that such variability suggests 

inconsistency in states’ interpretation and application of federal 

 13CMS’s Survey and Certification Group is responsible for oversight of state survey agency 

activities. 

14Surveys must be conducted at each home on average once every 12 months but no less 

than once every 15 months. 

15This analysis excluded 13 states because fewer than 100 homes were surveyed, and even a 

small increase or decrease in the number of homes with serious deficiencies in such states 

could produce a relatively large percentage-point change. In fiscal year 2005, about 

17 percent of the 16,337 homes surveyed had serious deficiencies. See GAO, Nursing 

Homes: Efforts to Strengthen Federal Enforcement Have Not Deterred Some Homes from 

Repeatedly Harming Residents, GAO-07-241 (Washington, D.C.: Mar. 26, 2007). 

Page 8 GAO-07-373 Quality Improvement Organizations
regulations; in addition, both we and CMS have found that state surveyors

do not identify all serious deficiencies.

16 

QMs are relatively new indicators of nursing home quality. Although 

survey deficiencies have been publicly reported since 1998, CMS did not 

begin posting QMs on its Nursing Home Compare Web site until November 

2002. QMs are derived from resident assessments known as the MDS that 

nursing homes routinely collect on all residents at specified intervals.17

Conducted by nursing home staff, MDS assessments cover 17 areas, such 

as skin conditions, pain, and physical functioning.

Quality Measures 

In developing QMs, CMS recognized that any publicly reported indicators 

must pass a rigorous standard for validity and reliability. In October 2002, 

we reported that national implementation of QMs was premature because

of validity and reliability concerns.18 Valid QMs would distinguish between 

good and poor care provided by nursing homes; reliable QMs would do so 

consistently. One of our main concerns about publicly reporting QMs was 

that the QM scores might be influenced by other factors, such as residents’ 

health status. As a result, the specification of appropriate risk adjustment 

was a key requirement for the validity of any QMs. Risk adjustment is 

important because it provides consumers with an “apples-to-apples” 

comparison of nursing homes by taking into consideration the 

characteristics of individual residents and adjusting the QM scores 

accordingly. For example, a home with a disproportionate number of 

residents who are bedfast or who present a challenge for maintaining an

adequate level of nutrition—factors that contribute to the development of 

pressure ulcers—may have a higher pressure ulcer score. Adjusting a 

home’s QM score to fairly represent to what extent a home does or does 

not admit such residents is important for consumers who wish to compare 

one home to another. Appendix II lists the 10 QMs initially adopted and 

publicly reported by CMS—6 applicable to residents with chronic care 

 16CMS is evaluating a new survey methodology to help ensure that surveyors do not miss 

serious care problems. National implementation will depend on the outcome of the 

evaluation. 

17MDS assessments are conducted for all nursing home residents within 14 days of 

admission and at quarterly and yearly intervals unless there is a significant change in 

condition. In addition, Medicare beneficiaries in a Medicare-covered stay are assessed 

through MDS on or before the 5th, 14th, 30th, 60th

, and 90th day of their stays to determine if 

their Medicare coverage should continue. 

18GAO, Nursing Homes: Public Reporting of Quality Indicators Has Merit, but National

Implementation Is Premature, GAO-03-187 (Washington, D.C.: Oct. 31, 2002). 

Page 9 GAO-07-373 Quality Improvement Organizations
problems (long-stay residents) and 4 applicable to residents with postacute-care needs (short-stay residents). 

MDS data are self-reported by nursing homes, and ensuring their accuracy

is critical for establishing resident care plans, setting nursing home 

payments, and publicly reporting QM scores. In February 2002, we 

concluded that CMS efforts to ensure the accuracy of MDS data, which are 

used to calculate the QMs, were inadequate because the agency relied too 

much on off-site review activities by its contractor and planned to conduct

on-site reviews in only 10 percent of its data accuracy assessments, 

representing fewer than 200 of the nation’s then approximately 17,000

nursing homes.19 Although we recommended that CMS reorient its review 

program to complement ongoing state MDS accuracy efforts as a more 

effective and efficient way to ensure MDS data accuracy, CMS disagreed 

and continued to emphasize off-site reviews.20

Evolution of the QIO 

Program and the Quality 

Improvement Process 

Over the past 24 years, the QIO program has evolved from a focus on 

quality assurance in the acute care setting to quality improvement in a 

broader mix of settings, including physician offices, home health agencies, 

and nursing homes. Established by the Peer Review Improvement Act of 

198221 and originally known as Peer Review Organizations (PRO), QIOs 

initially focused on ensuring minimum standards by conducting 

retrospective hospital-based utilization reviews that looked for 

inappropriate or unnecessary Medicare services. According to the 2006 

IOM report, as it became apparent that standards of care themselves 

required attention, QIOs gradually shifted from retrospective case reviews 

to collaboration with providers to improve the overall delivery of care—a 

shift consistent with transformational goals set by CMS’s Office of Clinical 

Standards and Quality, which oversees the QIO program.22

 19See GAO-02-279. 

20Some states that adjust nursing home payments to account for variation in resident care 

needs have their own separate MDS review programs. 

21Pub. L. No. 97-248, §141-50, 96 Stat. 381-95. PROs were renamed QIOs in 2002. Under the

provisions of the Peer Review Improvement Act of 1982 and implementing regulations, a

QIO can be either a physician-sponsored entity or a physician-access entity. See 42 C.F.R.

§475.101 (2005). QIOs are allowed to be either for- or not-for-profit entities and are required

to include at least one consumer representative on the QIO governing board. Funding for

QIO activities comes from the Medicare Trust Funds. 

22IOM, Medicare’s Quality Improvement Organization Program: Maximizing Potential. 

Page 10 GAO-07-373 Quality Improvement Organizations
In contrast to enforcing standards, quality improvement tries to ensure 

that organizations have effective processes for continually measuring and 

improving quality. The goal of quality improvement is to close the gap 

between an organization’s current performance and its ideal performance, 

which is defined by either evidence-based research or best practices 

demonstrated in high-performing organizations. According to the quality

improvement literature, successful quality improvement requires a 

commitment on the part of an organization’s leadership and active 

involvement of the staff. The 2006 IOM report notes that QIOs rely on 

various mechanisms to promote quality improvement, including one-onone consulting and collaboratives.23 While the former provides direct and 

specialized attention, the latter relies on workshops or meetings that offer 

opportunities for providers to share experiences and best practices. 

Quality improvement often relies on the involvement of early adopters of

best practices—providers who are highly regarded as leaders and can help 

convince others to change—for the diffusion of best practices. Key tools

for quality improvement include (1) root cause analysis, a technique used 

to identify the conditions that lead to an undesired outcome; 

(2) instruction on how to collect, aggregate, and interpret data; and 

(3) guidance on bringing about, sustaining, and diffusing internal system 

redesign and process changes, particularly those related to use of 

information technology for quality improvement. Quality improvement 

experts also emphasize the importance of protecting the confidentiality of

provider information, not only to protect the privacy of personal health

information but also to encourage providers to evaluate their peers 

honestly and to prevent the damage to providers’ reputations that might 

occur through the release of erroneous information. 

Section 1160 of the Social Security Act provides that information collected

by QIOs during the performance of their contract with CMS must be kept 

confidential and may not be disclosed except in specific instances; it 

provides the Secretary of HHS with some discretion to determine 

instances under which QIO information may be disclosed. The regulations 

implementing the statute limit the circumstances under which confidential

information obtained during QIO quality review studies, including the 

identities of the participants of those studies, may be disclosed by the QIO.

 23IOM defines collaboratives as interventions designed to bring together stakeholders 

working toward quality improvement for the same clinical topic. Participants usually 

follow the same processes to reach goals and interact on a regular basis to share 

knowledge, experiences, and best practices.

Page 11 GAO-07-373 Quality Improvement Organizations
During the 7th SOW, QIOs submitted a list of nursing home participants to 

CMS as a contract deliverable. 

CMS Contract Funding and 

Requirements 

During the 7th SOW, CMS awarded a total of $117 million to QIOs to 

improve the quality of care in nursing homes in all 50 states, the District of 

Columbia, and the territories. The performance-based contracts for QIO 

assistance to nursing homes delineated broad expectations regarding QIO 

assistance to nursing homes, provided deadlines for completing four 

contract deliverables, and laid out criteria for evaluating QIO 

performance.24 For contracting purposes, the QIOs were divided into three 

groups with staggered contract cycles. The four contract deliverables, 

however, were all due on the same dates, irrespective of the different 

contract cycles. The contracts also required QIOs to work with a QIO 

support contractor tasked to provide guidelines for recruiting and 

selecting nursing homes as intensive participants, train QIOs in standard 

models of quality improvement assistance, and provide tools and 

educational materials, as well as individualized consultation if needed, to 

help QIOs meet contractual requirements.25 QIOs and nursing homes were 

also involved in other quality improvement special studies with budgets 

separate from the QIO contracts for the 7th SOW. These studies varied 

greatly in terms of length, the clinical issue(s) covered, the number of 

QIOs involved, and the characteristics of the nursing homes that 

participated. Figure 1 shows the 7th SOW contract cycles, deliverables for 

the nursing home component, and the duration of the special studies. 

 24According to a CMS official, all QIO contracts prior to the 6th SOW, which began in 2000,

were considered “cost plus fixed fee” and there were no deliverables, or set targets, that 

QIOs had to meet in order to obtain payment. In the late 1990s, however, the Office of

Management and Budget instructed CMS to make QIO contracts performance-based with 

deliverables and objectives that QIOs had to meet during the contract cycle. In response,

CMS changed the QIO contract so that part of QIOs’ fee was based on their performance. 

25The Rhode Island QIO was awarded the support contract for nursing homes for the 7th

SOW. The contract defined roles for the QIO support contractor, including (1) providing 

QIOs with information on clinical topics and management systems’ approaches and

techniques for quality improvement; (2) facilitating coordination and communication 

between QIOs; (3) maintaining a nursing home informational clearinghouse Web site with 

best practices, tools, and interventions; and (4) being available for ongoing technical 

assistance. 

Page 12 GAO-07-373 Quality Improvement Organizations
Figure 1: Timeline for 7th SOW Contract and Concurrent Special Studies by QIOs to Improve the Quality of Nursing Home Care 

Source: GAO analysis of the 7th SOW and CMS descriptions of special studies.

 States Nursing home

No. Special study Objectives covered participants

 1 Depression in Improve the screening and treatment of depression in skilled nursing and long-term 1 14

 Nursing Homes care facilities

 2 Corporate Nursing Home Combine clinical expertise of 161 nursing homes with QIO expertise in rapid-cycle, group-based improvement 49 1,624

 Improvement Collaborative to achieve significant advancement in pain management across eight national nursing home corporations

 3 Achieving Nursing Identify achievable targets for pressure ulcer and restraint rates in nursing homes based on analysis 51 N/A

 Home Targets of QM data from 14,034 nursing homes, nationally

 4 Innovative Examine models of nursing home quality improvement and culture change among 50 188

 Approaches innovative homes

 5 National Nursing Home Identify improvement strategies and specific interventions that multifacility nursing home 39 52

 Improvement Collaborative corporations can implement to reduce prevalence of pressure ulcers in their residents

 6 Collaborative Focus Facilitate partnerships between state survey agencies and QIOs to identify whether QIOs 17 40

 Facility Project could effectively help poor performing nursing homes improve their clinical quality

 7 Improving Nursing Home Provide strategies to move homes from an institutionalized culture to an individualized culture of 30 254

 Culture: Person-Centered Care care to improve quality of life for residents, families, and staff, and increase workforce retention

 8 Inter-setting Protection Investigate ways to improve the management of skin integrity for residents transferring between nursing 1 10

 of Skin Integrity home and hospital settings by standardizing practices and enhancing communication of resident information

b

1/02 1/03 1/04 1/05 1/06

No. 4

No. 7

No. 8

No. 6

No. 2

No. 5

 No. 1

No. 3

QIO deliverable: Plan to partner with stakeholders (8/02)

QIO deliverable: List of 3 to 5 QMs on which to be evaluated (12/02)

QIO deliverable: List of intensive participants (2/03)

CMS action: Calculate baseline QM scores (6/02)

CMS action: Remeasure QM scores to evaluate improvement (6/04)

7th SOW

key dates >>

Contract cyclesa >>

Special

studies >>

QIO deliverable: List of QMs selected by intensive participants (6/03)

1 2

3

To 6/06

a

In the 7th SOW, QIOs were divided into three groups with staggered contract cycles. The four

contract deliverables, however, were all due on the same dates, irrespective of contract cycle. 

b

The term states includes the 50 United States and the District of Columbia. 

c

QIOs could add—but not delete or change—QMs for their intensive participants through September 

2003. 

Page 13 GAO-07-373 Quality Improvement Organizations
Contract funding. The $117 million awarded to QIOs to improve the 

quality of care in nursing homes during the 7th SOW included 

(1) $106 million awarded to provide statewide and intensive assistance to 

homes,26 (2) $5.6 million awarded to selected QIOs to conduct eight special 

studies focused on nursing home care, and (3) $5.3 million awarded to the 

QIO that served as the support contractor for the nursing home 

component.27 CMS allocated a specific amount for each component of the 

contracts, but allowed QIOs to move funds among certain components.28

Just over half of the 51 QIOs did not spend all of the funds allocated to the 

nursing home component, but on average the QIOs overspent the budget

for the nursing home work by 3 percent. 

Contract requirements for quality improvement activities. Per the 

contracts for the 7th SOW, QIOs were required to provide (1) all Medicareand Medicaid-certified homes with information about systems-based 

approaches to improving patient care and clinical outcomes, and 

(2) intensive assistance to a subset of homes in each state. The contracts 

directed QIOs working in states with 100 or more nursing homes to target 

10 to 15 percent of the homes for intensive assistance.29 Figure 2 illustrates 

that QIOs provided two levels of assistance—statewide and intensive—

and that homes’ participation was either nonintensive or intensive. 

Intensive participants received both statewide and intensive assistance. 

Selection of intensive participants from among the nursing homes that 

volunteered was at the discretion of each QIO, but the SOW required the 

QIO support contractor (the Rhode Island QIO) to provide guidelines and 

criteria for QIOs to use in determining which homes to select. 

Participation in the program was voluntary, and QIOs were prohibited 

 26The $106 million represented 13 percent of the total amount ($809 million) awarded to

QIOs for their base contracts. CMS did not budget separately for statewide and intensive 

assistance. 

27The QIO support contractor subcontracted with another QIO to provide data analysis. 

28For example, QIOs could move funds between the nursing home component and the

other components under task 1, which covered clinical quality improvement efforts with 

home health agencies, hospitals, physician offices, underserved and rural beneficiaries, and

Medicare Advantage organizations. 

29QIOs working in the 13 states with fewer than 100 nursing homes were expected to target 

at least 10 homes. 

Page 14 GAO-07-373 Quality Improvement Organizations
from releasing the names of participating nursing homes except as 

permitted by statute and regulation.30

Figure 2: Levels of QIO Assistance and Nursing Home Participation in the 7th SOW 

Source: GAO analysis of the 7th SOW.

Higher intensity assistance

Lower intensity assistance

Intensive QIO assistance

Assistance provided to nursing 

homes that volunteered and 

were selected by the QIO

Intensive participantsa

Nursing homes that volunteered and were 

selected to receive intensive assistance 

(generally 10-15 percent of homes)

Statewide QIO assistance

Assistance provided to all nursing 

homes in the state

Nonintensive participants

Nursing homes not selected to 

receive intensive assistance 

(generally 85-90 percent of homes)

All nursing homes in the state

a

Nursing homes on the official list of intensive participants submitted to CMS by the QIOs by 

February 3, 2003. 

Under the contracts, the quality improvement assistance provided by QIOs 

focused on areas related to eight chronic care and post-acute-care QMs 

publicly reported on the CMS Nursing Home Compare Web site. QIOs 

were required to consult with relevant stakeholders and select from three 

to five of the eight QMs on which QIOs’ quality improvement efforts would 

be evaluated (see table 1).

31 Intensive participant homes were also required

 30See Social Security Act §1160; 42 C.F.R. §480.140 (2005). 

31Stakeholders may include representatives of nursing homes, trade associations, 

ombudsmen, state survey agencies, medical directors, directors of nursing, geriatric 

nursing assistants, other licensed professionals, academicians, and consumers.

Page 15 GAO-07-373 Quality Improvement Organizations
to select one or more QMs on which to work with the QIO. Although they 

could select one QM, they were encouraged to select more than one. 

Table 1: Quality Measures on Which QIOs Could Focus Their Quality Improvement 

Efforts in the 7th SOW 

Chronic care QMs Post-acute-care QMs 

Decline in activities of daily living Failure to improve and manage delirium 

Pressure ulcers Inadequate pain management 

Inadequate pain management Improvement in walking 

Physical restraints 

Infections

Source: CMS. 

Note: Although CMS adopted 10 QMs, the QIOs were evaluated only on the 8 listed here (see 

app. II). 

To improve QM scores, QIOs were expected to develop and implement 

quality improvement projects focused on care processes known to 

improve patient outcomes in a manner that utilized resources efficiently 

and reduced burdens on providers. The QIO support contractor developed 

a model for QIOs to facilitate systems change in nursing homes. This 

model emphasized the importance of QIOs’ statewide activities to form 

and maintain partnerships, conduct workshops and seminars, and 

disseminate information on interventions to improve quality. For intensive 

participants, the model emphasized conducting one-on-one quality 

improvement assistance as well as conferences and small group meetings. 

According to contract language, QIOs were expected to coordinate their 

projects with other stakeholders that were working on similar 

improvement efforts or were interested in teaming with the QIO. But 

ultimately, each QIO determined for itself the type, level, duration, and 

intensity of support it would offer to nursing homes. 

Evaluation of QIO contract performance. CMS evaluated QIOs’ 

performance on the nursing home component of the contract using 

nursing home provider satisfaction with the QIO, QM improvement among 

intensive participants, and QM improvement statewide (see fig. 3).32

Nursing home provider satisfaction was assessed by surveying all 

 32Under the 8th SOW contract, QIOs will not be held accountable for QM improvement 

statewide. 

Page 16 GAO-07-373 Quality Improvement Organizations
intensive participants and a sample of nonintensive participants around 

the 28th month of each 36-month contract. CMS expected at least 80 

percent of respondents to report that they were either satisfied or very 

satisfied. 

Figure 3: QIO Contract Evaluation Scoring Methodology for the 7th SOW

Source: GAO analysis of the 7th SOW.

Step 1: Survey nursing homes.

Step 2: Determine the percentage of nursing 

homes that were “satisfied” or “very satisfied” 

with their interactions with the QIOs.

Step 3: Divide the percentage by the 80% 

expected satisfaction level.

Step 4: Weight the score.

Step 1: For each intensive participant, 

exclude the QM that improved least.

Step 2: Calculate average improvement in 

the remaining QMs for each participant.

Step 3: Calculate overall average for all 

intensive participants in the state.

Step 4: Divide the result by the 8% 

expected improvement.

Step 5: Weight the score.

Step 1: For each QM, calculate statewide 

average scores before and after assistance.

Step 2: Use these averages to calculate 

average improvement in each QM.

Step 3: Calculate overall statewide average 

improvement for all QMs.

Step 4: Divide the result by the 8% 

expected improvement.

Step 5: Weight the score.

Nursing home satisfaction scorea

Weighted = 20% of total score

Intensive participant score

Weighted = 44% to 66% of total score

Statewide participant score

+ + Weighted = 14% to 36% of total score

Total b c = score

Note: QM improvement was calculated using the following formula: (baseline rate minus 

remeasurement rate) / baseline rate. For example, if a nursing home had a baseline rate of 

20 percent for the pain management QM (e.g., 20 percent of the home’s residents had severe or 

moderate pain), a 10 percent improvement would mean that 18 percent of residents had moderate or 

severe pain at remeasurement [(20 percent – 18 percent) / 20 percent]. 

a

All intensive participants and a sample of nonintensive participants were surveyed to assess their 

satisfaction with the QIO. 

b

The weight (percentage of total score) given to this element depended on the proportion of the 

state’s homes that were included in the intensive participant group; the weight ranged from 

44 percent, if 10 percent of the homes were included, to 66 percent, if at least 15 percent of the 

homes were included. 

c

The weight (percentage of total score) given to this element was the difference between 80 percent 

and the weight given to the intensive participant element and ranged from 14 to 36 percent. 

QIOs were also expected to achieve an 8 percent improvement in QM 

scores among both intensive participants and homes statewide. The term 

improvement was defined mathematically to mean the relative change in 

the QM score from when it was measured at baseline to when it was 

remeasured. The statewide improvement score included the QM 

improvement scores for intensive participants averaged with those of 

nonintensive participants. 

Page 17 GAO-07-373 Quality Improvement Organizations
CMS established two scoring thresholds for the contracts that 

encompassed scores from all components of the SOW. If a QIO scored 

above the first threshold it was eligible for a noncompetitive contract 

renewal; if it scored below that threshold, it was eligible for a competitive 

renewal only upon providing information pertinent to its performance to a 

CMS-wide panel that decided whether to allow the QIO to bid again for 

another QIO contract.33

CMS contract monitoring. CMS formally evaluated each QIO at months 

9 and 18 of the 7th SOW. If CMS found that a QIO failed to meet contract 

deliverables or appeared to be in danger of failing to meet contract goals, 

it could require the QIO to make a performance improvement plan or 

corrective plan of action to address any barriers to the QIOs successfully 

fulfilling contract requirements. In addition, CMS reviewed materials such 

as QIOs internal quality control plans, which were intended to help QIOs 

monitor their own progress and to document any project changes made to 

improve their performance. 

Other Nursing Home 

Quality Improvement and 

Assurance Initiatives 

The QIO program operated in the context of other quality improvement 

initiatives sponsored by federal and state governments and nursing home 

trade associations. As stated earlier, CMS funded a number of special 

nursing home studies involving subsets of the QIOs and nursing homes, 

which addressed a variety of clinical quality-of-care issues and which are 

summarized in figure 1. Under CMS’s Special Focus Facility program, state 

survey agencies were required to conduct enhanced monitoring of nursing 

homes with histories of providing poor care. During the 7th SOW, CMS 

revised the method for selecting homes for the Special Focus Facility 

program to ensure that the homes performing most poorly were included;

increased the minimum number of homes that must be included, from a 

minimum of two per state to a minimum of up to six, depending on the 

number of homes in the state; and strengthened enforcement for those 

nursing homes with an ongoing pattern of substandard care.34 In addition, 

 33QIOs could fail to meet contract expectations for up to 2 of the 12 components and still 

remain eligible for noncompetitive renewal of their contracts.

34Initiated in January 1999, the Special Focus Facility program was expanded by CMS in 

December 2004. Expansion strengthened enforcement authority so that if homes in the 

program fail to significantly improve performance from one survey to the next, immediate 

sanctions must be imposed; if homes show no significant improvement in 18 months and 

three surveys, they must be terminated from participation in the Medicare and Medicaid

programs. 

Page 18 GAO-07-373 Quality Improvement Organizations
concurrent with the 7th SOW, at least eight states had programs that 

provided quality assurance and technical assistance to nursing homes in 

their states.35 These programs varied in terms of whether they were 

voluntary or mandatory, which homes received assistance, the focus and 

frequency of the assistance provided, and the number and type of staff 

employed. 

In addition to government-operated quality improvement initiatives, three 

long-term care professional associations joined together in July 2002 to 

implement the Quality First Initiative.

36 This initiative was based on a 

publicly articulated pledge on the part of the long-term care profession to 

establish an environment of continuous quality improvement, openness, 

and leadership in participating homes. 

Although QIOs generally had a choice of homes to select for intensive 

assistance because more homes volunteered than CMS expected QIOs to 

assist, QIOs typically did not target the low-performing homes that 

volunteered. Most QIOs reported in our Web-based survey that they did 

not have difficulty recruiting homes, and their primary consideration in 

selecting homes from the pool of volunteers was that the homes be 

committed to working with the QIOs. In the 7th SOW, CMS did not specify 

recruitment and selection criteria for intensive participants, leaving the 

development of guidelines to the QIO support contractor, which 

encouraged QIOs to select homes that seemed committed to quality 

improvement and to exclude homes with a high number of survey 

deficiencies, high management turnover, or QM scores that were too good

to improve significantly.37 Our analysis of state survey data showed that, 

QIOs Generally Had a 

Choice among Homes 

That Volunteered but 

Did Not Target 

Assistance to 

Low-Performing 

Homes

 35The eight states are Florida, Maryland, Texas, Washington, Maine, Michigan, Missouri, 

and North Carolina. We identified some of these states by reviewing reports and asking

officials in states that we knew had quality assurance programs to identify other states with

similar programs. We did not attempt to determine if additional states had similar 

programs. 

36The organizations included the American Health Care Association, the Alliance for 

Quality Nursing Home Care, and the American Association of Homes and Services for the 

Aging, which are three of the largest long-term care organizations and together represent

the majority of the approximately 16,400 nursing facilities in the United States. 

37In the 8th SOW contracts, CMS specified more selection parameters, requiring QIOs to

work with two groups of intensive participants, including some “persistently poorperforming” homes identified in consultation with state survey agencies; increasing the 

overall number of intensive participants; and requiring geographic distribution of these

homes. 

Page 19 GAO-07-373 Quality Improvement Organizations
nationwide, intensive participants were less likely to be low-performing 

than other homes in their state in terms of the number, scope, and severity 

of deficiencies for which they were cited in standard surveys from 1999 

through 2002. This result may reflect the nature of the homes that 

volunteered for assistance, the QIOs’ selection criteria, or a combination 

of the two. The stakeholders we interviewed—including officials of state 

survey agencies and nursing home trade associations—generally believed

QIOs’ resources should be targeted to low-performing homes. 

QIOs Generally Had a 

Choice of Which Nursing 

Homes to Assist 

Intensively 

Most QIOs had a choice of which nursing homes to assist intensively, as 

more homes volunteered than the QIOs could receive credit for serving 

under the terms of their contracts.38 Of the 38 QIOs in states with 100 or 

more homes, which were expected to work intensively with 10 to 

15 percent of the homes, 30 reported in our Web-based survey that more 

than 15 percent of homes expressed interest in intensive assistance, and 

8 reported that more than 30 percent of homes expressed interest.39 Most 

QIOs selected about as many intensive participants as needed to get the 

maximum weight for the intensive participant element of their contract 

evaluation score. Nationwide, the intensive participant group included just

under 15 percent (2,471) of the 16,552 homes identified by CMS at the 

beginning of the 7th SOW.40

Most QIOs—82 percent of the 51 that responded to our survey—reported 

that it was not difficult to recruit the target number of homes for intensive 

assistance; the remainder reported that it was difficult (12 percent) or very 

difficult (4 percent) to recruit enough volunteers.41 Among the QIOs we 

 38QIOs could select more than 15 percent of the homes in their state for intensive 

assistance. However, the weight given to this component in a QIO’s contract evaluation 

score could not exceed 66 percent—generally, the weight given if the intensive participant

group comprised 15 percent of homes in the state. 

39The 13 QIOs in states with fewer than 100 homes were expected to work intensively with 

at least 10 homes. 

40The 38 QIOs that were expected to work intensively with 10 to 15 percent of the homes in 

their state worked with an average of 15 percent. The other 13 QIOs worked with an

average of 15 homes. 

41The largest proportion of QIOs (27 percent) reported that their most effective recruiting 

tactic was hosting statewide or regional conferences for homes; however, 20 percent did 

not use this tactic at all. The vast majority of QIOs (84 to 98 percent) also sent materials to 

homes, contacted homes by telephone, and asked nursing home trade associations or other

groups to inform homes of the opportunity to participate. 

Page 20 GAO-07-373 Quality Improvement Organizations
interviewed, personnel at two that reported difficulties recruiting homes 

cited homes’ lack of familiarity with QIOs as a barrier. Personnel at one of

these two QIOs commented that the QIO’s first task was to build trust 

among homes and address confusion about its role, as some homes 

thought the QIO was a regulatory authority charged with investigating 

complaints and citing homes for deficiencies. 

Commitment to Working 

with QIOs Was QIOs’ 

Primary Consideration in 

Selecting Homes from 

among Those That 

Volunteered 

QIOs that responded to our Web-based survey almost uniformly cited 

homes’ commitment to working with them as a key consideration in 

choosing among the homes that volunteered to be intensive participants. 

QIOs had wide latitude in choosing among homes because CMS did not 

specify the characteristics of the homes they should recruit or select, 

leaving it to the QIO support contractor to provide voluntary guidelines. 

The QIO support contractor developed guidelines based on input from a 

variety of sources, including QIOs that worked with nursing homes during 

the 6th SOW. Issued at the beginning of the 7th SOW, the guidelines 

emphasized the important role the selected homes would play in the QIOs’ 

contract performance and encouraged QIOs to select homes that 

demonstrated a willingness and ability to commit time and resources to 

quality improvement. The QIO support contractor also encouraged QIOs to 

exclude homes with a high number of survey deficiencies, high 

management turnover, and QM scores that were too good to improve 

significantly. With respect to homes’ survey histories, the QIO support 

contractor reasoned that homes with a high number of deficiencies might 

be more focused on improving their survey results than on committing 

time and resources to quality improvement projects. For example, the care 

areas in which a home was cited for deficiencies might not correspond 

with any of the eight QMs to which CMS limited the QIOs’ quality 

improvement activities (see table 1). In fact, the quality of care area in

which homes were most frequently cited for serious deficiencies in 

surveys in 2006 was the provision of supervision and devices to prevent 

accidents, which does not have a corresponding QM.42

 42Deficiencies are deemed serious if they constitute either actual harm to residents or 

actual or potential for death/serious injury. 

Page 21 GAO-07-373 Quality Improvement Organizations
Consistent with the guidelines, 76 percent of the 41 QIOs that reported in

our Web-based survey their considerations in selecting homes for the 

intensive participant group ranked homes’ commitment as their primary 

consideration. Nearly all QIOs ranked commitment among their top three 

considerations (see fig. 4).

43

 43Although many QIOs excluded some interested homes from the official list of intensive 

participants submitted as a contract deliverable, most QIOs (75 percent) reported that they

gave these homes more assistance than they did other homes in the state, and 37 percent 

reported that they gave these homes as much assistance as they gave intensive 

participants. 

Page 22 GAO-07-373 Quality Improvement Organizations
Figure 4: QIOs’ Considerations in Choosing among Homes That Volunteered for 

Intensive Assistance in the 7th SOW 

0 10 20 30 40 50 60 70 80 90 100

Considerations

Source: GAO survey of QIOs.

Most important consideration

Second most important consideration

Third most important consideration

High level (number

and severity) of

survey deficiencies

Good overall

QM scores

Poor overall

QM scores

Low level (number

and severity) of

survey deficiencies

Survey deficiencies in

areas where the

QIO planned to focus

Mix of homes, with

varying survey

deficiency levels

Moderate overall

QM scores

Mix of homes, with

varying QM scores

Poor QM scores in

areas where the QIO

planned to focus

Committed

to working

with the QIO

98

41

41

20

17

12

10

10

2

0

Percentage of QIOs

Note: Forty-one QIOs reported their considerations in choosing among homes that volunteered for 

intensive assistance. 

Homes’ QM scores were also an important consideration for QIOs. QIOs 

were particularly interested in including homes that had poor QM scores 

in areas where the QIO planned to focus or in assembling a group of 

homes that represented a mix of QM scores. With respect to homes’ 

overall QM scores, the QIOs that responded to our survey were more likely 

Page 23 GAO-07-373 Quality Improvement Organizations
to seek homes with moderate overall scores than homes with poor or good 

overall scores. Similarly, personnel at most QIOs we contacted gave 

serious consideration to homes’ QM scores, looking for homes that 

appeared to need help and could demonstrate improvement. For example, 

personnel at one QIO said that they tended to select homes whose QM 

scores were worse than the statewide average; personnel at another QIO 

said that this QIO selected homes with scores it thought could be 

improved, eliminating homes with either very high or very low scores. 

Personnel at one QIO acknowledged that some QIOs might “cherry pick” 

homes in this way in order to satisfy CMS contract requirements but 

argued that it was not possible for QIOs to predict which homes would 

improve the most. 

QIOs generally gave less consideration to the number of deficiencies 

homes had on state surveys than to their QM scores. However, the 17 QIOs 

that ranked survey deficiencies among their top three considerations in 

our survey were more likely to seek homes with deficiencies in areas 

where they planned to focus or homes with an overall low level (number 

and severity) of survey deficiencies than homes with an overall high level. 

Moreover, of the 33 QIOs that reported in our survey systematically 

excluding some of the homes that volunteered from the intensive 

participant group, nearly one-quarter (8) excluded homes with a high 

number of survey deficiencies. None excluded homes with a low number 

of survey deficiencies.44

Personnel at the QIOs we interviewed offered several reasons for 

excluding homes with a high number of survey deficiencies from the 

intensive participant group. Personnel at several QIOs concurred with the 

QIO support contractor that such homes were likely to be too consumed 

with correcting survey issues to focus on quality improvement. Personnel 

at one QIO suggested that the kind of assistance very poor-performing 

homes need⎯help improving the basic underlying structures of 

operation⎯was not the kind the QIO offered. Personnel at some QIOs said 

they considered not just the level of deficiencies for which homes were 

cited on recent surveys but the level over multiple years or the specific 

 44Some QIOs also considered financial status and management stability in making their 

selections. Among the 51 QIOs surveyed, 8 excluded homes that were struggling financially 

and 5 excluded homes with recent management turnover. Personnel at one of the QIOs we

interviewed explained that the QIO excluded homes with known leadership instability in 

order to avoid having to perform a great deal of training and retraining as administrators

came and went. 

Page 24 GAO-07-373 Quality Improvement Organizations
categories of deficiencies. For example, personnel at one QIO said that 

although the QIO excluded homes with long-standing histories of poor 

performance, it actively recruited homes that had performed poorly only 

on recent surveys. Personnel at another QIO stated that their concern was 

to avoid homes with competing priorities. This QIO sought to include 

homes with deficiencies in the areas it planned to address but to exclude 

homes with deficiencies in other areas on the assumption that these 

homes would not benefit from the assistance it planned to offer. Personnel 

we interviewed at two QIOs said that they worked with some extremely 

poor-performing homes but did not include them on the official list of 

intensive participants submitted to CMS; personnel at one of these QIOs 

explained that they did not want to be held responsible if these homes 

were unable to improve. 

QIOs Did Not Target 

Intensive Assistance to 

Low-Performing Homes 

Our analysis of homes’ state survey histories from 1999 through 2002 

indicates that QIOs did not target intensive assistance to homes that had 

performed poorly in state surveys. Nationwide, the homes in the intensive 

participant group were less likely than other homes in their state to be 

low-performing in terms of the number, scope, and severity of deficiencies

for which they were cited in surveys during that time frame. As illustrated 

in figure 5, the intensive participant group included proportionately more 

homes in the middle of the performance spectrum and proportionately 

fewer at either end. Although our analysis focused on survey deficiencies 

rather than QMs, this result is generally consistent with the results of our 

Web-based survey concerning QIOs’ use of QM scores as selection criteria, 

which showed that QIOs were more likely to select homes with moderate 

overall scores than homes with poor or good overall scores and to seek a 

mix of performance levels among homes in the group. However, not 

knowing the composition of the pool of homes that volunteered for 

assistance, we cannot determine whether the composition of the intensive 

participant group⎯in particular, the disproportionately low number of 

low-performing homes in the group⎯was a function of which homes 

volunteered, which homes the QIOs selected from among the volunteers, 

or a combination of both factors. 

Page 25 GAO-07-373 Quality Improvement Organizations
Figure 5: Comparison of Nonintensive and Intensive Participants’ Performance on

State Surveys

Source: GAO analysis of state survey data.

Low-performing homes

Moderately performing homes

High-performing homes

National-level analysis

QIO-level analysis

0 20 40 60 80 100

Percentage

15.4% 64.8% 19.9%

13.6% 68.7% 17.7%

Nonintensive

participants

Intensive

participants

8 QIOs had 

significantly fewer 

low-performing

homes as intensive 

participants.

5 QIOs had 

significantly more 

moderately performing

homes as intensive 

participants.

9 QIOs had 

significantly fewer 

high-performing

homes as intensive 

participants.

Note: Homes are categorized as low-, moderately, or high-performing on the basis of the number, 

scope, and severity of deficiencies for which they were cited, relative to other homes in their state, in 

three standard state surveys from 1999 through 2002. All differences are statistically significant at 

p-value < 0.05 level. 

On a state-by-state basis, none of the QIOs targeted assistance to lowperforming homes by including proportionately more such homes in the 

intensive participant group. Most QIOs (33 of 51) worked intensively with 

homes that were generally representative of the range of homes in their 

state in terms of performance on state surveys from 1999 through 2002. In

these states, there was no significant difference in the proportion of high-,

moderately, or low-performing homes among intensive participants 

compared with nonintensive participants. However, 18 QIOs worked 

intensively with a group that differed significantly from other homes in the 

state: 8 of these QIOs worked with proportionately fewer low-performing 

Page 26 GAO-07-373 Quality Improvement Organizations
homes, 5 worked with proportionately more moderately performing 

homes, and 9 worked with proportionately fewer high-performing homes.45

Stakeholders Often Stated 

QIOs Should Target 

Intensive Assistance to 

Low-Performing Homes 

Stakeholders we interviewed who expressed an opinion about the homes 

QIOs should target for intensive assistance—11 of the 16 we interviewed—

almost uniformly said that the QIOs should concentrate on low-performing

homes.46 Survey officials in one state suggested that QIOs should use state 

survey data to assess homes’ need for assistance because these data are

often more current than QM data. In their emphasis on low-performing 

homes, stakeholders echoed the views expressed in the 2006 IOM report, 

which recommended that QIOs give priority for assistance to providers, 

including nursing homes, that most need improvement. Other stakeholder 

suggestions regarding the homes QIOs should target are listed in table 2. 

Because the QIOs were required to protect the confidentiality of QIO 

information about nursing homes that agreed to work with them, 

stakeholders were generally not informed which homes were receiving

intensive assistance. One exception was in Iowa, where the QIO obtained 

consent from the selected homes to reveal their identities. 

 45These numbers do not sum to 18 because 4 of the 5 QIOs that selected proportionately 

more moderately performing homes also selected proportionately fewer low- or highperforming homes. 

46Stakeholders included officials of state survey agencies and state nursing home trade

associations. 

Page 27 GAO-07-373 Quality Improvement Organizations
Table 2: Examples of Other Categories of Homes Stakeholders Suggested QIOs

Should Include as Intensive Participants

Category of home Explanation 

Special focus facilities One state survey official suggested that CMS mandate that 

QIOs work with the low-performing homes selected by state 

survey agencies for the Special Focus Facility program.a

Homes lacking resources 

for quality improvement 

Stakeholders suggested targeting small rural facilities, 

“stand-alone” facilities that lack the resources of corporate 

chains, or facilities that are struggling financially. 

High-performing homes Several stakeholders advocated including some highperforming homes. One stakeholder group suggested that

such homes could serve as models and share their 

approaches with homes that were struggling. Another 

suggested that QIOs may include homes at varying 

performance levels to avoid stigmatizing the intensive 

participants as “bad homes.” 

Source: GAO analysis.

Note: Eleven of the 16 stakeholders we interviewed expressed an opinion about which homes the 

QIOs should include as intensive participants. 

a

Seventeen (13 percent) of the 129 facilities in the Special Focus Facility program as of January 2005 

were also among the QIOs’ 2,471 intensive participants in the 7th SOW.

Several stakeholders said that low-performing homes can improve with 

assistance. However, one suggested that QIOs might have to adapt their 

approach⎯for example, by streamlining their training⎯to avoid 

overburdening homes that are struggling with competing demands. 

Another agreed that low-performing homes can benefit from working with 

a QIO but added that real improvements in the quality of care in these 

homes would require attention to staffing, turnover, pay, and recognition 

for staff. The results of one special study funded by CMS during the time 

frame of the 7

th SOW supported stakeholders’ contention that lowperforming homes can improve, although the improvements documented

in these homes cannot be definitively attributed to the QIOs.47 In this study, 

known as the Collaborative Focus Facility project, 17 QIOs worked 

intensively with one to five low-performing homes identified in 

 47One reason that improvements cannot be definitively attributed to the QIOs is that homes 

may have benefited from other quality improvement efforts as well. 

Page 28 GAO-07-373 Quality Improvement Organizations
consultation with the state survey agency.48 According to a QIO 

assessment of the project, the participating homes showed improvement 

in areas related to the assistance provided by the QIO in terms of both the

number of serious state survey deficiencies for which they were cited and 

their QM scores.49 CMS officials pointed out that these improvements were 

hard-won: one-third of the homes that were asked to participate in the 

Collaborative Focus Facility project refused, and those that did participate 

required more effort and resources from the QIOs to improve than did 

other homes assisted by the QIOs. 

Overall, CMS has specifically directed only a small share of QIO resources 

to low-performing homes. In the current contracts (the 8th SOW), CMS 

required QIOs to provide intensive assistance to some “persistently poorperforming homes” identified in consultation with each state survey 

agency. However, the number of such homes that the QIOs must serve is 

small⎯ranging from one to three, depending on the number of nursing 

homes in the state⎯and accounts for less than 10 percent of the homes 

the QIOs are expected to assist intensively. Less than 17 percent of the 144

persistently poor-performing homes the QIOs selected in consultation with 

state survey agencies to assist in the 8th SOW were also special focus 

facilities in 2005 or 2006.

QIOs and stakeholders tended to disagree about whether participation in 

the program should remain voluntary for all homes. QIO personnel we 

interviewed who expressed an opinion generally supported voluntary 

participation on the theory that homes that were forced to participate 

would probably be less engaged and put forth only minimal effort. 

 48In most cases, the state survey agencies and QIOs issued joint letters of invitation to the

homes, and those that agreed to work with the QIOs signed a participation agreement that

addressed issues of confidentiality and information sharing. The state survey agencies’ role 

was generally limited to identifying and helping recruit homes for the project. As with 

homes in the intensive participant group, there was little overlap between homes in the

Collaborative Focus Facility project and homes selected by state survey agencies for the

Special Focus Facility program. Although the Puerto Rico QIO participated in the 

Collaborative Focus Facility project, our analysis focused on QIOs in the 50 states and the

District of Columbia. 

49Over a 1-year period, the average number of survey deficiencies the homes received in 

five areas (comprehensive assessment, comprehensive care plan, pressure sore 

prevention/treatment, quality of care, and physical restraints) changed little, going from 

2.59 to 2.60, but the average number of serious deficiencies they received in these areas 

declined from 0.93 to 0.71. The homes’ QM scores for physical restraints and high- and lowrisk pressure ulcers improved an average of 31 percent (or 38 percent when the score with 

the lowest improvement was dropped from the average). 

Page 29 GAO-07-373 Quality Improvement Organizations
Personnel at some QIOs that opposed mandatory participation suggested 

that creating incentives for homes to improve their quality of care⎯for 

example, through pay for performance⎯would increase homes’ interest in 

working with the QIO. In contrast, most of the state survey agency and

trade association officials we interviewed who expressed an opinion about

the voluntary nature of the QIO program said that some homes should be 

required to work with the QIO. Officials at one state survey agency pointed 

out that the low-performing homes that really need assistance rarely seek

it; these officials believed that working with the QIO should be mandatory 

for low-performing homes and voluntary for moderately to highperforming homes. Another state survey agency official recommended that 

25 to 40 percent of the homes assisted intensively be chosen from among 

the lower-performing homes in the state and required to work with the 

QIO. 

The 7th SOW contracts allowed QIOs flexibility in the QMs they focused on 

and the interventions they used, and while the majority of QIOs selected 

the same QMs and most used the same interventions to assist homes 

statewide, the interventions for intensive participants and staffing to 

accomplish program goals varied. Most QIOs and intensive participants 

worked on the chronic pain and pressure ulcer QMs, but these were not 

the QMs that some intensive participants believed matched their greatest

quality-of-care challenges. To assist all homes statewide, QIOs generally

relied on conferences and the distribution of educational materials. The 

top three interventions for intensive participants included on-site visits 

(87 percent), followed by conferences (57 percent), and small group 

meetings (48 percent). According to nursing home staff we interviewed, 

turnover and experience levels of the QIO personnel that provided them

assistance affected their satisfaction with the program and the extent of

their quality improvements. 

Under the terms of the contracts, both QIOs and intensive participants 

could select QMs to focus on, but most chose to work on two of the same 

QMs.50 While nearly all QIOs chose to work statewide on chronic pain and 

pressure ulcers, they differed on their selection of additional QMs (see 

QIO Contract 

Flexibility Resulted in 

Variation in 

Assistance Provided 

to Intensive 

Participants 

Most Quality Improvement 

Efforts Focused on 

Chronic Pain and Pressure 

Ulcers 

 50For their statewide assistance, three-quarters of the QIOs selected three QMs, the 

minimum number contractually allowed; the remainder selected four QMs. No QIOs 

selected the maximum of five. 

Page 30 GAO-07-373 Quality Improvement Organizations
fig. 6). QIO personnel we interviewed told us they based the choice of QMs 

for their statewide work on input from stakeholders and nursing homes or 

QM data. For example, some stakeholders told us that specific QMs 

selected addressed existing long-term care challenges and were ones on 

which homes in the state ranked below the national average. Personnel 

from two QIOs said they selected QMs based on input from homes in their 

state about which QMs the homes were interested in working on, and 

personnel from several QIOs stated that they selected QMs on which their 

homes could improve. Personnel from one QIO specifically mentioned that 

they selected QMs related to the quality of life for nursing home residents. 

Figure 6: QMs Selected by QIOs for Statewide Interventions and QMs Selected by

Nursing Homes for Intensive Assistance, 7th SOW 

0

10

20

30

40

50

60

70

80

90

100

Activities Infections Walking Delirium

of daily

living

Post-acute- Restraints

care pain

Pressure

ulcers

Chronic

pain

Percentage

Source: GAO analysis of QIO support contractor data.

Quality measures

Quality measures selected by QIOs

Quality measures selected by intensive participants

98

77

84

58

53

32

29

26

33

19 18

8 8

1 2 <1

Most intensive participants worked on a subset of the QMs selected by 

their QIO—chronic pain and pressure ulcers (see fig. 6). The degree to 

which intensive participants knew they had a choice of QMs was unclear.

Of the 14 intensive participants we interviewed that commented on 

Page 31 GAO-07-373 Quality Improvement Organizations
whether they had a choice, 9 said that they did. Staff from these homes 

generally reported having selected QMs related to clinical issues on which 

they could improve. However, the remaining 5 homes indicated that their 

QIO selected the QMs on which they received assistance. Most of these 5 

homes’ staff reported that they would have preferred to work on different 

QMs from the list of eight that are publicly reported on the CMS Nursing 

Home Compare Web site or other clinical issues that reflect their greatest

quality-of-care challenges. 

Statewide Interventions 

Less Variable Than Those 

for Intensive Participants 

The terms of the QIO contract with CMS allowed QIOs to determine the 

kinds of quality improvement interventions they offered to homes, and 

those selected by QIOs were consistent with an approach recommended 

by the QIO support contractor: QIOs generally relied most on conferences 

and the distribution of educational materials to assist homes statewide and 

on on-site visits to assist intensive participants. However, there was a 

greater variety of interventions frequently relied on to assist intensive 

participants. In general, QIOs reported that the interventions they relied on 

most were also the most effective for improving the quality of resident 

care. 

Almost three-quarters of the QIOs included conferences among the two 

interventions they relied on most to provide quality improvement 

assistance to homes statewide (see fig. 7). These QIOs held an average of

nine conferences over the course of the 7th SOW, typically in various cities 

throughout the state to accommodate homes from different regions. Sixtyeight percent of these QIOs reported that more than half the homes in 

their state sent staff to least one conference, and 16 percent of QIOs 

reported that all or nearly all homes did so. QIO personnel reported 

holding conferences to educate homes on quality improvement, discuss 

the relationship between MDS assessments and the QMs, and provide QMspecific clinical information or best practices. Some conferences included

presentations by state or national experts. 

Statewide Assistance 

Page 32 GAO-07-373 Quality Improvement Organizations
Figure 7: Statewide Interventions Most Relied on by QIOs, 7th SOW 

Intervention

Percentage of QIOs

Source: GAO survey of QIOs.

0 10 20 30 40 50 60 70 80

Small group

meetings

Individualized

telephone calls

On-site visits

Conference calls

or video/Web

conferences

Mailings, faxes,

or e-mails

Conferences

Most relied on intervention

Second most relied on intervention

47 25 72

35 37 72

6 24 30

12 2 14

8 8

4 4

Almost three-quarters of QIOs also ranked the distribution of educational

materials by mail, fax, or e-mail among their top two statewide 

interventions. Thirty-two percent of these QIOs sent materials four or 

fewer times per year, whereas 27 percent sent materials 12 or more times 

per year to all or nearly all homes in the state. For the QIOs we 

interviewed, these materials included newsletters, QM-specific tools or 

clinical information related to the QMs, and QM data progress reports for 

the home or state, overall. 

Almost one-third of the QIOs (31 percent) reported that the type or 

intensity of interventions they used to assist homes statewide changed 

over the course of the 7th SOW.51 For example, two QIOs reported that they 

concentrated much of their statewide efforts into the first half of the 

3-year period; one QIO specifically reported doing so in the interest of 

ensuring that any improvements in QMs were reflected in its evaluation 

scores, which, as specified by the contract, were calculated near the mid-

 51The intensity of interventions varies by type of intervention (for example, on-site versus

telephone calls) and with the frequency of use. 

Page 33 GAO-07-373 Quality Improvement Organizations
point of the contract cycle.52 In contrast, five other QIOs reported that they 

increased the intensity of their statewide work over time, in some cases 

concentrating on homes whose performance was lagging. 

For the 8th SOW, CMS has focused resources on assistance to intensive 

participants by eliminating expectations for improvements in QMs 

statewide. However, the contracts still contain statewide elements, 

including a requirement to promote QM target-setting. 

Fifty-one percent of QIOs ranked on-site visits as their most relied on 

intervention with intensive participants and 87 percent ranked it among 

their top three interventions (see fig. 8).53 Both the number of visits and the 

time spent at sites varied considerably. The median number of visits was 5 

but ranged from 1 to 20.54 Sixty-eight percent of QIOs that included on-site 

visits among their top three interventions spent an average of 1 to 2 hours 

at sites each time they visited, while 20 percent spent 3 to 4 hours. QIOs 

that ranked on-site visits as their number one intervention made more and 

longer visits to intensive participants than did QIOs that ranked them 

lower. When surveyed about a typical on-site visit, the majority of QIO 

respondents reported that they generally reviewed the homes’ QM data, 

provided education or best practices, or both. Approximately one-third of 

QIOs that conducted site visits indicated that they had discussions with 

the home about their systems or processes for care, homework 

assignments, or quality improvement activities.55 Some QIOs (26 percent) 

Intensive Assistance 

 52In its 2006 report on QIOs, the IOM recommended that Congress permit extension of the

contract from 3 to 5 years to allow for measurement, refinement, and evaluation of

technical assistance efforts. 

53Because the largest component of the QIOs’ contract evaluation related to the intensive 

participants, we asked QIOs to rank and provide detailed information on a greater number

of interventions for intensive participants than for statewide participants. 

54The median number of times an intervention was provided is the midpoint of all the times

that an intervention was provided, as reported by QIOs. Half the QIOs reported a number 

above the median and half reported a number below. 

55Nearly all QIOs (94 percent) also reported asking intensive participants to complete 

homework assignments on their own. These assignments most frequently involved 

conducting self audits, comparing existing policies and procedures with checklists

provided by the QIO, and developing new practice protocols related to selected QMs. For

example, two homes told us they were given cause-and-effect analysis exercises to 

complete to identify possible causes of and solutions to a problem. Staff from another 

home told us that QIO personnel asked them to conduct a mock survey to prepare for their 

next standard survey. 

Page 34 GAO-07-373 Quality Improvement Organizations
reported that they conducted team-building exercises with the staff when 

on site. 

Figure 8: Intensive Interventions Most Relied on by QIOs and Frequency of Interventions (Range and Median Number) during 

the 7th SOW 

Intervention

Percentage of QIOs Frequency

Source: GAO survey of QIOs.

0 10 20 30 40 50 60 70 80 90 100

Conference calls

or video/Web

conferences

Mailings, faxes,

or e-mails

Individualized

telephone calls

Small group

meetings

Conferences

On-site visits

0 10 20 30 40 50 60 70 80 90 100

Most relied on intervention

Second most relied on intervention

Third most relied on intervention

Mediana

51 22 14 87

29 16 12 57

12 20 16 48

4 25 18 47

2 4 25 31

2 12 12 26 24

6

12 (Range of 12-36)

(Range of 2-40)

(Range of 6-80)

(Range of 1-20)

(Range of 3-15)

10 (Range of 3-44)

8

5

a

The median number of times an intervention was provided is the midpoint of all the times that an

intervention was provided in the 7th SOW, as reported by QIOs. 

QIOs varied in the interventions they used in addition to on-site visits, with 

conferences, small group meetings emphasizing peer-to-peer learning, and 

telephone calls being the three others most commonly used. QIOs that 

included conferences among their three most relied on interventions 

typically held between 3 and 10 during the 7th SOW, but as with site visits, 

some variation existed. After conferences, QIOs were most likely to rely

on small group meetings and telephone calls with individual homes. Nearly 

half of the QIOs ranked these two interventions among their three most 

relied on, but few ranked them highest. The number of homes that 

attended small group meetings varied. An average of 6 to 10 homes was 

most common, but one-fifth of QIOs reported having an average of 20 or 

more homes represented at each meeting. As for telephone calls, the vast 

majority of QIOs (92 percent) that ranked these calls among their three 

most relied on interventions called all or nearly all of their intensive 

participants, typically on a monthly basis. 

Page 35 GAO-07-373 Quality Improvement Organizations
Our interviews with QIOs and intensive participant homes suggested that 

the small group meetings they held generally followed a similar format, 

while telephone calls were used for a variety of purposes. For example, 

personnel from several QIOs and intensive participant homes told us that 

their small group meetings generally included a formal presentation on the 

QMs or related best practices, as well as a time for less formal information 

sharing and peer-to-peer learning among the attendees. Participants 

shared stories about their successes and challenges conducting quality 

improvement. Personnel from a number of QIOs told us they used 

telephone calls to follow up after visits or meetings, discuss the homes’ 

progress on quality improvement, and to decide on next steps. 

Almost two-thirds of QIOs indicated that the type or intensity of 

interventions for intensive participants varied over time. Of these QIOs, 

36 percent reduced the intensity of their interventions (substituting small 

group meetings or telephone calls for on-site visits), while 33 percent did 

the reverse (in some cases increasing the frequency of on-site visits or 

substituting small group meetings for conferences to increase 

participation). For example, personnel from a few QIOs told us that while 

they initially relied on on-site visits to begin the quality improvement 

process, they came to rely more on telephone calls or on small group 

meetings where intensive participants could share their success stories or 

ways to overcome barriers to quality improvement. Seventy-nine percent

of QIOs surveyed varied their interventions based on the needs of 

intensive participants. Thus, personnel from three QIOs told us they 

realized that some homes did not need frequent on-site visits, while others 

needed more. The two specific needs that QIOs cited most as having 

precipitated changes in their interventions were nursing home staffing 

changes and turnover (23 percent) and poorer performance by some 

homes relative to others (15 percent). A few QIOs also noted that 

interventions varied by the preferences or levels of readiness and 

participation of the homes with which they were working. 

Most QIOs we surveyed deemed conferences the most effective statewide

intervention and on-site visits the most effective intensive intervention; 

intensive participant homes we interviewed also found these interventions 

valuable. For homes statewide, most QIOs (54 percent) reported that 

conferences were their most effective intervention, followed by 

distribution of educational materials and on-site visits. Of the one-quarter 

of QIOs that reported they would change their statewide approach, the 

largest proportion (46 percent) would make conferences their primary 

intervention. Staff from several nursing homes we interviewed tended to 

concur that conferences were valuable aspects of the program because 

QIO and Nursing Home 

Perspectives on the 

Interventions 

Page 36 GAO-07-373 Quality Improvement Organizations
conferences included expert presenters, energized or motivated attendees, 

and were free. 

For intensive participants, most QIOs (63 percent) deemed on-site visits 

their most effective intervention, followed by conferences and small group 

meetings. Of the 15 QIOs that said they would change their approach with 

these homes, most (60 percent) would make on-site visits their primary 

intervention, while fewer would rely on small group meetings, 

conferences, and other interventions. One QIO began conducting on-site 

visits and small group meetings when it became apparent that telephone 

calls were less productive than had been anticipated because of the 

difficulty of getting the right staff on the telephone at the right time, the 

lack of speaker phones at many homes, and the lack of staff engagement

on the phone. Staff from a number of nursing homes we interviewed 

agreed that visits by QIO personnel were helpful. Some homes indicated

that having someone from the QIO visit the home maximized the number 

of staff that could take advantage of the quality improvement training 

offered. Furthermore, the on-site visits were motivating and kept staff on 

track with quality improvement efforts. Regarding small group meetings, 

staff we interviewed from a few homes stated that meeting with staff from 

other homes helped validate their own efforts or facilitated the sharing of 

materials and experiences. Staff from one nursing home specifically 

reported that they were disappointed not to have formally participated in

small group meetings with other facilities in the state. 

Homes also found particular types of assistance less helpful. Some homes’ 

staff reported that they did not feel they had the time or the staff necessary

to complete some of the homework assignments expected of them, such 

as conducting chart reviews. Staff at some homes stated that the QIO 

provided quality improvement information with which they were already 

familiar. 

Page 37 GAO-07-373 Quality Improvement Organizations
Our interviews with nursing home staff who worked intensively with the 

QIOs indicated that homes’ satisfaction with the program was influenced 

by the training and experience of the primary QIO personnel who served 

as their principal contact with the QIOs, as well as by turnover among 

these individuals during the course of the 7th SOW.56

QIO Staffing and Turnover 

Influenced Intensive 

Participants’ Satisfaction 

with Program 

When a home’s principal contact with the QIO was a nurse or someone 

with long-term care or quality improvement experience, nursing home 

staff tended to report that this person possessed the knowledge and skills

necessary to help them improve the quality of care in their home. 

Interviewees also spoke appreciatively of QIO personnel who were 

knowledgeable, motivating, and kept them on track with their efforts. 

However, when the QIO principal contact lacked these qualifications or 

characteristics, he or she was perceived as unable to effectively address 

clinical topics with staff. Staff at one home said explicitly that working 

with an experienced nurse, instead of a social worker who seemed to lack

knowledge of long-term care, would have led to greater improvement in 

clinical quality. 

The extent to which QIO primary personnel had the training or experience 

that homes considered important varied. More than half (58 percent) of 

the primary QIO personnel who worked with nursing homes during the 7th

SOW were trained in nursing, and 42 percent held an advanced degree. 

Nationwide, 27 percent of the primary personnel who worked with nursing

homes had less than 1 year of long-term care experience, while 30 percent 

had more than 10 years of such experience.57 Just over half of primary QIO 

personnel (54 percent) working with nursing homes had 4 or fewer years 

of quality improvement experience. Nine percent of QIO personnel had 

 56We defined primary personnel as individuals who devoted more than 20 percent of a fulltime work week to the nursing home component of the contract. Some primary QIO

personnel served as the principal contacts, providing quality improvement assistance to 

homes. According to our survey, 78 percent of QIOs also used outside experts (consultants

or subcontractors) for their quality improvement efforts. The majority of QIOs reported

using these experts to provide presentations or training at conferences, participate in 

conference calls, and develop or review materials. QIOs personnel we interviewed told us

they also used outside experts to train their primary personnel or to provide technical 

assistance to intensive participant homes. 

57Among individual QIOs, the extent of long-term care experience spanned a wide 

spectrum. At five QIOs, 75 percent or more of the primary personnel who worked with 

nursing homes had less than 1 year of long-term care experience, while at two QIOs, all of 

the primary personnel who worked with nursing homes had more than 10 years’

experience. 

Page 38 GAO-07-373 Quality Improvement Organizations
more than 10 years’ experience in both long-term care and quality 

improvement. Few of the personnel working with nursing homes during 

the 7th SOW gained any of their experience working for the QIO during the

6th SOW because there was little overlap in personnel across the two 

periods. 

Our interviews with intensive participants suggested that turnover among 

primary QIO personnel lowered nursing homes’ satisfaction with the 

program. Our survey revealed that turnover was particularly high at some 

QIOs. At 24 QIOs, 25 percent or more of primary personnel who worked

with nursing homes did so for less than half of the 36-month contract, and 

at 6 QIOs, the proportion was 50 percent or more. When a nursing home’s 

principal contact with a QIO changed frequently, nursing home staff we 

interviewed reported that they received inconsistent assistance that was 

disruptive to their efforts to improve quality of care. For example, one 

nursing home we visited had four different principal contacts over the 

course of the 7th SOW and found this to be frustrating because, just as they 

were establishing a relationship with a contact, the contact would leave.

Staff at another home complained that their interaction with QIO primary 

personnel turned out not to be the learning experience that the staff 

thought it would be. 

Staffing levels for the nursing home component also varied among QIOs. 

As would be expected, given the wide variation in the number of nursing 

homes per state, the number of full-time-equivalent (FTE) staff working 

with nursing homes varied across the QIOs, ranging from 0.50 to 12. 

However, the ratio of QIO staff FTEs to intensive participant homes also 

showed significant variation. On average, the ratio was about 1 to 14; but 

for at least 9 QIOs, the ratio of staff FTEs to homes was 1 to 10 or fewer, 

and for at least 8 QIOs, the ratio was 1 to 18 or more. 

Although the QIOs’ impact on the quality of nursing home care cannot be

determined from available data, staff we interviewed at most nursing 

homes attributed some improvements in the quality of resident care to 

their work with the QIOs. Nursing homes’ QM scores generally improved

enough for the QIOs to surpass by a wide margin the modest contract 

performance targets set by CMS; however, the overall impact of the QIOs 

on the quality of nursing home care cannot be determined from these data 

because of the shortcomings of the QMs as measures of nursing home 

quality and because confounding factors make it difficult to attribute 

quality improvements solely to the QIOs. Multiple long-term care 

professionals we interviewed indicated that QMs should not be used in 

QIOs’ Impact on 

Quality Is Not Clear,

but Staff at Homes We

Contacted Attributed 

Some Improvements 

to QIOs

Page 39 GAO-07-373 Quality Improvement Organizations
isolation to measure quality improvement, but combined with other 

indicators, such as state survey data. Moreover, the effectiveness of the 

individual interventions QIOs used to assist homes also cannot be 

evaluated with the available data. CMS planned to enhance evaluation of 

the program during the 8th SOW, but a 2005 determination by HHS’s Office 

of General Counsel that the QIO program regulations prohibit QIOs from

providing to CMS the identities of the homes they are assisting has 

hampered the agency’s efforts to collect the necessary data. Although the 

impact of the QIOs on the quality of nursing home care is not known, over 

two-thirds of the 32 nursing homes we interviewed attributed some 

improvements in care to their work with the QIOs. 

All QIOs Met Modest 

Targets for QM 

Improvement, but the 

Impact of the QIOs on the 

Quality of Nursing Home 

Care Cannot Be 

Determined 

Although all of the QIOs met the modest targets CMS set for QM 

improvement among homes both statewide and in the intensive participant 

group, the impact of the QIOs on the quality of nursing home care cannot

be determined because of the limitations of the QMs and because 

improvements cannot be definitively attributed to the QIOs. The 

effectiveness of the specific interventions used by the QIOs to assist 

homes also cannot be evaluated with the available data. 

All QIOs met the CMS performance targets for the nursing home 

component of the 7th SOW. In addition to receiving an overall passing score 

for this component, nearly all QIOs surpassed expectations for each of the 

three elements that contributed to the overall score: provider satisfaction, 

improvement in QM scores among intensive participants, and 

improvement in QM scores among homes statewide. In fact, about twothirds of the QIOs achieved at least five times the expected 8 percent 

improvement among intensive participants, and nearly half achieved at 

least twice the expected 8 percent improvement statewide.58

All QIOs Met CMS’s Modest 

Targets for Improvement in 

Nursing Home QMs 

CMS officials stated that the targets set for the nursing home component 

of the contract were purposely modest. Because the 7th SOW marked the 

first time all QIOs were required to work with nursing homes on quality 

improvement, and little data existed to predict how much improvement 

 58The improvement, or relative change, in a home’s QM scores is calculated by subtracting 

its score at remeasurement from its score at baseline and dividing by its score at baseline.

For example, if the number of residents with chronic pain in a 100-bed home decreased

from 20 to 12⎯which translates to a change in scores from 0.20 to 0.12⎯the improvement

in the home’s pain QM would be 40 percent ([0.20-0.12]/0.20). 

Page 40 GAO-07-373 Quality Improvement Organizations
could be expected, CMS deliberately designed performance criteria to 

limit QIOs’ chances of failing. For example, expectations for 

improvements in QM scores were set no higher for intensive participants 

than for homes statewide. In addition, CMS modified the evaluation plan 

so that if an intensive participant worked on more than one QM, the QM 

that improved least was dropped before the home’s average improvement 

was calculated. CMS officials told us that, based in large part on QIOs’ 

performance in the 7

th SOW, the agency raised its expectations for the 8th

SOW. For example, QIOs are required to work with most intensive 

participants on four specified QMs and to achieve an improvement rate of 

15 to 60 percent, depending on the QM and the homes’ baseline scores. In

addition, CMS will no longer drop the QM that improved least when 

calculating homes’ average improvement.59

Long-term care experts we interviewed generally agreed that CMS’s use of 

QMs to evaluate nursing home quality—and by extension, QIOs’ 

performance⎯is problematic because of unresolved issues related to the

QMs and the MDS data used to calculate them. 

CMS’s Use of QMs to Evaluate 

QIO Performance Is 

Problematic 

QMs. As we reported in 2002, the validity of the QMs CMS proposed to 

publicly report in November 2002 was unclear.60 Although the validation 

study commissioned by CMS found that most of the publicly reported QMs 

were valid and reflected the quality of care delivered by facilities, longterm care experts have criticized the study on several grounds. For 

example, a 2005 report concluded that (1) the statistical criteria for the 

validity assessments were not stringent and (2) the researchers did not 

attempt to determine whether QMs were associated with quality of care at

the resident level.61 As a result, it is not clear whether a resident who 

triggers a QM (e.g., is assessed as having his or her pain managed 

 59The four QMs specified in the contract are pressure ulcers among high-risk patients, 

restraints, depression management, and chronic pain management. With most intensive 

participants, QIOs are expected to work on all four QMs and achieve a relative 

improvement rate of 15 to 60 percent. With the small group of persistently poor-performing 

homes QIOs are now required to assist, they are expected to work on two QMs (pressure

ulcers among high-risk patients and restraints) and achieve an improvement rate of 

10 percent. 

60GAO-03-187. 

61Greg Arling and others, “Future Development of Nursing Home Quality Indicators,” The 

Gerontologist, vol. 45, no. 2 (2005). 

Page 41 GAO-07-373 Quality Improvement Organizations
inadequately) is actually receiving poor care.62 The lack of correlation 

among the QMs⎯a home may score well on some QMs and poorly on 

others⎯also calls into question their validity as measures of overall 

quality. Since 2002, CMS has removed or replaced 5 of the original 10

QMs⎯including some of those on which the QIOs were evaluated during 

the 7th SOW—to address limitations in the QMs, such as reliability and 

measurement problems. (See app. II for a list of the QMs as of November 

2002 and February 2007). 

Risk adjustment also impacts the validity of QMs. There is general 

recognition that some QMs should be adjusted to account for the 

characteristics of residents. However, there is disagreement about which 

QMs to adjust, what risk factors should be used, or how the adjustment 

should be made. For example, one expert we interviewed suggested that in 

many cases pressure ulcers start in hospitals; the pressure ulcer QM does 

not account for the origin of ulcers. Another expert highlighted the 

difficulty of making an appropriate adjustment—noting, for example, that

improperly risk-adjusting the pressure ulcer QM could mask poor care that 

contributed to the development of ulcers. 

MDS. We have also previously reported concerns about MDS 

reliability⎯that is, the consistency with which homes conduct and code

the assessments used to calculate the QMs.63 CMS awarded a contract for 

an MDS accuracy review program in 2001 but revamped the program in 

2005, near the end of the QIOs’ 7th SOW, acknowledging weaknesses—

mainly its reliance on off-site, rather than on-site, accuracy and 

verification reviews—that we had previously identified.64 Some states that 

sponsor on-site MDS accuracy reviews continue to report troubling rates 

of errors in the data. For example, officials of Iowa’s program reported an 

average MDS error rate of approximately 24 percent in 2005. 

Our interviews with long-term care experts and nursing home staff 

suggested that the chronic pain QM⎯which was selected as a focus of 

quality improvement work by many QIOs and intensive participant nursing 

homes⎯may be particularly vulnerable to error in the underlying MDS 

 62A resident who triggers a QM is included in both the numerator and denominator when a 

facility’s QM score is calculated.

63GAO-02-279 and GAO-03-187. 

64In April 2005, CMS ended work under its data assessment and verification contract but

signed a new contract in September 2005 that focused on on-site reviews of MDS accuracy. 

Page 42 GAO-07-373 Quality Improvement Organizations
data. Possible sources of error are systematic differences in the extent to

which facilities identify and assess residents in pain and 

misunderstandings about how to accurately code MDS questions specific 

to pain. For example, staff from two nursing homes told us that their pain 

management QM scores improved after staff realized that they had been 

mistakenly coding residents as having pain even though their pain was 

successfully managed. Moreover, experts we interviewed noted that 

higher-quality homes may have worse pain QM scores because they do a 

better job of identifying and reporting pain in residents. 

The use of MDS data to measure the quality of care in nursing homes is 

also problematic because the MDS was not designed as a quality 

measurement tool and does not reflect advances in clinical practice. CMS 

is updating the MDS now to address these limitations. For example, 

instead of asking homes to classify the severity of a pressure ulcer on the 

basis of a four-stage system, the draft MDS now under review includes a 

measurement tool intended to more accurately classify the severity of a 

pressure ulcer.65 In addition, facilities are asked to indicate whether the 

pressure ulcer developed at the facility or during a hospitalization. CMS 

does not yet have an official release date for the revised MDS but 

anticipates that all validation and reliability testing will be completed by 

December 2007. 

Other Measures of Quality. Multiple long-term care professionals we 

interviewed, including stakeholders and experts on quality measurement, 

recommended both that the QMs undergo continued refinement and that 

they not be used in isolation to assess the quality of care in nursing homes. 

They suggested a number of other sources of information as alternatives 

or complements to QMs for measuring quality. For example, a 

representative of the National Quality Forum (NQF), a group with which 

CMS contracted to provide recommendations on quality measures for 

public reporting, stated that experts do not consider the QMs sufficient in 

themselves to rate homes and that the other quality markers⎯such as 

perceptions of care by family members, residents, and staff; state survey 

data; and resident complaints⎯also provide useful information about 

quality of care. Other long-term care professionals we interviewed 

 65Stages of pressure ulcer formation are I—skin of involved area is reddened, II—upper

layer of skin is involved and blistered or abraded, III—skin has an open sore and involves 

all layers of skin down to underlying connective tissue, IV—tissue surrounding the sore has

died, exposing muscle and bone. 

Page 43 GAO-07-373 Quality Improvement Organizations
suggested these and other measures, including nursing home staffing 

levels and staff turnover and retention rates. 

Factors such as the existence of other quality improvement efforts make it 

difficult to evaluate QIOs’ work with nursing homes and attribute quality 

improvement solely to QIOs. In an assessment of the QIO program during 

the 7th SOW, CMS and QIO officials acknowledged this difficulty. The 

assessment found that intensive participants improved more than 

nonintensive participants on all five QMs studied, and for each QM, 

intensive participants that worked on the QM improved more than 

intensive participants that did not.66 However, the authors noted that these 

results could not be definitively attributed to the efforts of the QIOs 

because improvements may have been influenced by a variety of factors, 

including preexisting differences between intensive participants and 

nonintensive participants;

67 public reporting of the QMs, which may have 

focused homes’ attention on improving these measures; and other quality 

improvement efforts to which homes may have been exposed. As noted 

earlier in this report, these other efforts included, but were not necessarily 

limited to, initiatives sponsored by state governments, nursing home trade 

associations, and CMS. While these other efforts varied considerably in the 

intensity of technical assistance offered⎯ranging from a trade associationsponsored program that homes characterized as essentially signing a 

quality improvement pledge, to state-sponsored programs that involved

on-site visits by experienced long-term care nurses who provided bestpractice guidelines, educational materials, and clinical tools⎯the fact that 

the efforts were present made it impossible to attribute quality 

improvements solely to the QIOs. 

Influence of Other Factors on 

Nursing Home Quality Makes It 

Difficult to Evaluate QIOs’ 

Impact 

In its 2006 report on all aspects of the QIO program, IOM highlighted 

similar shortcomings in previous studies of the QIO program and called for 

more systematic and rigorous evaluations. IOM concluded that although 

the QIOs may have contributed to improvements in the quality of care, the 

existing evidence was inadequate to determine the extent of their 

contribution. In its response to the IOM study, CMS acknowledged the 

 66William Rollow and others, “Assessment of the Medicare Quality Improvement 

Organization Program,” Annals of Internal Medicine, vol. 145, no. 5 (2006). 

67Because homes must volunteer and be selected by the QIOs to receive intensive 

assistance, intensive participants may differ from nonintensive participants in ways that 

affect their capacity to improve their QM scores, such as differences in motivation and 

commitment, available resources, and competing priorities. 

Page 44 GAO-07-373 Quality Improvement Organizations
need to strengthen its methods of evaluating the program and outlined 

plans to convene an evaluation expert advisory panel to make

recommendations on the framework for the next contracts (the 9th SOW, 

which will begin in 2008). CMS also stated that it will collect information 

during the 8th SOW that will allow it to control for differences in motivation 

between intensive and nonintensive participants but did not specify the 

nature of this information.68 Subsequently, HHS’s Office of General 

Counsel determined that QIO program regulations prohibited QIOs from

providing to CMS the identities of intensive participants.69 CMS officials 

acknowledged that this prohibition posed a considerable challenge to their 

evaluation plans and said that as a short-term solution the agency might 

contract with one of the QIOs to evaluate the program, with the possible 

stipulation that the findings be verified by an independent auditor. 

CMS collected little information about the specific interventions QIOs 

used to assist nursing homes and acknowledged that the information it did 

have was not sufficiently comprehensive or consistent to be used to 

evaluate the interventions’ effectiveness. In general, CMS’s oversight of 

QIOs’ work on the nursing home component consisted of ensuring that the 

QIOs produced the reports and deliverables specified in the contracts and

appeared on track to meet performance targets. 

CMS Data Are Too Limited to 

Evaluate Effectiveness of 

Specific QIO Interventions 

CMS’s primary source of data about QIOs’ interventions was the monthly 

activity reports the QIOs were required to submit through the Program 

Activity Reporting Tool (PARTner). In these reports, QIOs were to 

document the specific interventions they provided to each home, using 

such activity codes as “on-site support” and “stand-alone workshops on 

quality improvement.” However, with only seven activity codes for QIOs to 

choose from, the level of detail in these reports was low. For example, the 

 68At a meeting on October 31, 2006, of the Technical Expert Panel convened by the

contractor tasked to design an evaluation of the QIO program for the Office of the Assistant 

Secretary for Planning and Evaluation of HHS, panel members underscored the difficulty of 

controlling for a subjective condition such as motivation to improve the quality of care and 

noted the potential for biased assessments of the impact of the QIOs if differences in 

motivation are not accounted for appropriately. 

69According to CMS guidance, the names of participants in collaborative quality 

improvement projects constitute quality review study information. See QIO Manual, §16005

(Rev. 07-11-03). Federal regulations specify that quality review study information revealing

the identities of practitioners and institutions must be disclosed to CMS “on site” or at the 

QIOs’ place of operation. See 42 C.F.R. §480.140 (2005). That restriction does not apply to

disclosures to certain other federal agencies, such as HHS Office of Inspector General or

GAO. See 42 C.F.R. §480.140(b)(2005). 

Page 45 GAO-07-373 Quality Improvement Organizations
same code would be used for a full-day visit as for an hour visit. Moreover, 

because QIOs were not expected to enter any code more than once per 

month for a home, a code for on-site support could indicate a single visit 

or multiple visits. The system also captured no information about the 

content of visits or other interventions. From the perspective of the QIOs, 

the system was of limited use: More than half of the 52 QIOs surveyed by

IOM rated PARTner fair or poor in terms of both value and ease of use. 

Staff at one QIO we interviewed reported using tracking systems they 

developed themselves, rather than PARTner, to monitor their work. 

CMS regional offices and the nursing home satisfaction survey gathered 

some additional information about the interventions used by QIOs. The 

CMS regional offices gathered information through telephone calls and 

visits to the QIOs and by participating in quarterly conference calls during 

which QIOs and CMS regional and central offices discussed issues related 

to the nursing home component of the contract. The regional office staff 

also reviewed information entered into the PARTner data system by QIOs, 

but they focused their evaluations on QIO contract compliance and not on 

the effectiveness of specific interventions because—as some regional staff 

emphasized—the contracts were performance-based, and therefore it was 

not their place to “micromanage” the QIOs or to advocate for or against 

specific interventions. Feedback from nursing homes was gathered 

through the nursing home satisfaction survey, conducted after the 

midpoint of the contract cycle by a contractor for CMS.70 The survey 

collected information about the frequency of, and homes’ satisfaction 

with, a range of interventions, including on-site visits, training workshops, 

one-on-one telephone calls, conference calls, one-to-one e-mails, and 

broadcast e-mails. However, the survey collected no information about the 

content of these interventions or the aspects that contributed to providers’ 

satisfaction or dissatisfaction. 

In its 2006 report on the QIO program, IOM emphasized the need for CMS 

to gather more information about specific interventions and noted that 

CMS was uniquely positioned to determine which interventions lead to 

high levels of quality improvement. The agency responded that it will 

collect information during the 8th SOW to better explore the relationship 

between the intensity of assistance provided by the QIO and the level of 

improvement, but did not specify the type of information it will collect. As 

 70For the survey conducted during the 7th SOW, the response rate for nursing homes was 

95 percent. 

Page 46 GAO-07-373 Quality Improvement Organizations
of March 2007, CMS had not yet implemented a revamped PARTner 

system. In addition, the agency cancelled its plans to conduct an initial 

survey of nursing homes early in the contract period and now plans to 

conduct only one, later in the contract period. CMS officials explained that 

the delay and cancellation were due at least in part to the determination by 

HHS’s Office of General Counsel that QIOs could not provide to CMS the 

identities of intensive participants to CMS. 

Homes That Received 

Intensive Assistance 

Generally Attributed Some 

Improvements in Quality 

of Care to Work with QIOs 

Although the impact of the QIOs on the overall quality of nursing home 

care cannot be determined, staff we interviewed at over two-thirds of the 

32 nursing homes stated that they improved the care delivered to residents 

as a result of working intensively with the QIOs. Staff at 23 of the 32

homes told us that they implemented new, or made changes to existing, 

policies and procedures related to pain or pressure ulcers. Of the 23 

nursing homes, staff from 21 stated that they changed the way they 

addressed resident pain. In general, these changes involved implementing 

pain scales or new assessment forms. Staff at some facilities noted that 

working with the QIO heightened staff awareness of resident pain, 

including awareness of cultural differences in the expression of pain. Staff 

at 8 of the 23 nursing homes stated that they changed the way they 

addressed pressure ulcers. In general, these 8 homes implemented new 

assessment tools, changed assessment plans, or revised facility policies 

using materials provided by the QIO. (Table 3 provides examples of 

resident care improvements related to pain assessment and treatment and 

pressure ulcers.) Staff at 13 of the 32 nursing homes stated that the 

changes they made as a result of working with the QIOs were sustainable, 

but staff from some nursing homes noted that staffing turnover at their 

facilities could affect sustainability. 

Page 47 GAO-07-373 Quality Improvement Organizations
Table 3: Examples of Resident Care Improvements Made by Homes as a Result of

Intensive Assistance Provided by QIOs, 7th SOW 

Care area Example 

Pain Had nurses evaluate acute pain management at end of each shift with 

nurse aide involvement 

Used interventions other than medications, such as massage, 

compresses, and repositioning 

Recorded signs of pain when providing care for wounds such as 

pressure ulcers 

Began using or resumed using pain scales to assess resident pain 

Implemented pain policy that addresses both cognitively intact 

residents and residents who have dementia or are nonverbal 

Pressure ulcers Increased skin assessments to four times a week and had nurse aides 

document changes on a daily basis 

Established a wound care team 

Used a tracking tool to measure depth and width of pressure ulcers 

Conducted skin checks when a resident returned to the facility, such 

as after a hospitalization 

Source: GAO interviews with staff from nursing homes assisted intensively by the QIOs. 

Of the 32 nursing homes we contacted, staff from 4 specifically stated that

working with the QIO did not change their quality of care. For example, 

staff from one home stated that the QIO did not offer their facility any new 

or helpful information and did not offer feedback on how the facility’s 

processes could improve. Staff from another home reported that the 

information provided by the QIO was on techniques their facility had 

already implemented. Staff at a third home noted that while the QIO was a 

good resource, the home could have done as much on its own, without 

assistance from the QIO. Staff at three facilities, none of which reported 

making any policy or procedural changes, said the facilities experienced 

worse survey results while working with their QIO; staff from two of the 

three reported being cited for quality deficiencies in the specific areas they

had been addressing with the QIO. Staff at one of these facilities believed 

they were cited because their work with the QIO had made the surveyor

more aware of the facility’s problems in this area. 

Although it is difficult to evaluate the impact of QIO assistance, the QIO 

program does have the potential to help improve the quality of nursing 

home care. CMS program improvements for the 8th SOW, such as the 

agency’s decision to focus resources on intensive rather than statewide 

assistance and its plans to improve evaluation, are positive steps that 

Conclusions 

Page 48 GAO-07-373 Quality Improvement Organizations
could result in more effective use of available funds and provide more 

insight into the program’s impact. Our evaluation of assistance provided 

during the 7th SOW, however, raised two major questions about the future 

focus, oversight, and evaluation of the QIO program, which we address 

below. 

Given the available resources, which homes and quality-of-care 

areas should CMS direct QIOs to target for intensive assistance?

We found that QIOs generally did not target intensive assistance to homes 

that performed poorly in state surveys, partly because of concerns about 

the willingness and ability of such homes to simultaneously focus on 

quality improvement and cooperate with the QIOs. However, the 

Collaborative Focus Facility project during the 7th SOW demonstrated that 

low-performing homes could improve their survey results and QM scores; 

subsequently, CMS required that during the 8th SOW each QIO work with 

up to three such homes—about 10 percent of the total number that QIOs 

are expected to assist intensively. Stakeholders we interviewed believed 

that even more emphasis should be placed on assisting low-performing 

homes. We found that there was little overlap between homes that 

participated in the QIO Collaborative Focus Facility project and in CMS’s 

Special Focus Facility program, which is a program involving about 130 

nursing homes nationwide that, on the basis of their survey results, receive 

increased scrutiny and enforcement by state survey agencies. The limited 

overlap suggests that each state has more than three low-performing 

facilities that could benefit from QIO assistance. 

Targeting assistance to low-performing homes could pose challenges given 

the voluntary nature of the program—homes must agree to work with a 

QIO. QIOs maintain that voluntary participation is critical to ensuring 

homes’ commitment to the program. However, the risk in this approach is

that some of the homes that need help most will not get it. Indeed, in the 

Collaborative Focus Facility project, some of the low-performing homes 

that were asked to participate refused QIO assistance. In addition, QIOs 

expended more resources working to improve these low-performing 

homes than were required to assist better-performing homes. Thus, 

increasing the number of low-performing homes QIOs are required to 

assist above the small number mandated for the 8th SOW might necessitate 

decreasing the total number of homes assisted. However, existing 

resources might be maximized if QIOs worked with each home only on the 

quality-of-care areas that pose particular challenges for that home. 

Page 49 GAO-07-373 Quality Improvement Organizations
Could interim steps be taken to improve oversight and evaluation 

of QIOs’ work with nursing homes before the contracting cycle that 

begins in August 2008? Currently, CMS collects data primarily on QIO 

outcomes—specifically, changes in QM scores—and costs. CMS needs 

more detailed data, particularly about the type and intensity of

interventions used to assist nursing homes, to improve its oversight and 

evaluation of the QIO program. Without such data, CMS cannot hold QIOs 

fully accountable for their performance under their contract with CMS. 

Some evaluation activities are now scaled back or on hold because HHS 

determined early in the 8th SOW that program regulations prohibited the 

QIOs from providing to CMS the identities of the intensive participants. 

Such a firewall presents a major impediment to improved oversight and 

evaluation of the QIO program and prevented CMS from implementing 

interim changes it planned to make. For example, for the 7th SOW, CMS

contracted for one nursing home satisfaction survey to be conducted near 

the end of the contract period—too late to be of use in interim monitoring

of the QIOs’ performance. For the 8th SOW, CMS had planned to contract 

for two surveys but was forced to cancel the one planned for early in the 

contract period because it was unable to provide the names of intensive 

participants to its survey contractor. Moreover, the lack of these data 

would preclude CMS from independently verifying QIO compliance with

such contract requirements as the geographic dispersion of intensive 

participants in each state.

CMS evaluated QIOs’ work with nursing homes primarily on the basis of 

changes in QM scores; given the weaknesses of QM data, the current 

reliance on these data appears unwarranted. While CMS actions to 

improve the MDS instrument as a quality measurement tool are important, 

the agency has not yet established an implementation date. Although 

multiple long-term care professionals believe that multiple indicators of 

quality, including deficiencies on homes’ standard and complaint surveys 

and residents’ and family members’ satisfaction with care, should be used 

to measure quality improvement, CMS is not currently drawing on these 

data sources to evaluate QIOs’ efforts. Recognized shortcomings in these 

other data sources—such as the understatement of survey deficiencies by

state surveyors—underscore the importance of using multiple data 

sources to evaluate QIO outcomes. 

Page 50 GAO-07-373 Quality Improvement Organizations
To ensure that available resources are better targeted to the nursing 

homes and quality-of-care areas most in need of improvement, we 

recommend that the Administrator of CMS take the following two actions: 

• Further increase the number of low-performing homes that QIOs assist 

intensively. 

• Direct QIOs to focus intensive assistance on those quality-of-care areas on

which homes most need improvement. 

To improve monitoring of QIO assistance to nursing homes and to 

overcome limitations of the QMs as an evaluation tool, we recommend 

that the Administrator of CMS take the following three actions: 

• Revise the QIO program regulations to require QIOs to provide to CMS the 

identities of the nursing homes they are assisting in order to facilitate 

evaluation. 

• Collect more complete and detailed data on the interventions QIOs are 

using to assist homes. 

• Identify a broader spectrum of measures than QMs to evaluate changes in

nursing home quality. 

We obtained written comments from CMS on our draft report. CMS 

addressed three of our five recommendations. It concurred with two of the 

three recommendations but did not specify how it would implement them, 

and it continues to explore options for implementing the third

recommendation. Our evaluation of CMS’s comments follows the order we 

presented each recommendation in the report. CMS’s comments are 

included in app. III. 

Further increase the number of low-performing homes that QIOs 

assist intensively. CMS agreed with this recommendation but did not 

specify a time frame for addressing it or indicate how many lowperforming homes it will expect QIOs to assist in the future. Although our 

report focused on the most recently completed contract period (the 7th

SOW), we acknowledged that in the current contract period, CMS required 

QIOs to provide intensive assistance to some “persistently poorperforming” homes identified in consultation with each state survey 

agency. However, we pointed out that the number of these homes the 

QIOs were required to serve was small, accounting for less than 10 percent 

Recommendations for 

Executive Action 

Agency Comments 

and Our Evaluation 

Page 51 GAO-07-373 Quality Improvement Organizations
of the homes they were expected to assist intensively. CMS commented 

that preliminary estimates from a special study conducted during the 7th

SOW indicated that assisting chronically poor-performing homes cost the 

QIOs 5 to 10 times as much as assisting the “usual” home.71 Our report 

acknowledged that additional resources were required for QIOs to assist 

low-performing homes but suggested that CMS could decrease the total 

number of homes assisted in order to increase the number of lowperforming homes beyond the small number mandated for the 8th SOW. 

Direct QIOs to focus intensive assistance on those quality-of-care 

areas on which homes most need improvement. CMS did not directly 

respond to this recommendation, but did point out that about one-third of

QIOs were working primarily with homes on QMs on which the homes 

scored worse than the national average during the 8th SOW. Our 

recommendation was to direct all QIOs to focus intensive assistance on 

QMs that reflect homes’ greatest quality-of-care challenges. We had 

reported that some nursing homes assisted intensively by QIOs did not 

have a choice of QMs on which to work. We concluded that having QIOs 

work intensively with homes only on the quality-of-care issues that posed 

particular challenges to them would maximize program resources. 

Revise QIO program regulations to require QIOs to provide CMS 

with the identities of the homes assisted in order to facilitate 

evaluation. CMS did not specifically indicate whether it agreed with this 

recommendation, but did indicate that it continues to explore options 

which would allow access to data on the homes assisted intensively in 

order to facilitate evaluation. However, CMS expressed concern that 

providing this access could potentially subject the information to laws that 

could afford third parties similar access. We believe that CMS should 

continue to evaluate how best to maintain an appropriate balance between 

disclosure and confidentiality. If CMS’s evaluation indicates that it is 

unable to incorporate adequate confidentiality safeguards to promote 

voluntary participation in QIOs’ quality improvement initiatives, the 

agency could seek legislation that would provide such safeguards. 

 71CMS did not provide this cost estimate during the course of our work. 

Page 52 GAO-07-373 Quality Improvement Organizations
Collect more complete and detailed data on the interventions QIOs 

use to assist homes. CMS responded to this recommendation, although it 

labeled it “improve the monitoring of QIO activities,” and agreed with our 

recommendation. CMS noted that, in concert with HHS, it is reviewing 

recommendations from the IOM’s 2006 report on QIOs, which may result 

in redesigning the program, including systems for evaluating QIO activities 

in different care settings, such as nursing homes. CMS did not discuss how 

it planned to collect additional data on QIO nursing home interventions. 

Further, it stated that it may incorporate data-handling and -reporting 

features of the nursing home subtask into overall program improvements. 

We have reservations about this plan because we found that CMS 

collected little information about specific QIO interventions with nursing

homes during the 7th SOW, the information collected was not sufficiently 

comprehensive or consistent to be used to evaluate the interventions’ 

effectiveness, and QIOs themselves reported that the data collection 

system was of limited use to them. 

Identify a broader spectrum of measures than QMs to evaluate 

changes in nursing home quality. CMS did not directly address this 

recommendation. However, the agency took issue with our judgment that 

the use of QMs to evaluate nursing home quality—and by extension, QIOs’ 

performance—is problematic. CMS commented that the QMs have passed 

through rigorous development, testing, deployment, and national 

consensus processes. We reported that the study commissioned by CMS to 

validate the QMs has been criticized by experts on several grounds, 

including a lack of statistical rigor. We also noted that CMS has revised or 

is currently revising both the QMs and the MDS data used to calculate 

them to address limitations, such as reliability and measurement 

problems. For example, CMS has removed or replaced 5 of the original 10

QMs since 2002, including some of those on which the QIOs were 

evaluated during the 7th SOW. In addition, CMS is currently updating the 

MDS to reflect advances in clinical practice and to improve its utility as a 

quality measure tool. While we expect that these efforts will improve the 

QMs as measures of nursing home quality, we believe that the QMs’ 

current limitations argue for the use of a broader spectrum of measures to 

evaluate changes in nursing home quality. Multiple long-term care 

professionals we interviewed recommended that the QMs not be used in 

isolation to assess the quality of care in nursing homes; these professionals 

suggested a range of measures that could be used to supplement the QMs, 

including perceptions of care by family members, residents, and staff; 

state survey data; and nursing home staffing levels.

Page 53 GAO-07-373 Quality Improvement Organizations
As arranged with your office, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 30 days after its 

issue date. At that time, we will send copies to the Administrator of the 

Centers for Medicare & Medicaid Services and appropriate congressional 

committees. We will also make copies available to others upon request. In

addition, the report will be available at no charge on the GAO Web site at 

http://www.gao.gov. 

If you or your staff have any questions about this report, please contact me 

at (202) 512-7118 or allenk@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. GAO staff who made major contributions to this report are 

listed in appendix IV. 

Sincerely yours, 

Kathryn G. Allen 

Director, Health Care 

Page 54 GAO-07-373 Quality Improvement Organizations
Appendix I: Scope and Methodology 

Page 55 GAO-07-373 

Appendix I: Scope and Methodology 

Our analysis of QIOs’ work with nursing homes had three major 

components: (1) site visits to five QIOs, (2) analysis of state survey data to

compare homes that were assisted intensively with homes that were not, 

and (3) a Web-based survey of 51 QIOs. 

We visited a QIO in each of five states to gather detailed information about 

QIOs’ work with nursing homes from the perspective of the QIOs, nursing 

homes in the intensive participant group, and stakeholders; we used this 

information to address all three objectives.1

 We selected the states-⎯and 

by extension, the QIOs that worked in those states—on the basis of six 

criteria described in the following section. After selecting the QIOs, we 

identified nursing homes that received intensive assistance and 

stakeholders to contact for interviews. We conducted most of our site visit 

interviews in March and April 2006. 

We based our selection of QIOs on the following criteria: 

Site Visits 

Selection of QIOs 

• Number of nursing home beds in the state. We divided the states into

three groups of 17 states each based on the number of nursing home beds 

at the beginning of the 7

th SOW (2002). We over-sampled states with high 

numbers of nursing home beds by selecting one state with a low number 

of beds, one state with a medium number, and three states with a high 

number. 

• Evaluation score for the nursing home component of the 7th SOW 

relative to scores of other QIOs. We divided the states into three 

groups of 17 based on the QIOs’ evaluation scores for the 7th SOW. To help

us identify the possible determinants of scores, we selected more states at 

each end of the spectrum than in the middle: two states with scores in the 

bottom third, one state with a score in the middle third, and two states 

with scores in the top third. 

 1

To assist in the development of our site visit interview protocols, we interviewed 

personnel from three additional QIOs (Massachusetts, Rhode Island, and Washington) and 

staff from one nursing home in each of four other states (Maryland, Massachusetts, New 

Hampshire, and Virginia).

Quality Improvement Organizations
Appendix I: Scope and Methodology 

• State survey performance of homes selected for intensive 

assistance relative to homes not selected. We also considered the 

extent to which the homes selected for intensive assistance by a given QIO 

at the beginning of the 7

th SOW differed from the homes that were not 

selected, in terms of serious deficiencies cited on state surveys (both the 

proportion of homes cited in each group and the average number of 

serious deficiencies per home). We chose one QIO that selected worse 

homes, three QIOs that selected homes that were neither better nor worse, 

and one QIO that selected better homes. 

• Presence of a state-sponsored nursing home quality improvement 

program. At the time we selected QIOs for site visits, we were aware of 

four states that had state-sponsored quality improvement initiatives in 

place during the 7th SOW.2

 To learn more about these efforts and how they 

interacted with and compared with efforts by the QIOs, we included one 

state (Florida) with its own quality improvement initiative.3

 After we made 

our selection, we learned that another state we had selected (Maine) had a 

state-sponsored quality improvement program. 

• QIO participation in the Collaborative Focus Facilities project.

CMS has funded QIOs to conduct several special studies with nursing 

homes, including one in which the 17 participating QIOs each worked 

intensively with up to five nursing homes identified by their state survey 

agencies as having significant quality problems. To learn more about the 

challenges involved in working with low-performing homes, we selected

two states whose QIOs participated in this project. 

• Census region. We selected states from four different regions of the 

country: Northeast, Midwest, South, and West. 

Using these criteria, we selected the following five states: Colorado, 

Florida, Iowa, Maine, and New York. Together these states represented 

15 percent of nursing home beds nationwide at the beginning of the 7th

SOW (2002).

 2

The four states were Florida, Maryland, Texas, and Washington. We subsequently learned 

that four other states, (Maine, Michigan, Missouri, and North Carolina) also had statesponsored quality improvement programs. 

3

We contacted officials of programs in six states: Florida, Maryland, Michigan, North 

Carolina, Texas, and Washington. 

Page 56 GAO-07-373 Quality Improvement Organizations
Appendix I: Scope and Methodology 

Overall, we interviewed staff from 32 nursing homes in nine states. To 

assist in the development of our site visit protocols, we interviewed staff 

from 4 homes in four states. During the site visits to five states, we 

interviewed staff from 28 nursing homes. In each state, we interviewed 

staff from 4 to 8 nursing homes that received intensive assistance from the 

QIO, for a total of 28 homes in these five states. The number of homes we

selected in each of the five states visited varied depending on the number 

of homes the QIO was expected to select for intensive assistance, an

expectation based on the number of homes in the state. Specifically, we 

selected either four homes or 7 percent of the maximum number of homes 

that each of the five QIOs was expected to assist intensively, whichever 

was greater.4

We chose homes on the basis of four characteristics: number of serious 

deficiencies in the standard state survey at the beginning of the 7th SOW 

(2002), improvement in QM scores during the 7th SOW, distance from the 

QIO (in order to include homes that were more difficult for QIOs to visit), 

and urban versus rural location. Specifically, we sought to include (1) at 

least one home that had one or more serious deficiencies and that finished 

in the top third of the intensively assisted homes in their state in terms of 

improvement on QM scores, and (2) at least one home that had one or 

more serious deficiencies and that finished in the bottom third of the 

intensively assisted homes in their state in terms of improvement on QM 

scores. For the remaining homes, we sought a group whose state survey 

deficiency levels and QM improvement scores were representative of the 

range among intensive participants in their state. However, the 

experiences of this sample of 32 homes cannot be generalized to the entire 

group of homes that received intensive assistance from the QIOs 

nationwide.

In each state we also interviewed officials from three stakeholder groups: 

(1) the state survey agency; (2) the local affiliate of the American Health 

Care Association, which generally represents for-profit homes; and (3) the 

local affiliate of the American Association of Homes and Services for the

Aging, which represents not-for-profit homes. 

Selection of Nursing 

Homes 

Selection of Stakeholders 

 4

QIOs working in states with at least 100 nursing homes were expected to target 10 to 

15 percent of all homes in the state for intensive assistance. In the state we selected that 

had the highest number of homes (Florida), 7 percent of the homes in the state equaled 

approximately 8 homes. 

Page 57 GAO-07-373 Quality Improvement Organizations
Appendix I: Scope and Methodology 

To assess the characteristics of the nursing homes that were selected by 

the QIOs for intensive assistance from among the homes that volunteered, 

we analyzed 3 years of standard state survey data on deficiencies cited at

nursing homes and compared the results for homes that were assisted 

intensively with results for homes that were not; we used this information 

to address our first objective.5

 The analysis involved three steps: 

1. identifying nursing homes that had three standard state surveys from 

1999 through 2002; 

2. ranking nursing homes in each state in each year, based on the number 

of serious and other deficiencies, and then classifying homes as 

consistently low-, moderately, or high-performing; and 

3. identifying on a nationwide and state-by-state basis any statistically 

significant differences between homes selected and not selected by the

QIO, in terms of the proportion of low-, moderately, or high-performing 

nursing homes. 

To identify homes whose performance was consistently lower or higher 

than other homes in their state prior to the selection of homes by the 

QIOs, we included in our analysis only homes for which we were able to

identify three standard surveys from January 1, 1999, through November 1, 

2002. Using the state survey calendar year summary files for 1999 through

2002 for the 50 states and the District of Columbia, we obtained 3 years of

deficiency data from standard surveys for 16,303 homes.6

CMS classifies deficiencies according to their scope and severity. For each

of the three surveys, we ranked all of the nursing homes in each state 

based on the number of deficiencies in two categories: (1) actual harm or 

immediate jeopardy and (2) potential for more than minimal harm.7

Analysis of State 

Survey Data 

Identifying Homes with 

Three Standard Surveys 

Classifying Homes as 

Low-, Moderately, or 

High-Performing 

 5

This analysis drew on data from the On-line Survey, Certification, and Reporting system 

(OSCAR), a database maintained by CMS that compiles the results of every state survey 

conducted at Medicare- and Medicaid-certified facilities nationwide. 

6

We eliminated from the analysis 1,946 homes that had a standard survey in the year prior 

to November 1, 2002, but for which we were unable to identify two additional surveys

during the period we specified. The homes that we eliminated represented a larger 

proportion of the group of homes not selected by the QIOs (11.8 percent) than of the group 

of homes that were selected by the QIOs (3.4 percent). 

7

CMS defines immediate jeopardy as actual or potential for death/serious injury. 

Page 58 GAO-07-373 Quality Improvement Organizations
Appendix I: Scope and Methodology 

Deficiencies in the first category are considered serious deficiencies. We

gave more weight to the serious deficiencies by sorting the homes first on 

the number of deficiencies in the first category and then on the number of 

deficiencies in the second category. Homes with the same number of 

deficiencies in each category were assigned the same rank. Based on these 

rankings, we identified homes in the bottom and top quartile in each state 

in each survey.8

We classified homes as low-performing if they ranked in the bottom 

quartile in the most recent of the three surveys and in at least one of the 

two preceding surveys. We classified homes as high-performing if they 

ranked in the top quartile in the most recent of the three surveys and in at 

least one of the two preceding surveys. We classified homes as moderately 

performing if they did not meet the criteria for inclusion in either the lowor high-performing group. Of the 16,303 homes with three standard state 

surveys during the period we specified, we classified 15 percent as lowperforming, 65 percent as moderately performing, and 20 percent as highperforming. 

To assess the stability of our categorization of homes as low- (or high-) 

performing, we ran a logistic regression model to predict the probability of

a home being categorized as low- (or high-) performing in the most recent 

of the three surveys given its categorization in the two prior surveys. The 

regression results showed that homes that were categorized as low- (or 

high-) performing in one survey were significantly more likely to be 

categorized as low- (or high-) performing in the other surveys as well. 

 8

Because homes with the same number of deficiencies were assigned the same rank, in 

some cases the top and bottom quartiles included more than 25 percent of the homes in the 

state. We based our classification of homes on their performance level relative to other 

homes in the state to take into account the inconsistency in how states conduct surveys, a

problem we have reported on since 1998. An alternative approach, which would not take 

into account the inconsistency in how states conduct surveys, would be to classify homes 

based on the absolute number of deficiencies they had received⎯for example, to classify 

all homes with five or more serious deficiencies as low-performing homes. For data on

inconsistencies, see GAO-06-117 and GAO-07-241. 

Page 59 GAO-07-373 Quality Improvement Organizations
Appendix I: Scope and Methodology 

Our final step was to determine, on both a nationwide and state-by-state 

basis, whether there was a statistically significant difference in the 

proportion of (1) low-performing homes, (2) moderately performing 

homes, and (3) high-performing homes in the group assisted intensively by

the QIOs compared with the group not assisted intensively.9

To gather information about the characteristics of the QIOs, including 

their process for selecting homes for intensive assistance from the pool of 

volunteers and the interventions they used, on July 19, 2006, we launched 

a two-part Web-based survey of QIOs in all 50 states and the District of 

Columbia; we used this information to address objectives one and two.10

We achieved a 100 percent response rate. The first part of the survey 

gathered information about the primary personnel who worked with 

nursing homes during the 7th SOW, including information about their 

employment with the QIO, and their relevant credentials and experience.

11

The second part of the survey gathered information on a range of other 

topics, including information about stakeholder involvement with the QIO, 

recruitment and selection of nursing homes for intensive assistance, 

interventions used with intensive participants, interventions used with 

homes statewide, and QIOs’ communication with CMS. We specifically

inquired about QIOs’ use of six interventions: (1) mailings, faxes, and emails; (2) conferences; (3) small group meetings; (4) conference calls and 

video or Web conferences with multiple homes; (5) telephone 

conversations with individual homes; and (6) on-site visits.12 We asked 

QIOs to rank and provide information on the two interventions they relied 

on most to assist homes statewide and on the three interventions they 

Determining Statistically 

Significant Differences 

between Homes Assisted 

Intensively and Homes Not 

Assisted Intensively by the 

QIOs 

Web-Based Survey of 

QIOs 

 9

We used the Satterthwaite t-test because it does not require the variances of the two 

groups to be equal. We rejected the null hypothesis that the proportions of two groups

were equal when the p-value from the Satterthwaite t-test was less than 0.05. 

10We asked the QIOs to complete a separate survey for each state in which they worked 

during the 7th SOW.

11We defined primary personnel as employees, subcontractors, or consultants who worked 

with nursing homes or provided direct oversight of those individuals, excluding 

administrative support staff and individuals who worked less than 20 percent of a full-time 

work week on the nursing home component. 

12QIOs were also given the option of specifying other interventions they used. 

Page 60 GAO-07-373 Quality Improvement Organizations
Appendix I: Scope and Methodology 

relied on most to assist homes in the intensive participant group.13 We also 

asked QIOs to rank the effectiveness of the interventions they used and to 

identify the interventions they would use if they could do the 7

th SOW over 

again. 

 13Because QM improvement among intensive participants constituted the largest part of the 

QIOs’ contract evaluation score, we asked QIOs to rank and provide detailed information

on a greater number of interventions for intensive participants than for statewide 

participants. 

Page 61 GAO-07-373 Quality Improvement Organizations
Appendix II: Publicly Reported Quality Appendix II: Publicly Reported Quality Measures 

Measures 

In November 2002, CMS began a national Nursing Home Quality Initiative

that included the development of QMs that would be publicly reported on

the CMS Web site called Nursing Home Compare. CMS has continued to 

refine the QMs and, as shown in table 4, has dropped some QMs and added 

others. 

Table 4: QMs as of November 2002 and as of February 2007 

QM QM as of November 2002 QM as of February 2007 

Chronic care QM

Decline in activities of daily living 3 3

Pressure ulcers 3

Pressure ulcers

a

3

Pressure ulcers in high-risk residents 3

Pressure ulcers in low-risk residents 3

Inadequate pain management 3 3

Physical restraints 3 3

Infections 3

Weight loss 3

Urinary tract infection 3

Catheter insertion 3

Depression 3

Bowel or bladder control in low-risk residents 3

Bedfast 3

Worsening ability to move about room 3

Administration of influenza vaccination during flu season 3

Assessment for and administration of pneumococcal vaccination 3

Post-acute-care QM

Failure to improve and manage delirium 3 3

Failure to improve and manage delirium (facility-adjusted)a3

Inadequate pain management 3 3

Improvement in walking 3

Pressure ulcers 3

Administration of influenza vaccination during flu season 3

Assessment for and administration of pneumococcal vaccination 3

Source: CMS. 

a

Facility-level risk adjustment was intended to take into account the fact that some homes may admit 

frailer, sicker residents, or may specialize in a particular area of care that may account for a larger

proportion of residents for a particular measure. CMS reported the delirium measure both with and 

without facility adjustment. 

Page 62 GAO-07-373 Quality Improvement Organizations
Appendix III: Comments from the Centers for Appendix III: Comments from the Centers for Medicare & Medicaid Services 

Medicare & Medicaid Services 

Page 63 GAO-07-373 Quality Improvement Organizations
Appendix III: Comments from the Centers for 

Medicare & Medicaid Services 

Page 64 GAO-07-373 Quality Improvement Organizations
Appendix III: Comments from the Centers for 

Medicare & Medicaid Services 

Page 65 GAO-07-373 Quality Improvement Organizations
Appendix III: Comments from the Centers for 

Medicare & Medicaid Services 

Page 66 GAO-07-373 Quality Improvement Organizations
Appendix IV: GAO Contact and Staff Acknowledgments 

Page 67 GAO-07-373 

Appendix IV: GAO Contact and Staff 

Acknowledgments 

Kathryn G. Allen, (202) 512-7118 or allenk@gao.gov

In addition to the contact named above, Walter Ochinko, 

Assistant Director; Nancy Fasciano; Sara Imhof; Elizabeth T. Morrison; 

Colbie Porter; and Andrea Richardson made key contributions to this 

report. 

GAO Contact 

Acknowledgments 

Quality Improvement Organizations
Related GAO Products Related GAO Products 

Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not 

Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241. 

Washington, D.C.: March 26, 2007. 

Nursing Homes: Despite Increased Oversight, Challenges Remain in 

Ensuring High-Quality Care and Resident Safety. GAO-06-117. 

Washington, D.C.: December 28, 2005. 

Nursing Home Deaths: Arkansas Coroner Referrals Confirm Weaknesses 

in State and Federal Oversight of Quality of Care. GAO-05-78. 

Washington, D.C.: November 12, 2004. 

Nursing Home Fire Safety: Recent Fires Highlight Weaknesses in 

Federal Standards and Oversight. GAO-04-660. Washington D.C.: 

July 16, 2004.

Nursing Home Quality: Prevalence of Serious Problems, While 

Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561. 

Washington, D.C.: July 15, 2003. 

Nursing Homes: Public Reporting of Quality Indicators Has Merit, but 

National Implementation Is Premature. GAO-03-187. Washington, D.C.: 

October 31, 2002. 

Nursing Homes: Quality of Care More Related to Staffing than Spending. 

GAO-02-431R. Washington, D.C.: June 13, 2002. 

Nursing Homes: More Can Be Done to Protect Residents from Abuse. 

GAO-02-312. Washington, D.C.: March 1, 2002. 

Nursing Homes: Federal Efforts to Monitor Resident Assessment Data 

Should Complement State Activities. GAO-02-279. Washington, D.C.: 

February 15, 2002. 

Nursing Homes: Sustained Efforts Are Essential to Realize Potential of 

the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.: 

September 28, 2000. 

Nursing Home Care: Enhanced HCFA Oversight of State Programs 

Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.: 

November 4, 1999. 

Page 68 GAO-07-373 Quality Improvement Organizations
Related GAO Products 

Nursing Home Oversight: Industry Examples Do Not Demonstrate That 

Regulatory Actions Were Unreasonable. GAO/HEHS-99-154R. Washington, 

D.C.: August 13, 1999. 

Nursing Homes: Proposal to Enhance Oversight of Poorly Performing 

Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999. 

Nursing Homes: Complaint Investigation Processes Often Inadequate to 

Protect Residents. GAO/HEHS-99-80. Washington, D.C.: March 22, 1999. 

Nursing Homes: Additional Steps Needed to Strengthen Enforcement of 

Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.: 

March 18, 1999. 

California Nursing Homes: Care Problems Persist Despite Federal and

State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998. 

(290500) Page 69 GAO-07-373 Quality Improvement Organizations
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER