Nursing Homes: Federal Actions Needed to Improve Targeting and Evaluation of Assistance by Quality Improvement Organizations
Published: 2007-05-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2002, CMS contracted with Quality Improvement Organizations (QIO) to help nursing homes address quality problems such as pressure ulcers, a deficiency frequently identified during routine inspections conducted by state survey agencies. CMS awarded $117 million over a 3-year period to the QIOs to assist all homes and to work intensively with a subset of homes in each state. Homes' participation was voluntary. To evaluate QIO performance, CMS relied largely on changes in homes' quality measures (QM), data based on resident assessments routinely conducted by homes. GAO assessed QIO activities during the 3-year contract starting in 2002, focusing on (1) characteristics of homes assisted intensively, (2) types of assistance provided, and (3) effect of assistance on the quality of nursing home care. GAO conducted a Web-based survey of all 51 QIOs, visited QIOs and homes in five states, and interviewed experts on using QMs to evaluate QIOs.
section_title: What GAO Found
paragraphs: Although more homes volunteered to work with the QIOs than CMS expected them to assist intensively, QIOs typically did not target their assistance to the low-performing homes that volunteered. Most QIOs' primary consideration in selecting homes was their commitment to working with the QIO. CMS did not specify selection criteria for intensive participants but contracted with a QIO that developed guidelines encouraging QIOs to select committed homes and exclude those with many survey deficiencies or QM scores that were too good to improve significantly. Consistent with the guidelines, few QIOs targeted homes with a high level of survey deficiencies, and eight QIOs explicitly excluded these homes. GAO's analysis of state survey data confirmed that selected homes were less likely than other homes to be low-performing in terms of identified deficiencies. Most state survey and nursing home trade association officials interviewed by GAO believed QIO resources should be targeted to low-performing homes. QIOs were provided flexibility both in the QMs on which they focused their work with nursing homes and in the interventions they used. Most QIOs chose to work on chronic pain and pressure ulcers, and most used the same interventions⎯conferences and distribution of educational materials⎯to assist homes statewide. The interventions used to assist individual homes intensively varied and included on-site visits, conferences, and small group meetings. Just over half the QIOs reported that they relied most on on-site visits to assist intensive participants. Sixty-three percent said such visits were their most effective intervention. Of the 15 QIOs that would have changed the interventions used, most would make on-site visits their primary intervention. Homes indicated that they were less satisfied with the program when their QIO experienced high staff turnover or when their QIO contact possessed insufficient expertise. Shortcomings in the QMs as measures of nursing home quality and other factors make it difficult to measure the overall impact of the QIOs on nursing home quality, although staff at most of the nursing homes GAO contacted attributed some improvements in the quality of resident care to their work with the QIOs. The extent to which changes in homes' QM scores reflect improvements in the quality of care is questionable, given the concerns raised by GAO and others about the validity of the QMs and the reliability of the resident assessment data used to calculate them. In addition, quality improvements cannot be attributed solely to the QIOs, in part because the homes that volunteered and were selected for intensive assistance may have differed from other homes in ways that would affect their scores; these homes may also have participated in other quality improvement initiatives. Ongoing CMS evaluation of QIO activities for the contract that began in August 2005 is being hampered by a 2005 Department of Health and Human Services decision that QIO program regulations prohibit QIOs from providing to CMS the identities of homes being assisted intensively.

section_title: Letter
paragraphs: The federal government plays a major role in the financing and oversight  of nursing home care for individuals who are aged or disabled. Medicare  and Medicaid payments for nursing home services totaled $67 billion in  2004, including a $46 billion federal share. The Centers for Medicare &  Medicaid Services (CMS) defines quality standards that the nation’s  approximately 16,400 nursing homes must meet to participate in the  Medicare and Medicaid programs and contracts with state survey agencies  to assess homes’ compliance through routine inspections, known as  standard surveys, and through complaint investigations. Under 3-year  contracts beginning in August 2002 and referred to as the 7th statement of  work (SOW), CMS directed Medicare Quality Improvement Organizations  (QIO) to work with nursing homes to improve the quality of care provided  to residents in the 50 states, the District of Columbia, and the territories.
As a condition of their contracts, QIOs were required to provide   (1) information to all Medicare- or Medicaid-certified nursing homes in  each state about systems-based approaches to improving resident care and  clinical outcomes and (2) intensive assistance to a subset of each state’s  homes, typically 10 to 15 percent, that were selected by the QIOs from  among those homes that volunteered for assistance.
In a series of congressionally requested studies undertaken since 1998, we  have reported on the unacceptably high proportion of nursing homes  providing poor care to residents. Based in part on our recommendations,  CMS has undertaken a number of enforcement initiatives to encourage  nursing home compliance with federal quality standards, including  improved oversight by both state survey agencies and CMS, and tougher  enforcement measures to ensure that homes correct deficiencies and  maintain compliance with federal standards. For example, CMS expanded  its Special Focus Facility program in which state agencies survey selected  homes more frequently and terminate those that fail to improve  significantly within 18 months.
CMS’s decision to offer direct assistance to nursing homes that volunteer  to work with QIOs represents a new strategy in the effort to help address  long-standing quality problems in nursing homes. To evaluate QIO  performance in improving nursing home care, CMS relied primarily on  changes in nursing homes’ quality measures (QM) during the contract  period. QMs are numeric measures derived from resident assessments— known as the minimum data set (MDS)—that nursing homes routinely  conduct and submit to CMS. The QMs were developed to permit  comparisons across nursing homes of the quality of care provided to  residents and have been publicly reported on CMS’s Nursing Home  Compare Web site since 2002.
In 2005, CMS renewed the QIO contracts, including the nursing home  component, for another 3-year period, with a budget of $96 million to  assist nursing homes. Given the decision to continue the program, you  asked us to assess QIOs’ work with nursing homes for the 7th SOW,  covering the period August 2002 through January 2006. For this report, we  assessed (1) characteristics of nursing homes the QIOs assisted  intensively, (2) the assistance the QIOs provided to nursing homes, and   (3) the effect of QIOs’ assistance on the quality of nursing home care.
To assess the characteristics of nursing homes that the QIOs selected to  assist intensively from among the homes that volunteered, we analyzed  CMS data on deficiencies cited in standard surveys of nursing homes and  compared the results for homes assisted intensively by the QIOs with  homes that were not assisted intensively. To gather information about the  QIOs’ criteria for selecting homes for intensive assistance, we fielded a  Web-based survey to the 37 organizations that held the 51 QIO contracts in  the states and the District of Columbia, achieving a 100 percent response  rate. To determine the type of quality improvement assistance QIOs  provided to nursing homes, our Web-based survey collected data on the  types, frequency, and perceived effectiveness of specific interventions  used to assist homes both statewide and in the group assisted intensively;  interventions included activities such as on-site visits, mailings, and  conferences. To gather more detailed information about QIOs’ work with  nursing homes, we conducted site visits to five states—Colorado, Florida,  Iowa, Maine, and New York—where we interviewed QIO personnel, staff  from nursing homes that had received intensive assistance, and key  stakeholders. The five states accounted for 15 percent of nursing home  beds nationwide in 2002 and represented a range in terms of such  characteristics as number of nursing home beds, region of the country,  and QIOs’ performance on the nursing home component in the 7th SOW. In  the five states, we interviewed staff from 28 nursing homes⎯4 to 8 per  state; in addition, we interviewed staff from 4 homes in four other states  for a total of 32 homes. We sought to select a group of homes that  represented a range in terms of state survey deficiencies, improvement in  QM scores during the 7th SOW, distance from the QIO, and urban versus  rural location. However, the experiences of the 32 homes in our sample  cannot be generalized to all homes that received intensive assistance from  the QIOs nationwide. To assess the effect of QIOs’ assistance on nursing  home quality, we reviewed performance requirements in the QIO contracts  for both the 7th and the 8th SOWs; reports on QIOs’ work with nursing  homes, including the 2006 report on the QIO program by the Institute of  Medicine (IOM); and other documents. We also conducted interviews  with nursing homes, CMS officials, officials from state quality assurance  programs and state MDS accuracy review programs, and experts on the  nursing home QMs and the MDS data on which they are based. We  conducted our review from October 2005 through May 2007 in accordance  with generally accepted government auditing standards. (For a more  detailed description of our scope and methodology, see app. I.)
subsections: section_title: Background
paragraphs: Beginning in the late 1990s, CMS took steps to broaden the mechanisms in  place intended to help ensure that nursing home residents receive quality  care. To augment the periodic assessment of homes’ compliance with  federal quality requirements, CMS contracted for the development of QMs  and tasked QIOs with providing assistance to homes to improve quality.  CMS used QMs both to provide the public with information on nursing  home quality of care and to help evaluate QIO efforts to address quality-of- care issues, such as pressure ulcers. During the 7th SOW, organizations  other than QIOs were also working with nursing homes to improve quality.
subsections: section_title: Indicators of Nursing Home Quality
paragraphs: Two indicators used by CMS to assess the quality of care that nursing  homes provide to residents are (1) deficiencies identified during standard  surveys and complaint investigations and (2) QMs. Both indicators are  publicly reported on CMS’s Nursing Home Compare Web site.
subsections: section_title: Survey Deficiencies
paragraphs: Under contract with CMS, state agencies conduct standard surveys to  determine whether the care and services provided by nursing homes meet  the assessed needs of residents and whether nursing homes are in  compliance with federal quality standards. These standards include  preventing avoidable pressure ulcers; avoiding unnecessary restraints,  either physical or chemical; and averting a decline in a resident’s ability to  perform activities of daily living, such as toileting or walking. During a  standard survey, a team that includes registered nurses spends several  days at a home reviewing the quality of care provided to a sample of  residents. States are also required to investigate complaints filed against  nursing homes by residents, families, and others. Complaint investigations  are less comprehensive than standard surveys because they generally  target specific allegations raised by the complainants.
Any deficiencies identified during standard surveys or complaint  investigations are classified according to the number of residents  potentially or actually affected (isolated, pattern, or widespread) and their  severity (potential for minimal harm, potential for more than minimal  harm, actual harm, or immediate jeopardy). Deficiencies cited at the actual  harm and immediate jeopardy level are considered serious and could  trigger enforcement actions such as civil money penalties. We have  previously reported on the considerable interstate variation in the  proportion of homes cited for serious care problems, which ranged during  fiscal year 2005 from 4 percent of Florida’s 691 homes to 44 percent of  Connecticut’s 247 homes. We reported that such variability suggests  inconsistency in states’ interpretation and application of federal  regulations; in addition, both we and CMS have found that state surveyors  do not identify all serious deficiencies.
section_title: Quality Measures
paragraphs: QMs are relatively new indicators of nursing home quality. Although  survey deficiencies have been publicly reported since 1998, CMS did not  begin posting QMs on its Nursing Home Compare Web site until November  2002. QMs are derived from resident assessments known as the MDS that  nursing homes routinely collect on all residents at specified intervals.  Conducted by nursing home staff, MDS assessments cover 17 areas, such  as skin conditions, pain, and physical functioning.
In developing QMs, CMS recognized that any publicly reported indicators  must pass a rigorous standard for validity and reliability. In October 2002,  we reported that national implementation of QMs was premature because  of validity and reliability concerns. Valid QMs would distinguish between  good and poor care provided by nursing homes; reliable QMs would do so  consistently. One of our main concerns about publicly reporting QMs was  that the QM scores might be influenced by other factors, such as residents’  health status. As a result, the specification of appropriate risk adjustment  was a key requirement for the validity of any QMs. Risk adjustment is  important because it provides consumers with an “apples-to-apples”  comparison of nursing homes by taking into consideration the  characteristics of individual residents and adjusting the QM scores  accordingly. For example, a home with a disproportionate number of  residents who are bedfast or who present a challenge for maintaining an  adequate level of nutrition—factors that contribute to the development of  pressure ulcers—may have a higher pressure ulcer score. Adjusting a  home’s QM score to fairly represent to what extent a home does or does  not admit such residents is important for consumers who wish to compare  one home to another. Appendix II lists the 10 QMs initially adopted and  publicly reported by CMS—6 applicable to residents with chronic care  problems (long-stay residents) and 4 applicable to residents with post- acute-care needs (short-stay residents).
MDS data are self-reported by nursing homes, and ensuring their accuracy  is critical for establishing resident care plans, setting nursing home  payments, and publicly reporting QM scores. In February 2002, we  concluded that CMS efforts to ensure the accuracy of MDS data, which are  used to calculate the QMs, were inadequate because the agency relied too  much on off-site review activities by its contractor and planned to conduct  on-site reviews in only 10 percent of its data accuracy assessments,  representing fewer than 200 of the nation’s then approximately 17,000  nursing homes. Although we recommended that CMS reorient its review  program to complement ongoing state MDS accuracy efforts as a more  effective and efficient way to ensure MDS data accuracy, CMS disagreed  and continued to emphasize off-site reviews.
section_title: Evolution of the QIO Program and the Quality Improvement Process
paragraphs: Over the past 24 years, the QIO program has evolved from a focus on  quality assurance in the acute care setting to quality improvement in a  broader mix of settings, including physician offices, home health agencies,  and nursing homes. Established by the Peer Review Improvement Act of  1982 and originally known as Peer Review Organizations (PRO), QIOs  initially focused on ensuring minimum standards by conducting  retrospective hospital-based utilization reviews that looked for  inappropriate or unnecessary Medicare services. According to the 2006  IOM report, as it became apparent that standards of care themselves  required attention, QIOs gradually shifted from retrospective case reviews  to collaboration with providers to improve the overall delivery of care—a  shift consistent with transformational goals set by CMS’s Office of Clinical  Standards and Quality, which oversees the QIO program.
In contrast to enforcing standards, quality improvement tries to ensure  that organizations have effective processes for continually measuring and  improving quality. The goal of quality improvement is to close the gap  between an organization’s current performance and its ideal performance,  which is defined by either evidence-based research or best practices  demonstrated in high-performing organizations.  According to the quality  improvement literature, successful quality improvement requires a  commitment on the part of an organization’s leadership and active  involvement of the staff. The 2006 IOM report notes that QIOs rely on  various mechanisms to promote quality improvement, including one-on- one consulting and collaboratives. While the former provides direct and  specialized attention, the latter relies on workshops or meetings that offer  opportunities for providers to share experiences and best practices.  Quality improvement often relies on the involvement of early adopters of  best practices—providers who are highly regarded as leaders and can help  convince others to change—for the diffusion of best practices. Key tools  for quality improvement include (1) root cause analysis, a technique used  to identify the conditions that lead to an undesired outcome;   (2) instruction on how to collect, aggregate, and interpret data; and   (3) guidance on bringing about, sustaining, and diffusing internal system  redesign and process changes, particularly those related to use of  information technology for quality improvement. Quality improvement  experts also emphasize the importance of protecting the confidentiality of  provider information, not only to protect the privacy of personal health  information but also to encourage providers to evaluate their peers  honestly and to prevent the damage to providers’ reputations that might  occur through the release of erroneous information.
Section 1160 of the Social Security Act provides that information collected  by QIOs during the performance of their contract with CMS must be kept  confidential and may not be disclosed except in specific instances; it  provides the Secretary of HHS with some discretion to determine  instances under which QIO information may be disclosed. The regulations  implementing the statute limit the circumstances under which confidential  information obtained during QIO quality review studies, including the  identities of the participants of those studies, may be disclosed by the QIO.
During the 7th SOW, QIOs submitted a list of nursing home participants to  CMS as a contract deliverable.
section_title: CMS Contract Funding and Requirements
paragraphs: During the 7th SOW, CMS awarded a total of $117 million to QIOs to  improve the quality of care in nursing homes in all 50 states, the District of  Columbia, and the territories. The performance-based contracts for QIO  assistance to nursing homes delineated broad expectations regarding QIO  assistance to nursing homes, provided deadlines for completing four  contract deliverables, and laid out criteria for evaluating QIO  performance. For contracting purposes, the QIOs were divided into three  groups with staggered contract cycles. The four contract deliverables,  however, were all due on the same dates, irrespective of the different  contract cycles. The contracts also required QIOs to work with a QIO  support contractor tasked to provide guidelines for recruiting and  selecting nursing homes as intensive participants, train QIOs in standard  models of quality improvement assistance, and provide tools and  educational materials, as well as individualized consultation if needed, to  help QIOs meet contractual requirements. QIOs and nursing homes were  also involved in other quality improvement special studies with budgets  separate from the QIO contracts for the 7th SOW. These studies varied  greatly in terms of length, the clinical issue(s) covered, the number of  QIOs involved, and the characteristics of the nursing homes that  participated. Figure 1 shows the 7th SOW contract cycles, deliverables for  the nursing home component, and the duration of the special studies.
Contract funding. The $117 million awarded to QIOs to improve the  quality of care in nursing homes during the 7th SOW included   (1) $106 million awarded to provide statewide and intensive assistance to  homes, (2) $5.6 million awarded to selected QIOs to conduct eight special  studies focused on nursing home care, and (3) $5.3 million awarded to the  QIO that served as the support contractor for the nursing home  component. CMS allocated a specific amount for each component of the  contracts, but allowed QIOs to move funds among certain components.  Just over half of the 51 QIOs did not spend all of the funds allocated to the  nursing home component, but on average the QIOs overspent the budget  for the nursing home work by 3 percent.
Contract requirements for quality improvement activities. Per the  contracts for the 7th SOW, QIOs were required to provide (1) all Medicare-  and Medicaid-certified homes with information about systems-based  approaches to improving patient care and clinical outcomes, and   (2) intensive assistance to a subset of homes in each state. The contracts  directed QIOs working in states with 100 or more nursing homes to target  10 to 15 percent of the homes for intensive assistance. Figure 2 illustrates  that QIOs provided two levels of assistance—statewide and intensive— and that homes’ participation was either nonintensive or intensive.  Intensive participants received both statewide and intensive assistance.  Selection of intensive participants from among the nursing homes that  volunteered was at the discretion of each QIO, but the SOW required the  QIO support contractor (the Rhode Island QIO) to provide guidelines and  criteria for QIOs to use in determining which homes to select.  Participation in the program was voluntary, and QIOs were prohibited  from releasing the names of participating nursing homes except as  permitted by statute and regulation.
Under the contracts, the quality improvement assistance provided by QIOs  focused on areas related to eight chronic care and post-acute-care QMs  publicly reported on the CMS Nursing Home Compare Web site. QIOs  were required to consult with relevant stakeholders and select from three  to five of the eight QMs on which QIOs’ quality improvement efforts would  be evaluated (see table 1). Intensive participant homes were also required  to select one or more QMs on which to work with the QIO. Although they  could select one QM, they were encouraged to select more than one.
To improve QM scores, QIOs were expected to develop and implement  quality improvement projects focused on care processes known to  improve patient outcomes in a manner that utilized resources efficiently  and reduced burdens on providers. The QIO support contractor developed  a model for QIOs to facilitate systems change in nursing homes. This  model emphasized the importance of QIOs’ statewide activities to form  and maintain partnerships, conduct workshops and seminars, and  disseminate information on interventions to improve quality. For intensive  participants, the model emphasized conducting one-on-one quality  improvement assistance as well as conferences and small group meetings.  According to contract language, QIOs were expected to coordinate their  projects with other stakeholders that were working on similar  improvement efforts or were interested in teaming with the QIO. But  ultimately, each QIO determined for itself the type, level, duration, and  intensity of support it would offer to nursing homes.
Evaluation of QIO contract performance. CMS evaluated QIOs’  performance on the nursing home component of the contract using  nursing home provider satisfaction with the QIO, QM improvement among  intensive participants, and QM improvement statewide (see fig. 3).  Nursing home provider satisfaction was assessed by surveying all  intensive participants and a sample of nonintensive participants around  the 28th month of each 36-month contract. CMS expected at least 80  percent of respondents to report that they were either satisfied or very  satisfied.
QIOs were also expected to achieve an 8 percent improvement in QM  scores among both intensive participants and homes statewide. The term  improvement was defined mathematically to mean the relative change in  the QM score from when it was measured at baseline to when it was  remeasured. The statewide improvement score included the QM  improvement scores for intensive participants averaged with those of  nonintensive participants.
CMS established two scoring thresholds for the contracts that  encompassed scores from all components of the SOW. If a QIO scored  above the first threshold it was eligible for a noncompetitive contract  renewal; if it scored below that threshold, it was eligible for a competitive  renewal only upon providing information pertinent to its performance to a  CMS-wide panel that decided whether to allow the QIO to bid again for  another QIO contract.
CMS contract monitoring. CMS formally evaluated each QIO at months  9 and 18 of the 7th SOW. If CMS found that a QIO failed to meet contract  deliverables or appeared to be in danger of failing to meet contract goals,  it could require the QIO to make a performance improvement plan or  corrective plan of action to address any barriers to the QIOs successfully  fulfilling contract requirements. In addition, CMS reviewed materials such  as QIOs internal quality control plans, which were intended to help QIOs  monitor their own progress and to document any project changes made to  improve their performance.
section_title: Other Nursing Home Quality Improvement and Assurance Initiatives
paragraphs: The QIO program operated in the context of other quality improvement  initiatives sponsored by federal and state governments and nursing home  trade associations. As stated earlier, CMS funded a number of special  nursing home studies involving subsets of the QIOs and nursing homes,  which addressed a variety of clinical quality-of-care issues and which are  summarized in figure 1. Under CMS’s Special Focus Facility program, state  survey agencies were required to conduct enhanced monitoring of nursing  homes with histories of providing poor care. During the 7th SOW, CMS  revised the method for selecting homes for the Special Focus Facility  program to ensure that the homes performing most poorly were included;  increased the minimum number of homes that must be included, from a  minimum of two per state to a minimum of up to six, depending on the  number of homes in the state; and strengthened enforcement for those  nursing homes with an ongoing pattern of substandard care. In addition,  concurrent with the 7th SOW, at least eight states had programs that  provided quality assurance and technical assistance to nursing homes in  their states. These programs varied in terms of whether they were  voluntary or mandatory, which homes received assistance, the focus and  frequency of the assistance provided, and the number and type of staff  employed.
In addition to government-operated quality improvement initiatives, three  long-term care professional associations joined together in July 2002 to  implement the Quality First Initiative. This initiative was based on a  publicly articulated pledge on the part of the long-term care profession to  establish an environment of continuous quality improvement, openness,  and leadership in participating homes.
section_title: QIOs Generally Had a Choice among Homes That Volunteered but Did Not Target Assistance to Low-Performing Homes
paragraphs: Although QIOs generally had a choice of homes to select for intensive  assistance because more homes volunteered than CMS expected QIOs to  assist, QIOs typically did not target the low-performing homes that  volunteered. Most QIOs reported in our Web-based survey that they did  not have difficulty recruiting homes, and their primary consideration in  selecting homes from the pool of volunteers was that the homes be  committed to working with the QIOs. In the 7th SOW, CMS did not specify  recruitment and selection criteria for intensive participants, leaving the  development of guidelines to the QIO support contractor, which  encouraged QIOs to select homes that seemed committed to quality  improvement and to exclude homes with a high number of survey  deficiencies, high management turnover, or QM scores that were too good  to improve significantly. Our analysis of state survey data showed that,  nationwide, intensive participants were less likely to be low-performing  than other homes in their state in terms of the number, scope, and severity  of deficiencies for which they were cited in standard surveys from 1999  through 2002. This result may reflect the nature of the homes that  volunteered for assistance, the QIOs’ selection criteria, or a combination  of the two. The stakeholders we interviewed—including officials of state  survey agencies and nursing home trade associations—generally believed  QIOs’ resources should be targeted to low-performing homes.
subsections: section_title: QIOs Generally Had a Choice of Which Nursing Homes to Assist Intensively
paragraphs: Most QIOs had a choice of which nursing homes to assist intensively, as  more homes volunteered than the QIOs could receive credit for serving  under the terms of their contracts. Of the 38 QIOs in states with 100 or  more homes, which were expected to work intensively with 10 to   15 percent of the homes, 30 reported in our Web-based survey that more  than 15 percent of homes expressed interest in intensive assistance, and   8 reported that more than 30 percent of homes expressed interest. Most  QIOs selected about as many intensive participants as needed to get the  maximum weight for the intensive participant element of their contract  evaluation score. Nationwide, the intensive participant group included just  under 15 percent (2,471) of the 16,552 homes identified by CMS at the  beginning of the 7th SOW.
Most QIOs—82 percent of the 51 that responded to our survey—reported  that it was not difficult to recruit the target number of homes for intensive  assistance; the remainder reported that it was difficult (12 percent) or very  difficult (4 percent) to recruit enough volunteers. Among the QIOs we  interviewed, personnel at two that reported difficulties recruiting homes  cited homes’ lack of familiarity with QIOs as a barrier. Personnel at one of  these two QIOs commented that the QIO’s first task was to build trust  among homes and address confusion about its role, as some homes  thought the QIO was a regulatory authority charged with investigating  complaints and citing homes for deficiencies.
section_title: Commitment to Working with QIOs Was QIOs’ Primary Consideration in Selecting Homes from among Those That Volunteered
paragraphs: QIOs that responded to our Web-based survey almost uniformly cited  homes’ commitment to working with them as a key consideration in  choosing among the homes that volunteered to be intensive participants.  QIOs had wide latitude in choosing among homes because CMS did not  specify the characteristics of the homes they should recruit or select,  leaving it to the QIO support contractor to provide voluntary guidelines.  The QIO support contractor developed guidelines based on input from a  variety of sources, including QIOs that worked with nursing homes during  the 6th SOW. Issued at the beginning of the 7th SOW, the guidelines  emphasized the important role the selected homes would play in the QIOs’  contract performance and encouraged QIOs to select homes that  demonstrated a willingness and ability to commit time and resources to  quality improvement. The QIO support contractor also encouraged QIOs to  exclude homes with a high number of survey deficiencies, high  management turnover, and QM scores that were too good to improve  significantly. With respect to homes’ survey histories, the QIO support  contractor reasoned that homes with a high number of deficiencies might  be more focused on improving their survey results than on committing  time and resources to quality improvement projects. For example, the care  areas in which a home was cited for deficiencies might not correspond  with any of the eight QMs to which CMS limited the QIOs’ quality  improvement activities (see table 1). In fact, the quality of care area in  which homes were most frequently cited for serious deficiencies in  surveys in 2006 was the provision of supervision and devices to prevent  accidents, which does not have a corresponding QM.
Consistent with the guidelines, 76 percent of the 41 QIOs that reported in  our Web-based survey their considerations in selecting homes for the  intensive participant group ranked homes’ commitment as their primary  consideration. Nearly all QIOs ranked commitment among their top three  considerations (see fig. 4).
Homes’ QM scores were also an important consideration for QIOs. QIOs  were particularly interested in including homes that had poor QM scores  in areas where the QIO planned to focus or in assembling a group of  homes that represented a mix of QM scores. With respect to homes’  overall QM scores, the QIOs that responded to our survey were more likely  to seek homes with moderate overall scores than homes with poor or good  overall scores. Similarly, personnel at most QIOs we contacted gave  serious consideration to homes’ QM scores, looking for homes that  appeared to need help and could demonstrate improvement. For example,  personnel at one QIO said that they tended to select homes whose QM  scores were worse than the statewide average; personnel at another QIO  said that this QIO selected homes with scores it thought could be  improved, eliminating homes with either very high or very low scores.  Personnel at one QIO acknowledged that some QIOs might “cherry pick”  homes in this way in order to satisfy CMS contract requirements but  argued that it was not possible for QIOs to predict which homes would  improve the most.
QIOs generally gave less consideration to the number of deficiencies  homes had on state surveys than to their QM scores. However, the 17 QIOs  that ranked survey deficiencies among their top three considerations in  our survey were more likely to seek homes with deficiencies in areas  where they planned to focus or homes with an overall low level (number  and severity) of survey deficiencies than homes with an overall high level.  Moreover, of the 33 QIOs that reported in our survey systematically  excluding some of the homes that volunteered from the intensive  participant group, nearly one-quarter (8) excluded homes with a high  number of survey deficiencies. None excluded homes with a low number  of survey deficiencies.
Personnel at the QIOs we interviewed offered several reasons for  excluding homes with a high number of survey deficiencies from the  intensive participant group. Personnel at several QIOs concurred with the  QIO support contractor that such homes were likely to be too consumed  with correcting survey issues to focus on quality improvement. Personnel  at one QIO suggested that the kind of assistance very poor-performing  homes need⎯help improving the basic underlying structures of  operation⎯was not the kind the QIO offered. Personnel at some QIOs said  they considered not just the level of deficiencies for which homes were  cited on recent surveys but the level over multiple years or the specific  categories of deficiencies. For example, personnel at one QIO said that  although the QIO excluded homes with long-standing histories of poor  performance, it actively recruited homes that had performed poorly only  on recent surveys. Personnel at another QIO stated that their concern was  to avoid homes with competing priorities. This QIO sought to include  homes with deficiencies in the areas it planned to address but to exclude  homes with deficiencies in other areas on the assumption that these  homes would not benefit from the assistance it planned to offer. Personnel  we interviewed at two QIOs said that they worked with some extremely  poor-performing homes but did not include them on the official list of  intensive participants submitted to CMS; personnel at one of these QIOs  explained that they did not want to be held responsible if these homes  were unable to improve.
section_title: QIOs Did Not Target Intensive Assistance to Low-Performing Homes
paragraphs: Our analysis of homes’ state survey histories from 1999 through 2002  indicates that QIOs did not target intensive assistance to homes that had  performed poorly in state surveys. Nationwide, the homes in the intensive  participant group were less likely than other homes in their state to be  low-performing in terms of the number, scope, and severity of deficiencies  for which they were cited in surveys during that time frame. As illustrated  in figure 5, the intensive participant group included proportionately more  homes in the middle of the performance spectrum and proportionately  fewer at either end. Although our analysis focused on survey deficiencies  rather than QMs, this result is generally consistent with the results of our  Web-based survey concerning QIOs’ use of QM scores as selection criteria,  which showed that QIOs were more likely to select homes with moderate  overall scores than homes with poor or good overall scores and to seek a  mix of performance levels among homes in the group. However, not  knowing the composition of the pool of homes that volunteered for  assistance, we cannot determine whether the composition of the intensive  participant group⎯in particular, the disproportionately low number of  low-performing homes in the group⎯was a function of which homes  volunteered, which homes the QIOs selected from among the volunteers,  or a combination of both factors.
On a state-by-state basis, none of the QIOs targeted assistance to low- performing homes by including proportionately more such homes in the  intensive participant group. Most QIOs (33 of 51) worked intensively with  homes that were generally representative of the range of homes in their  state in terms of performance on state surveys from 1999 through 2002. In  these states, there was no significant difference in the proportion of high-,  moderately, or low-performing homes among intensive participants  compared with nonintensive participants. However, 18 QIOs worked  intensively with a group that differed significantly from other homes in the  state: 8 of these QIOs worked with proportionately fewer low-performing  homes, 5 worked with proportionately more moderately performing  homes, and 9 worked with proportionately fewer high-performing homes.
section_title: Stakeholders Often Stated QIOs Should Target Intensive Assistance to Low-Performing Homes
paragraphs: Stakeholders we interviewed who expressed an opinion about the homes  QIOs should target for intensive assistance—11 of the 16 we interviewed— almost uniformly said that the QIOs should concentrate on low-performing  homes. Survey officials in one state suggested that QIOs should use state  survey data to assess homes’ need for assistance because these data are  often more current than QM data. In their emphasis on low-performing  homes, stakeholders echoed the views expressed in the 2006 IOM report,  which recommended that QIOs give priority for assistance to providers,  including nursing homes, that most need improvement. Other stakeholder  suggestions regarding the homes QIOs should target are listed in table 2.  Because the QIOs were required to protect the confidentiality of QIO  information about nursing homes that agreed to work with them,  stakeholders were generally not informed which homes were receiving  intensive assistance. One exception was in Iowa, where the QIO obtained  consent from the selected homes to reveal their identities.
Several stakeholders said that low-performing homes can improve with  assistance. However, one suggested that QIOs might have to adapt their  approach⎯for example, by streamlining their training⎯to avoid  overburdening homes that are struggling with competing demands.  Another agreed that low-performing homes can benefit from working with  a QIO but added that real improvements in the quality of care in these  homes would require attention to staffing, turnover, pay, and recognition  for staff. The results of one special study funded by CMS during the time  frame of the 7th SOW supported stakeholders’ contention that low- performing homes can improve, although the improvements documented  in these homes cannot be definitively attributed to the QIOs. In this study,  known as the Collaborative Focus Facility project, 17 QIOs worked  intensively with one to five low-performing homes identified in  consultation with the state survey agency. According to a QIO  assessment of the project, the participating homes showed improvement  in areas related to the assistance provided by the QIO in terms of both the  number of serious state survey deficiencies for which they were cited and  their QM scores. CMS officials pointed out that these improvements were  hard-won: one-third of the homes that were asked to participate in the  Collaborative Focus Facility project refused, and those that did participate  required more effort and resources from the QIOs to improve than did  other homes assisted by the QIOs.
Overall, CMS has specifically directed only a small share of QIO resources  to low-performing homes. In the current contracts (the 8th SOW), CMS  required QIOs to provide intensive assistance to some “persistently poor- performing homes” identified in consultation with each state survey  agency. However, the number of such homes that the QIOs must serve is  small⎯ranging from one to three, depending on the number of nursing  homes in the state⎯and accounts for less than 10 percent of the homes  the QIOs are expected to assist intensively. Less than 17 percent of the 144  persistently poor-performing homes the QIOs selected in consultation with  state survey agencies to assist in the 8th SOW were also special focus  facilities in 2005 or 2006.
QIOs and stakeholders tended to disagree about whether participation in  the program should remain voluntary for all homes. QIO personnel we  interviewed who expressed an opinion generally supported voluntary  participation on the theory that homes that were forced to participate  would probably be less engaged and put forth only minimal effort.
Personnel at some QIOs that opposed mandatory participation suggested  that creating incentives for homes to improve their quality of care⎯for  example, through pay for performance⎯would increase homes’ interest in  working with the QIO. In contrast, most of the state survey agency and  trade association officials we interviewed who expressed an opinion about  the voluntary nature of the QIO program said that some homes should be  required to work with the QIO. Officials at one state survey agency pointed  out that the low-performing homes that really need assistance rarely seek  it; these officials believed that working with the QIO should be mandatory  for low-performing homes and voluntary for moderately to high- performing homes. Another state survey agency official recommended that  25 to 40 percent of the homes assisted intensively be chosen from among  the lower-performing homes in the state and required to work with the  QIO.
section_title: QIO Contract Flexibility Resulted in Variation in Assistance Provided to Intensive Participants
paragraphs: The 7th SOW contracts allowed QIOs flexibility in the QMs they focused on  and the interventions they used, and while the majority of QIOs selected  the same QMs and most used the same interventions to assist homes  statewide, the interventions for intensive participants and staffing to  accomplish program goals varied. Most QIOs and intensive participants  worked on the chronic pain and pressure ulcer QMs, but these were not  the QMs that some intensive participants believed matched their greatest  quality-of-care challenges. To assist all homes statewide, QIOs generally  relied on conferences and the distribution of educational materials. The  top three interventions for intensive participants included on-site visits   (87 percent), followed by conferences (57 percent), and small group  meetings (48 percent). According to nursing home staff we interviewed,  turnover and experience levels of the QIO personnel that provided them  assistance affected their satisfaction with the program and the extent of  their quality improvements.
subsections: section_title: Most Quality Improvement Efforts Focused on Chronic Pain and Pressure Ulcers
paragraphs: Under the terms of the contracts, both QIOs and intensive participants  could select QMs to focus on, but most chose to work on two of the same  QMs. While nearly all QIOs chose to work statewide on chronic pain and  pressure ulcers, they differed on their selection of additional QMs (see   fig. 6). QIO personnel we interviewed told us they based the choice of QMs  for their statewide work on input from stakeholders and nursing homes or  QM data. For example, some stakeholders told us that specific QMs  selected addressed existing long-term care challenges and were ones on  which homes in the state ranked below the national average. Personnel  from two QIOs said they selected QMs based on input from homes in their  state about which QMs the homes were interested in working on, and  personnel from several QIOs stated that they selected QMs on which their  homes could improve. Personnel from one QIO specifically mentioned that  they selected QMs related to the quality of life for nursing home residents.
Most intensive participants worked on a subset of the QMs selected by  their QIO—chronic pain and pressure ulcers (see fig. 6). The degree to  which intensive participants knew they had a choice of QMs was unclear.  Of the 14 intensive participants we interviewed that commented on  whether they had a choice, 9 said that they did. Staff from these homes  generally reported having selected QMs related to clinical issues on which  they could improve. However, the remaining 5 homes indicated that their  QIO selected the QMs on which they received assistance. Most of these 5  homes’ staff reported that they would have preferred to work on different  QMs from the list of eight that are publicly reported on the CMS Nursing  Home Compare Web site or other clinical issues that reflect their greatest  quality-of-care challenges.
section_title: Statewide Interventions Less Variable Than Those for Intensive Participants
paragraphs: The terms of the QIO contract with CMS allowed QIOs to determine the  kinds of quality improvement interventions they offered to homes, and  those selected by QIOs were consistent with an approach recommended  by the QIO support contractor: QIOs generally relied most on conferences  and the distribution of educational materials to assist homes statewide and  on on-site visits to assist intensive participants. However, there was a  greater variety of interventions frequently relied on to assist intensive  participants. In general, QIOs reported that the interventions they relied on  most were also the most effective for improving the quality of resident  care.
subsections: section_title: Statewide Assistance
paragraphs: Almost three-quarters of the QIOs included conferences among the two  interventions they relied on most to provide quality improvement  assistance to homes statewide (see fig. 7). These QIOs held an average of  nine conferences over the course of the 7th SOW, typically in various cities  throughout the state to accommodate homes from different regions. Sixty- eight percent of these QIOs reported that more than half the homes in  their state sent staff to least one conference, and 16 percent of QIOs  reported that all or nearly all homes did so. QIO personnel reported  holding conferences to educate homes on quality improvement, discuss  the relationship between MDS assessments and the QMs, and provide QM- specific clinical information or best practices. Some conferences included  presentations by state or national experts.
Almost three-quarters of QIOs also ranked the distribution of educational  materials by mail, fax, or e-mail among their top two statewide  interventions. Thirty-two percent of these QIOs sent materials four or  fewer times per year, whereas 27 percent sent materials 12 or more times  per year to all or nearly all homes in the state. For the QIOs we  interviewed, these materials included newsletters, QM-specific tools or  clinical information related to the QMs, and QM data progress reports for  the home or state, overall.
Almost one-third of the QIOs (31 percent) reported that the type or  intensity of interventions they used to assist homes statewide changed  over the course of the 7th SOW. For example, two QIOs reported that they  concentrated much of their statewide efforts into the first half of the   3-year period; one QIO specifically reported doing so in the interest of  ensuring that any improvements in QMs were reflected in its evaluation  scores, which, as specified by the contract, were calculated near the mid- point of the contract cycle. In contrast, five other QIOs reported that they  increased the intensity of their statewide work over time, in some cases  concentrating on homes whose performance was lagging.
For the 8th SOW, CMS has focused resources on assistance to intensive  participants by eliminating expectations for improvements in QMs  statewide. However, the contracts still contain statewide elements,  including a requirement to promote QM target-setting.
section_title: Intensive Assistance
paragraphs: Fifty-one percent of QIOs ranked on-site visits as their most relied on  intervention with intensive participants and 87 percent ranked it among  their top three interventions (see fig. 8). Both the number of visits and the  time spent at sites varied considerably. The median number of visits was 5  but ranged from 1 to 20. Sixty-eight percent of QIOs that included on-site  visits among their top three interventions spent an average of 1 to 2 hours  at sites each time they visited, while 20 percent spent 3 to 4 hours. QIOs  that ranked on-site visits as their number one intervention made more and  longer visits to intensive participants than did QIOs that ranked them  lower. When surveyed about a typical on-site visit, the majority of QIO  respondents reported that they generally reviewed the homes’ QM data,  provided education or best practices, or both. Approximately one-third of  QIOs that conducted site visits indicated that they had discussions with  the home about their systems or processes for care, homework  assignments, or quality improvement activities. Some QIOs (26 percent)  reported that they conducted team-building exercises with the staff when  on site.
QIOs varied in the interventions they used in addition to on-site visits, with  conferences, small group meetings emphasizing peer-to-peer learning, and  telephone calls being the three others most commonly used. QIOs that  included conferences among their three most relied on interventions  typically held between 3 and 10 during the 7th SOW, but as with site visits,  some variation existed. After conferences, QIOs were most likely to rely  on small group meetings and telephone calls with individual homes. Nearly  half of the QIOs ranked these two interventions among their three most  relied on, but few ranked them highest. The number of homes that  attended small group meetings varied. An average of 6 to 10 homes was  most common, but one-fifth of QIOs reported having an average of 20 or  more homes represented at each meeting. As for telephone calls, the vast  majority of QIOs (92 percent) that ranked these calls among their three  most relied on interventions called all or nearly all of their intensive  participants, typically on a monthly basis.
Our interviews with QIOs and intensive participant homes suggested that  the small group meetings they held generally followed a similar format,  while telephone calls were used for a variety of purposes. For example,  personnel from several QIOs and intensive participant homes told us that  their small group meetings generally included a formal presentation on the  QMs or related best practices, as well as a time for less formal information  sharing and peer-to-peer learning among the attendees. Participants  shared stories about their successes and challenges conducting quality  improvement. Personnel from a number of QIOs told us they used  telephone calls to follow up after visits or meetings, discuss the homes’  progress on quality improvement, and to decide on next steps.
Almost two-thirds of QIOs indicated that the type or intensity of  interventions for intensive participants varied over time. Of these QIOs,   36 percent reduced the intensity of their interventions (substituting small  group meetings or telephone calls for on-site visits), while 33 percent did  the reverse (in some cases increasing the frequency of on-site visits or  substituting small group meetings for conferences to increase  participation). For example, personnel from a few QIOs told us that while  they initially relied on on-site visits to begin the quality improvement  process, they came to rely more on telephone calls or on small group  meetings where intensive participants could share their success stories or  ways to overcome barriers to quality improvement. Seventy-nine percent  of QIOs surveyed varied their interventions based on the needs of  intensive participants. Thus, personnel from three QIOs told us they  realized that some homes did not need frequent on-site visits, while others  needed more. The two specific needs that QIOs cited most as having  precipitated changes in their interventions were nursing home staffing  changes and turnover (23 percent) and poorer performance by some  homes relative to others (15 percent). A few QIOs also noted that  interventions varied by the preferences or levels of readiness and  participation of the homes with which they were working.
section_title: QIO and Nursing Home Perspectives on the Interventions
paragraphs: Most QIOs we surveyed deemed conferences the most effective statewide  intervention and on-site visits the most effective intensive intervention;  intensive participant homes we interviewed also found these interventions  valuable. For homes statewide, most QIOs (54 percent) reported that  conferences were their most effective intervention, followed by  distribution of educational materials and on-site visits. Of the one-quarter  of QIOs that reported they would change their statewide approach, the  largest proportion (46 percent) would make conferences their primary  intervention. Staff from several nursing homes we interviewed tended to  concur that conferences were valuable aspects of the program because  conferences included expert presenters, energized or motivated attendees,  and were free.
For intensive participants, most QIOs (63 percent) deemed on-site visits  their most effective intervention, followed by conferences and small group  meetings. Of the 15 QIOs that said they would change their approach with  these homes, most (60 percent) would make on-site visits their primary  intervention, while fewer would rely on small group meetings,  conferences, and other interventions. One QIO began conducting on-site  visits and small group meetings when it became apparent that telephone  calls were less productive than had been anticipated because of the  difficulty of getting the right staff on the telephone at the right time, the  lack of speaker phones at many homes, and the lack of staff engagement  on the phone. Staff from a number of nursing homes we interviewed  agreed that visits by QIO personnel were helpful. Some homes indicated  that having someone from the QIO visit the home maximized the number  of staff that could take advantage of the quality improvement training  offered. Furthermore, the on-site visits were motivating and kept staff on  track with quality improvement efforts. Regarding small group meetings,  staff we interviewed from a few homes stated that meeting with staff from  other homes helped validate their own efforts or facilitated the sharing of  materials and experiences. Staff from one nursing home specifically  reported that they were disappointed not to have formally participated in  small group meetings with other facilities in the state.
Homes also found particular types of assistance less helpful. Some homes’  staff reported that they did not feel they had the time or the staff necessary  to complete some of the homework assignments expected of them, such  as conducting chart reviews. Staff at some homes stated that the QIO  provided quality improvement information with which they were already  familiar.
section_title: QIO Staffing and Turnover Influenced Intensive Participants’ Satisfaction with Program
paragraphs: Our interviews with nursing home staff who worked intensively with the  QIOs indicated that homes’ satisfaction with the program was influenced  by the training and experience of the primary QIO personnel who served  as their principal contact with the QIOs, as well as by turnover among  these individuals during the course of the 7th SOW.
When a home’s principal contact with the QIO was a nurse or someone  with long-term care or quality improvement experience, nursing home  staff tended to report that this person possessed the knowledge and skills  necessary to help them improve the quality of care in their home.  Interviewees also spoke appreciatively of QIO personnel who were  knowledgeable, motivating, and kept them on track with their efforts.  However, when the QIO principal contact lacked these qualifications or  characteristics, he or she was perceived as unable to effectively address  clinical topics with staff. Staff at one home said explicitly that working  with an experienced nurse, instead of a social worker who seemed to lack  knowledge of long-term care, would have led to greater improvement in  clinical quality.
The extent to which QIO primary personnel had the training or experience  that homes considered important varied. More than half (58 percent) of  the primary QIO personnel who worked with nursing homes during the 7th  SOW were trained in nursing, and 42 percent held an advanced degree.  Nationwide, 27 percent of the primary personnel who worked with nursing  homes had less than 1 year of long-term care experience, while 30 percent  had more than 10 years of such experience. Just over half of primary QIO  personnel (54 percent) working with nursing homes had 4 or fewer years  of quality improvement experience. Nine percent of QIO personnel had  more than 10 years’ experience in both long-term care and quality  improvement. Few of the personnel working with nursing homes during  the 7th SOW gained any of their experience working for the QIO during the  6th SOW because there was little overlap in personnel across the two  periods.
Our interviews with intensive participants suggested that turnover among  primary QIO personnel lowered nursing homes’ satisfaction with the  program. Our survey revealed that turnover was particularly high at some  QIOs. At 24 QIOs, 25 percent or more of primary personnel who worked  with nursing homes did so for less than half of the 36-month contract, and  at 6 QIOs, the proportion was 50 percent or more. When a nursing home’s  principal contact with a QIO changed frequently, nursing home staff we  interviewed reported that they received inconsistent assistance that was  disruptive to their efforts to improve quality of care. For example, one  nursing home we visited had four different principal contacts over the  course of the 7th SOW and found this to be frustrating because, just as they  were establishing a relationship with a contact, the contact would leave.  Staff at another home complained that their interaction with QIO primary  personnel turned out not to be the learning experience that the staff  thought it would be.
Staffing levels for the nursing home component also varied among QIOs.  As would be expected, given the wide variation in the number of nursing  homes per state, the number of full-time-equivalent (FTE) staff working  with nursing homes varied across the QIOs, ranging from 0.50 to 12.  However, the ratio of QIO staff FTEs to intensive participant homes also  showed significant variation. On average, the ratio was about 1 to 14; but  for at least 9 QIOs, the ratio of staff FTEs to homes was 1 to 10 or fewer,  and for at least 8 QIOs, the ratio was 1 to 18 or more.
section_title: QIOs’ Impact on Quality Is Not Clear, but Staff at Homes We Contacted Attributed Some Improvements to QIOs
paragraphs: Although the QIOs’ impact on the quality of nursing home care cannot be  determined from available data, staff we interviewed at most nursing  homes attributed some improvements in the quality of resident care to  their work with the QIOs. Nursing homes’ QM scores generally improved  enough for the QIOs to surpass by a wide margin the modest contract  performance targets set by CMS; however, the overall impact of the QIOs  on the quality of nursing home care cannot be determined from these data  because of the shortcomings of the QMs as measures of nursing home  quality and because confounding factors make it difficult to attribute  quality improvements solely to the QIOs. Multiple long-term care  professionals we interviewed indicated that QMs should not be used in  isolation to measure quality improvement, but combined with other  indicators, such as state survey data. Moreover, the effectiveness of the  individual interventions QIOs used to assist homes also cannot be  evaluated with the available data. CMS planned to enhance evaluation of  the program during the 8th SOW, but a 2005 determination by HHS’s Office  of General Counsel that the QIO program regulations prohibit QIOs from  providing to CMS the identities of the homes they are assisting has  hampered the agency’s efforts to collect the necessary data. Although the  impact of the QIOs on the quality of nursing home care is not known, over  two-thirds of the 32 nursing homes we interviewed attributed some  improvements in care to their work with the QIOs.
subsections: section_title: All QIOs Met Modest Targets for QM Improvement, but the Impact of the QIOs on the Quality of Nursing Home Care Cannot Be Determined
paragraphs: Although all of the QIOs met the modest targets CMS set for QM  improvement among homes both statewide and in the intensive participant  group, the impact of the QIOs on the quality of nursing home care cannot  be determined because of the limitations of the QMs and because  improvements cannot be definitively attributed to the QIOs. The  effectiveness of the specific interventions used by the QIOs to assist  homes also cannot be evaluated with the available data.
subsections: section_title: All QIOs Met CMS’s Modest Targets for Improvement in Nursing Home QMs
paragraphs: All QIOs met the CMS performance targets for the nursing home  component of the 7th SOW. In addition to receiving an overall passing score  for this component, nearly all QIOs surpassed expectations for each of the  three elements that contributed to the overall score: provider satisfaction,  improvement in QM scores among intensive participants, and  improvement in QM scores among homes statewide. In fact, about two- thirds of the QIOs achieved at least five times the expected 8 percent  improvement among intensive participants, and nearly half achieved at  least twice the expected 8 percent improvement statewide.
CMS officials stated that the targets set for the nursing home component  of the contract were purposely modest. Because the 7th SOW marked the  first time all QIOs were required to work with nursing homes on quality  improvement, and little data existed to predict how much improvement  could be expected, CMS deliberately designed performance criteria to  limit QIOs’ chances of failing. For example, expectations for  improvements in QM scores were set no higher for intensive participants  than for homes statewide. In addition, CMS modified the evaluation plan  so that if an intensive participant worked on more than one QM, the QM  that improved least was dropped before the home’s average improvement  was calculated. CMS officials told us that, based in large part on QIOs’  performance in the 7th SOW, the agency raised its expectations for the 8th  SOW. For example, QIOs are required to work with most intensive  participants on four specified QMs and to achieve an improvement rate of  15 to 60 percent, depending on the QM and the homes’ baseline scores. In  addition, CMS will no longer drop the QM that improved least when  calculating homes’ average improvement.
section_title: CMS’s Use of QMs to Evaluate QIO Performance Is Problematic
paragraphs: Long-term care experts we interviewed generally agreed that CMS’s use of  QMs to evaluate nursing home quality—and by extension, QIOs’  performance⎯is problematic because of unresolved issues related to the  QMs and the MDS data used to calculate them.
QMs. As we reported in 2002, the validity of the QMs CMS proposed to  publicly report in November 2002 was unclear. Although the validation  study commissioned by CMS found that most of the publicly reported QMs  were valid and reflected the quality of care delivered by facilities, long- term care experts have criticized the study on several grounds. For  example, a 2005 report concluded that (1) the statistical criteria for the  validity assessments were not stringent and (2) the researchers did not  attempt to determine whether QMs were associated with quality of care at  the resident level. As a result, it is not clear whether a resident who  triggers a QM (e.g., is assessed as having his or her pain managed  inadequately) is actually receiving poor care. The lack of correlation  among the QMs⎯a home may score well on some QMs and poorly on  others⎯also calls into question their validity as measures of overall  quality. Since 2002, CMS has removed or replaced 5 of the original 10  QMs⎯including some of those on which the QIOs were evaluated during  the 7th SOW—to address limitations in the QMs, such as reliability and  measurement problems. (See app. II for a list of the QMs as of November  2002 and February 2007).
Risk adjustment also impacts the validity of QMs. There is general  recognition that some QMs should be adjusted to account for the  characteristics of residents. However, there is disagreement about which  QMs to adjust, what risk factors should be used, or how the adjustment  should be made. For example, one expert we interviewed suggested that in  many cases pressure ulcers start in hospitals; the pressure ulcer QM does  not account for the origin of ulcers. Another expert highlighted the  difficulty of making an appropriate adjustment—noting, for example, that  improperly risk-adjusting the pressure ulcer QM could mask poor care that  contributed to the development of ulcers.
MDS. We have also previously reported concerns about MDS  reliability⎯that is, the consistency with which homes conduct and code  the assessments used to calculate the QMs. CMS awarded a contract for  an MDS accuracy review program in 2001 but revamped the program in  2005, near the end of the QIOs’ 7th SOW, acknowledging weaknesses— mainly its reliance on off-site, rather than on-site, accuracy and  verification reviews—that we had previously identified. Some states that  sponsor on-site MDS accuracy reviews continue to report troubling rates  of errors in the data. For example, officials of Iowa’s program reported an  average MDS error rate of approximately 24 percent in 2005.
Our interviews with long-term care experts and nursing home staff  suggested that the chronic pain QM⎯which was selected as a focus of  quality improvement work by many QIOs and intensive participant nursing  homes⎯may be particularly vulnerable to error in the underlying MDS  data. Possible sources of error are systematic differences in the extent to  which facilities identify and assess residents in pain and  misunderstandings about how to accurately code MDS questions specific  to pain. For example, staff from two nursing homes told us that their pain  management QM scores improved after staff realized that they had been  mistakenly coding residents as having pain even though their pain was  successfully managed. Moreover, experts we interviewed noted that  higher-quality homes may have worse pain QM scores because they do a  better job of identifying and reporting pain in residents.
The use of MDS data to measure the quality of care in nursing homes is  also problematic because the MDS was not designed as a quality  measurement tool and does not reflect advances in clinical practice. CMS  is updating the MDS now to address these limitations. For example,  instead of asking homes to classify the severity of a pressure ulcer on the  basis of a four-stage system, the draft MDS now under review includes a  measurement tool intended to more accurately classify the severity of a  pressure ulcer. In addition, facilities are asked to indicate whether the  pressure ulcer developed at the facility or during a hospitalization. CMS  does not yet have an official release date for the revised MDS but  anticipates that all validation and reliability testing will be completed by  December 2007.
Other Measures of Quality. Multiple long-term care professionals we  interviewed, including stakeholders and experts on quality measurement,  recommended both that the QMs undergo continued refinement and that  they not be used in isolation to assess the quality of care in nursing homes.  They suggested a number of other sources of information as alternatives  or complements to QMs for measuring quality. For example, a  representative of the National Quality Forum (NQF), a group with which  CMS contracted to provide recommendations on quality measures for  public reporting, stated that experts do not consider the QMs sufficient in  themselves to rate homes and that the other quality markers⎯such as  perceptions of care by family members, residents, and staff; state survey  data; and resident complaints⎯also provide useful information about  quality of care. Other long-term care professionals we interviewed  suggested these and other measures, including nursing home staffing  levels and staff turnover and retention rates.
section_title: Influence of Other Factors on Nursing Home Quality Makes It Difficult to Evaluate QIOs’ Impact
paragraphs: Factors such as the existence of other quality improvement efforts make it  difficult to evaluate QIOs’ work with nursing homes and attribute quality  improvement solely to QIOs. In an assessment of the QIO program during  the 7th SOW, CMS and QIO officials acknowledged this difficulty. The  assessment found that intensive participants improved more than  nonintensive participants on all five QMs studied, and for each QM,  intensive participants that worked on the QM improved more than  intensive participants that did not. However, the authors noted that these  results could not be definitively attributed to the efforts of the QIOs  because improvements may have been influenced by a variety of factors,  including preexisting differences between intensive participants and  nonintensive participants; public reporting of the QMs, which may have  focused homes’ attention on improving these measures; and other quality  improvement efforts to which homes may have been exposed. As noted  earlier in this report, these other efforts included, but were not necessarily  limited to, initiatives sponsored by state governments, nursing home trade  associations, and CMS. While these other efforts varied considerably in the  intensity of technical assistance offered⎯ranging from a trade association- sponsored program that homes characterized as essentially signing a  quality improvement pledge, to state-sponsored programs that involved  on-site visits by experienced long-term care nurses who provided best- practice guidelines, educational materials, and clinical tools⎯the fact that  the efforts were present made it impossible to attribute quality  improvements solely to the QIOs.
In its 2006 report on all aspects of the QIO program, IOM highlighted  similar shortcomings in previous studies of the QIO program and called for  more systematic and rigorous evaluations. IOM concluded that although  the QIOs may have contributed to improvements in the quality of care, the  existing evidence was inadequate to determine the extent of their  contribution. In its response to the IOM study, CMS acknowledged the  need to strengthen its methods of evaluating the program and outlined  plans to convene an evaluation expert advisory panel to make  recommendations on the framework for the next contracts (the 9th SOW,  which will begin in 2008). CMS also stated that it will collect information  during the 8th SOW that will allow it to control for differences in motivation  between intensive and nonintensive participants but did not specify the  nature of this information. Subsequently, HHS’s Office of General  Counsel determined that QIO program regulations prohibited QIOs from  providing to CMS the identities of intensive participants. CMS officials  acknowledged that this prohibition posed a considerable challenge to their  evaluation plans and said that as a short-term solution the agency might  contract with one of the QIOs to evaluate the program, with the possible  stipulation that the findings be verified by an independent auditor.
section_title: CMS Data Are Too Limited to Evaluate Effectiveness of Specific QIO Interventions
paragraphs: CMS collected little information about the specific interventions QIOs  used to assist nursing homes and acknowledged that the information it did  have was not sufficiently comprehensive or consistent to be used to  evaluate the interventions’ effectiveness.  In general, CMS’s oversight of  QIOs’ work on the nursing home component consisted of ensuring that the  QIOs produced the reports and deliverables specified in the contracts and  appeared on track to meet performance targets.
CMS’s primary source of data about QIOs’ interventions was the monthly  activity reports the QIOs were required to submit through the Program  Activity Reporting Tool (PARTner). In these reports, QIOs were to  document the specific interventions they provided to each home, using  such activity codes as “on-site support” and “stand-alone workshops on  quality improvement.” However, with only seven activity codes for QIOs to  choose from, the level of detail in these reports was low. For example, the  same code would be used for a full-day visit as for an hour visit. Moreover,  because QIOs were not expected to enter any code more than once per  month for a home, a code for on-site support could indicate a single visit  or multiple visits. The system also captured no information about the  content of visits or other interventions. From the perspective of the QIOs,  the system was of limited use: More than half of the 52 QIOs surveyed by  IOM rated PARTner fair or poor in terms of both value and ease of use.  Staff at one QIO we interviewed reported using tracking systems they  developed themselves, rather than PARTner, to monitor their work.
CMS regional offices and the nursing home satisfaction survey gathered  some additional information about the interventions used by QIOs. The  CMS regional offices gathered information through telephone calls and  visits to the QIOs and by participating in quarterly conference calls during  which QIOs and CMS regional and central offices discussed issues related  to the nursing home component of the contract. The regional office staff  also reviewed information entered into the PARTner data system by QIOs,  but they focused their evaluations on QIO contract compliance and not on  the effectiveness of specific interventions because—as some regional staff  emphasized—the contracts were performance-based, and therefore it was  not their place to “micromanage” the QIOs or to advocate for or against  specific interventions. Feedback from nursing homes was gathered  through the nursing home satisfaction survey, conducted after the  midpoint of the contract cycle by a contractor for CMS. The survey  collected information about the frequency of, and homes’ satisfaction  with, a range of interventions, including on-site visits, training workshops,  one-on-one telephone calls, conference calls, one-to-one e-mails, and  broadcast e-mails. However, the survey collected no information about the  content of these interventions or the aspects that contributed to providers’  satisfaction or dissatisfaction.
In its 2006 report on the QIO program, IOM emphasized the need for CMS  to gather more information about specific interventions and noted that  CMS was uniquely positioned to determine which interventions lead to  high levels of quality improvement. The agency responded that it will  collect information during the 8th SOW to better explore the relationship  between the intensity of assistance provided by the QIO and the level of  improvement, but did not specify the type of information it will collect. As  of March 2007, CMS had not yet implemented a revamped PARTner  system. In addition, the agency cancelled its plans to conduct an initial  survey of nursing homes early in the contract period and now plans to  conduct only one, later in the contract period. CMS officials explained that  the delay and cancellation were due at least in part to the determination by  HHS’s Office of General Counsel that QIOs could not provide to CMS the  identities of intensive participants to CMS.
section_title: Homes That Received Intensive Assistance Generally Attributed Some Improvements in Quality of Care to Work with QIOs
paragraphs: Although the impact of the QIOs on the overall quality of nursing home  care cannot be determined, staff we interviewed at over two-thirds of the  32 nursing homes stated that they improved the care delivered to residents  as a result of working intensively with the QIOs. Staff at 23 of the 32  homes told us that they implemented new, or made changes to existing,  policies and procedures related to pain or pressure ulcers. Of the 23  nursing homes, staff from 21 stated that they changed the way they  addressed resident pain. In general, these changes involved implementing  pain scales or new assessment forms. Staff at some facilities noted that  working with the QIO heightened staff awareness of resident pain,  including awareness of cultural differences in the expression of pain. Staff  at 8 of the 23 nursing homes stated that they changed the way they  addressed pressure ulcers. In general, these 8 homes implemented new  assessment tools, changed assessment plans, or revised facility policies  using materials provided by the QIO. (Table 3 provides examples of  resident care improvements related to pain assessment and treatment and  pressure ulcers.) Staff at 13 of the 32 nursing homes stated that the  changes they made as a result of working with the QIOs were sustainable,  but staff from some nursing homes noted that staffing turnover at their  facilities could affect sustainability.
Of the 32 nursing homes we contacted, staff from 4 specifically stated that  working with the QIO did not change their quality of care. For example,  staff from one home stated that the QIO did not offer their facility any new  or helpful information and did not offer feedback on how the facility’s  processes could improve. Staff from another home reported that the  information provided by the QIO was on techniques their facility had  already implemented. Staff at a third home noted that while the QIO was a  good resource, the home could have done as much on its own, without  assistance from the QIO. Staff at three facilities, none of which reported  making any policy or procedural changes, said the facilities experienced  worse survey results while working with their QIO; staff from two of the  three reported being cited for quality deficiencies in the specific areas they  had been addressing with the QIO. Staff at one of these facilities believed  they were cited because their work with the QIO had made the surveyor  more aware of the facility’s problems in this area.
section_title: Conclusions
paragraphs: Although it is difficult to evaluate the impact of QIO assistance, the QIO  program does have the potential to help improve the quality of nursing  home care. CMS program improvements for the 8th SOW, such as the  agency’s decision to focus resources on intensive rather than statewide  assistance and its plans to improve evaluation, are positive steps that  could result in more effective use of available funds and provide more  insight into the program’s impact. Our evaluation of assistance provided  during the 7th SOW, however, raised two major questions about the future  focus, oversight, and evaluation of the QIO program, which we address  below.
Given the available resources, which homes and quality-of-care  areas should CMS direct QIOs to target for intensive assistance?  We found that QIOs generally did not target intensive assistance to homes  that performed poorly in state surveys, partly because of concerns about  the willingness and ability of such homes to simultaneously focus on  quality improvement and cooperate with the QIOs. However, the  Collaborative Focus Facility project during the 7th SOW demonstrated that  low-performing homes could improve their survey results and QM scores;  subsequently, CMS required that during the 8th SOW each QIO work with  up to three such homes—about 10 percent of the total number that QIOs  are expected to assist intensively. Stakeholders we interviewed believed  that even more emphasis should be placed on assisting low-performing  homes. We found that there was little overlap between homes that  participated in the QIO Collaborative Focus Facility project and in CMS’s  Special Focus Facility program, which is a program involving about 130  nursing homes nationwide that, on the basis of their survey results, receive  increased scrutiny and enforcement by state survey agencies. The limited  overlap suggests that each state has more than three low-performing  facilities that could benefit from QIO assistance.
Targeting assistance to low-performing homes could pose challenges given  the voluntary nature of the program—homes must agree to work with a  QIO. QIOs maintain that voluntary participation is critical to ensuring  homes’ commitment to the program. However, the risk in this approach is  that some of the homes that need help most will not get it. Indeed, in the  Collaborative Focus Facility project, some of the low-performing homes  that were asked to participate refused QIO assistance. In addition, QIOs  expended more resources working to improve these low-performing  homes than were required to assist better-performing homes. Thus,  increasing the number of low-performing homes QIOs are required to  assist above the small number mandated for the 8th SOW might necessitate  decreasing the total number of homes assisted. However, existing  resources might be maximized if QIOs worked with each home only on the  quality-of-care areas that pose particular challenges for that home.
Could interim steps be taken to improve oversight and evaluation  of QIOs’ work with nursing homes before the contracting cycle that  begins in August 2008? Currently, CMS collects data primarily on QIO  outcomes—specifically, changes in QM scores—and costs. CMS needs  more detailed data, particularly about the type and intensity of  interventions used to assist nursing homes, to improve its oversight and  evaluation of the QIO program. Without such data, CMS cannot hold QIOs  fully accountable for their performance under their contract with CMS.  Some evaluation activities are now scaled back or on hold because HHS  determined early in the 8th SOW that program regulations prohibited the  QIOs from providing to CMS the identities of the intensive participants.  Such a firewall presents a major impediment to improved oversight and  evaluation of the QIO program and prevented CMS from implementing  interim changes it planned to make. For example, for the 7th SOW, CMS  contracted for one nursing home satisfaction survey to be conducted near  the end of the contract period—too late to be of use in interim monitoring  of the QIOs’ performance. For the 8th SOW, CMS had planned to contract  for two surveys but was forced to cancel the one planned for early in the  contract period because it was unable to provide the names of intensive  participants to its survey contractor. Moreover, the lack of these data  would preclude CMS from independently verifying QIO compliance with  such contract requirements as the geographic dispersion of intensive  participants in each state.
CMS evaluated QIOs’ work with nursing homes primarily on the basis of  changes in QM scores; given the weaknesses of QM data, the current  reliance on these data appears unwarranted. While CMS actions to  improve the MDS instrument as a quality measurement tool are important,  the agency has not yet established an implementation date. Although  multiple long-term care professionals believe that multiple indicators of  quality, including deficiencies on homes’ standard and complaint surveys  and residents’ and family members’ satisfaction with care, should be used  to measure quality improvement, CMS is not currently drawing on these  data sources to evaluate QIOs’ efforts. Recognized shortcomings in these  other data sources—such as the understatement of survey deficiencies by  state surveyors—underscore the importance of using multiple data  sources to evaluate QIO outcomes.
section_title: Recommendations for Executive Action
paragraphs: To ensure that available resources are better targeted to the nursing  homes and quality-of-care areas most in need of improvement, we  recommend that the Administrator of CMS take the following two actions:    Further increase the number of low-performing homes that QIOs assist  intensively.
Direct QIOs to focus intensive assistance on those quality-of-care areas on  which homes most need improvement.
To improve monitoring of QIO assistance to nursing homes and to  overcome limitations of the QMs as an evaluation tool, we recommend  that the Administrator of CMS take the following three actions:    Revise the QIO program regulations to require QIOs to provide to CMS the  identities of the nursing homes they are assisting in order to facilitate  evaluation.
Collect more complete and detailed data on the interventions QIOs are  using to assist homes.
Identify a broader spectrum of measures than QMs to evaluate changes in  nursing home quality.
section_title: Agency Comments and Our Evaluation
paragraphs: We obtained written comments from CMS on our draft report. CMS  addressed three of our five recommendations. It concurred with two of the  three recommendations but did not specify how it would implement them,  and it continues to explore options for implementing the third  recommendation. Our evaluation of CMS’s comments follows the order we  presented each recommendation in the report. CMS’s comments are  included in app. III.
Further increase the number of low-performing homes that QIOs  assist intensively. CMS agreed with this recommendation but did not  specify a time frame for addressing it or indicate how many low- performing homes it will expect QIOs to assist in the future. Although our  report focused on the most recently completed contract period (the 7th  SOW), we acknowledged that in the current contract period, CMS required  QIOs to provide intensive assistance to some “persistently poor- performing” homes identified in consultation with each state survey  agency. However, we pointed out that the number of these homes the  QIOs were required to serve was small, accounting for less than 10 percent  of the homes they were expected to assist intensively. CMS commented  that preliminary estimates from a special study conducted during the 7th  SOW indicated that assisting chronically poor-performing homes cost the  QIOs 5 to 10 times as much as assisting the “usual” home. Our report  acknowledged that additional resources were required for QIOs to assist  low-performing homes but suggested that CMS could decrease the total  number of homes assisted in order to increase the number of low-  performing homes beyond the small number mandated for the 8th SOW.
Direct QIOs to focus intensive assistance on those quality-of-care  areas on which homes most need improvement. CMS did not directly  respond to this recommendation, but did point out that about one-third of  QIOs were working primarily with homes on QMs on which the homes  scored worse than the national average during the 8th SOW. Our  recommendation was to direct all QIOs to focus intensive assistance on  QMs that reflect homes’ greatest quality-of-care challenges. We had  reported that some nursing homes assisted intensively by QIOs did not  have a choice of QMs on which to work. We concluded that having QIOs  work intensively with homes only on the quality-of-care issues that posed  particular challenges to them would maximize program resources.
Revise QIO program regulations to require QIOs to provide CMS  with the identities of the homes assisted in order to facilitate  evaluation. CMS did not specifically indicate whether it agreed with this  recommendation, but did indicate that it continues to explore options  which would allow access to data on the homes assisted intensively in  order to facilitate evaluation. However, CMS expressed concern that  providing this access could potentially subject the information to laws that  could afford third parties similar access. We believe that CMS should  continue to evaluate how best to maintain an appropriate balance between  disclosure and confidentiality. If CMS’s evaluation indicates that it is  unable to incorporate adequate confidentiality safeguards to promote  voluntary participation in QIOs’ quality improvement initiatives, the  agency could seek legislation that would provide such safeguards.
Collect more complete and detailed data on the interventions QIOs  use to assist homes. CMS responded to this recommendation, although it  labeled it “improve the monitoring of QIO activities,” and agreed with our  recommendation. CMS noted that, in concert with HHS, it is reviewing  recommendations from the IOM’s 2006 report on QIOs, which may result  in redesigning the program, including systems for evaluating QIO activities  in different care settings, such as nursing homes. CMS did not discuss how  it planned to collect additional data on QIO nursing home interventions.  Further, it stated that it may incorporate data-handling and -reporting  features of the nursing home subtask into overall program improvements.  We have reservations about this plan because we found that CMS  collected little information about specific QIO interventions with nursing  homes during the 7th SOW, the information collected was not sufficiently  comprehensive or consistent to be used to evaluate the interventions’  effectiveness, and QIOs themselves reported that the data collection  system was of limited use to them.
Identify a broader spectrum of measures than QMs to evaluate  changes in nursing home quality. CMS did not directly address this  recommendation. However, the agency took issue with our judgment that  the use of QMs to evaluate nursing home quality—and by extension, QIOs’  performance—is problematic. CMS commented that the QMs have passed  through rigorous development, testing, deployment, and national  consensus processes. We reported that the study commissioned by CMS to  validate the QMs has been criticized by experts on several grounds,  including a lack of statistical rigor. We also noted that CMS has revised or  is currently revising both the QMs and the MDS data used to calculate  them to address limitations, such as reliability and measurement  problems. For example, CMS has removed or replaced 5 of the original 10  QMs since 2002, including some of those on which the QIOs were  evaluated during the 7th SOW. In addition, CMS is currently updating the  MDS to reflect advances in clinical practice and to improve its utility as a  quality measure tool. While we expect that these efforts will improve the  QMs as measures of nursing home quality, we believe that the QMs’  current limitations argue for the use of a broader spectrum of measures to  evaluate changes in nursing home quality. Multiple long-term care  professionals we interviewed recommended that the QMs not be used in  isolation to assess the quality of care in nursing homes; these professionals  suggested a range of measures that could be used to supplement the QMs,  including perceptions of care by family members, residents, and staff;  state survey data; and nursing home staffing levels.
As arranged with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies to the Administrator of the  Centers for Medicare & Medicaid Services and appropriate congressional  committees. We will also make copies available to others upon request. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-7118 or allenk@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: Our analysis of QIOs’ work with nursing homes had three major  components: (1) site visits to five QIOs, (2) analysis of state survey data to  compare homes that were assisted intensively with homes that were not,  and (3) a Web-based survey of 51 QIOs.
subsections: section_title: Site Visits
paragraphs: We visited a QIO in each of five states to gather detailed information about  QIOs’ work with nursing homes from the perspective of the QIOs, nursing  homes in the intensive participant group, and stakeholders; we used this  information to address all three objectives. We selected the states-⎯and  by extension, the QIOs that worked in those states—on the basis of six  criteria described in the following section. After selecting the QIOs, we  identified nursing homes that received intensive assistance and  stakeholders to contact for interviews. We conducted most of our site visit  interviews in March and April 2006.
subsections: section_title: Selection of QIOs
paragraphs: We based our selection of QIOs on the following criteria:    Number of nursing home beds in the state. We divided the states into  three groups of 17 states each based on the number of nursing home beds  at the beginning of the 7th SOW (2002). We over-sampled states with high  numbers of nursing home beds by selecting one state with a low number  of beds, one state with a medium number, and three states with a high  number.
Evaluation score for the nursing home component of the 7th SOW  relative to scores of other QIOs. We divided the states into three  groups of 17 based on the QIOs’ evaluation scores for the 7th SOW. To help  us identify the possible determinants of scores, we selected more states at  each end of the spectrum than in the middle: two states with scores in the  bottom third, one state with a score in the middle third, and two states  with scores in the top third.
State survey performance of homes selected for intensive  assistance relative to homes not selected. We also considered the  extent to which the homes selected for intensive assistance by a given QIO  at the beginning of the 7th SOW differed from the homes that were not  selected, in terms of serious deficiencies cited on state surveys (both the  proportion of homes cited in each group and the average number of  serious deficiencies per home). We chose one QIO that selected worse  homes, three QIOs that selected homes that were neither better nor worse,  and one QIO that selected better homes.
Presence of a state-sponsored nursing home quality improvement  program. At the time we selected QIOs for site visits, we were aware of  four states that had state-sponsored quality improvement initiatives in  place during the 7th SOW. To learn more about these efforts and how they  interacted with and compared with efforts by the QIOs, we included one  state (Florida) with its own quality improvement initiative. After we made  our selection, we learned that another state we had selected (Maine) had a  state-sponsored quality improvement program.
QIO participation in the Collaborative Focus Facilities project.
CMS has funded QIOs to conduct several special studies with nursing  homes, including one in which the 17 participating QIOs each worked  intensively with up to five nursing homes identified by their state survey  agencies as having significant quality problems. To learn more about the  challenges involved in working with low-performing homes, we selected  two states whose QIOs participated in this project.
Census region. We selected states from four different regions of the  country: Northeast, Midwest, South, and West.
Using these criteria, we selected the following five states: Colorado,  Florida, Iowa, Maine, and New York. Together these states represented   15 percent of nursing home beds nationwide at the beginning of the 7th  SOW (2002).
section_title: Selection of Nursing Homes
paragraphs: Overall, we interviewed staff from 32 nursing homes in nine states. To  assist in the development of our site visit protocols, we interviewed staff  from 4 homes in four states. During the site visits to five states, we  interviewed staff from 28 nursing homes. In each state, we interviewed  staff from 4 to 8 nursing homes that received intensive assistance from the  QIO, for a total of 28 homes in these five states. The number of homes we  selected in each of the five states visited varied depending on the number  of homes the QIO was expected to select for intensive assistance, an  expectation based on the number of homes in the state. Specifically, we  selected either four homes or 7 percent of the maximum number of homes  that each of the five QIOs was expected to assist intensively, whichever  was greater.
We chose homes on the basis of four characteristics: number of serious  deficiencies in the standard state survey at the beginning of the 7th SOW  (2002), improvement in QM scores during the 7th SOW, distance from the  QIO (in order to include homes that were more difficult for QIOs to visit),  and urban versus rural location. Specifically, we sought to include (1) at  least one home that had one or more serious deficiencies and that finished  in the top third of the intensively assisted homes in their state in terms of  improvement on QM scores, and (2) at least one home that had one or  more serious deficiencies and that finished in the bottom third of the  intensively assisted homes in their state in terms of improvement on QM  scores. For the remaining homes, we sought a group whose state survey  deficiency levels and QM improvement scores were representative of the  range among intensive participants in their state. However, the  experiences of this sample of 32 homes cannot be generalized to the entire  group of homes that received intensive assistance from the QIOs  nationwide.
section_title: Selection of Stakeholders
paragraphs: In each state we also interviewed officials from three stakeholder groups:  (1) the state survey agency; (2) the local affiliate of the American Health  Care Association, which generally represents for-profit homes; and (3) the  local affiliate of the American Association of Homes and Services for the  Aging, which represents not-for-profit homes.
section_title: Analysis of State Survey Data
paragraphs: To assess the characteristics of the nursing homes that were selected by  the QIOs for intensive assistance from among the homes that volunteered,  we analyzed 3 years of standard state survey data on deficiencies cited at  nursing homes and compared the results for homes that were assisted  intensively with results for homes that were not; we used this information  to address our first objective. The analysis involved three steps:  1. identifying nursing homes that had three standard state surveys from  1999 through 2002;  2.  ranking nursing homes in each state in each year, based on the number  of serious and other deficiencies, and then classifying homes as  consistently low-, moderately, or high-performing; and  3. identifying on a nationwide and state-by-state basis any statistically  significant differences between homes selected and not selected by the  QIO, in terms of the proportion of low-, moderately, or high-performing  nursing homes.
subsections: section_title: Identifying Homes with Three Standard Surveys
paragraphs: To identify homes whose performance was consistently lower or higher  than other homes in their state prior to the selection of homes by the  QIOs, we included in our analysis only homes for which we were able to  identify three standard surveys from January 1, 1999, through November 1,  2002. Using the state survey calendar year summary files for 1999 through  2002 for the 50 states and the District of Columbia, we obtained 3 years of  deficiency data from standard surveys for 16,303 homes.
section_title: Classifying Homes as Low-, Moderately, or High-Performing
paragraphs: CMS classifies deficiencies according to their scope and severity. For each  of the three surveys, we ranked all of the nursing homes in each state  based on the number of deficiencies in two categories: (1) actual harm or  immediate jeopardy and (2) potential for more than minimal harm.
Deficiencies in the first category are considered serious deficiencies. We  gave more weight to the serious deficiencies by sorting the homes first on  the number of deficiencies in the first category and then on the number of  deficiencies in the second category. Homes with the same number of  deficiencies in each category were assigned the same rank. Based on these  rankings, we identified homes in the bottom and top quartile in each state  in each survey.
We classified homes as low-performing if they ranked in the bottom  quartile in the most recent of the three surveys and in at least one of the  two preceding surveys. We classified homes as high-performing if they  ranked in the top quartile in the most recent of the three surveys and in at  least one of the two preceding surveys. We classified homes as moderately  performing if they did not meet the criteria for inclusion in either the low-  or high-performing group. Of the 16,303 homes with three standard state  surveys during the period we specified, we classified 15 percent as low- performing, 65 percent as moderately performing, and 20 percent as high- performing.
To assess the stability of our categorization of homes as low- (or high-)  performing, we ran a logistic regression model to predict the probability of  a home being categorized as low- (or high-) performing in the most recent  of the three surveys given its categorization in the two prior surveys. The  regression results showed that homes that were categorized as low- (or  high-) performing in one survey were significantly more likely to be  categorized as low- (or high-) performing in the other surveys as well.
section_title: Determining Statistically Significant Differences between Homes Assisted Intensively and Homes Not Assisted Intensively by the QIOs
paragraphs: Our final step was to determine, on both a nationwide and state-by-state  basis, whether there was a statistically significant difference in the  proportion of (1) low-performing homes, (2) moderately performing  homes, and (3) high-performing homes in the group assisted intensively by  the QIOs compared with the group not assisted intensively.
section_title: Web-Based Survey of QIOs
paragraphs: To gather information about the characteristics of the QIOs, including  their process for selecting homes for intensive assistance from the pool of  volunteers and the interventions they used, on July 19, 2006, we launched  a two-part Web-based survey of QIOs in all 50 states and the District of  Columbia; we used this information to address objectives one and two.  We achieved a 100 percent response rate. The first part of the survey  gathered information about the primary personnel who worked with  nursing homes during the 7th SOW, including information about their  employment with the QIO, and their relevant credentials and experience.  The second part of the survey gathered information on a range of other  topics, including information about stakeholder involvement with the QIO,  recruitment and selection of nursing homes for intensive assistance,  interventions used with intensive participants, interventions used with  homes statewide, and QIOs’ communication with CMS. We specifically  inquired about QIOs’ use of six interventions: (1) mailings, faxes, and e- mails; (2) conferences; (3) small group meetings; (4) conference calls and  video or Web conferences with multiple homes; (5) telephone  conversations with individual homes; and (6) on-site visits. We asked  QIOs to rank and provide information on the two interventions they relied  on most to assist homes statewide and on the three interventions they  relied on most to assist homes in the intensive participant group. We also  asked QIOs to rank the effectiveness of the interventions they used and to  identify the interventions they would use if they could do the 7th SOW over  again.
section_title: Appendix II: Publicly Reported Quality Measures
paragraphs: In November 2002, CMS began a national Nursing Home Quality Initiative  that included the development of QMs that would be publicly reported on  the CMS Web site called Nursing Home Compare. CMS has continued to  refine the QMs and, as shown in table 4, has dropped some QMs and added  others.
section_title: Appendix III: Comments from the Centers for Medicare & Medicaid Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Walter Ochinko,   Assistant Director; Nancy Fasciano; Sara Imhof; Elizabeth T. Morrison;  Colbie Porter; and Andrea Richardson made key contributions to this  report.
section_title: Related GAO Products
paragraphs: Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not  Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241.  Washington, D.C.: March 26, 2007.
Nursing Homes: Despite Increased Oversight, Challenges Remain in  Ensuring High-Quality Care and Resident Safety. GAO-06-117.  Washington, D.C.: December 28, 2005.
Nursing Home Deaths: Arkansas Coroner Referrals Confirm Weaknesses  in State and Federal Oversight of Quality of Care. GAO-05-78.  Washington, D.C.: November 12, 2004.
Nursing Home Fire Safety: Recent Fires Highlight Weaknesses in  Federal Standards and Oversight. GAO-04-660. Washington D.C.:   July 16, 2004.
Nursing Home Quality: Prevalence of Serious Problems, While  Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561.  Washington, D.C.: July 15, 2003.
Nursing Homes: Public Reporting of Quality Indicators Has Merit, but  National Implementation Is Premature. GAO-03-187. Washington, D.C.:  October 31, 2002.
Nursing Homes: Quality of Care More Related to Staffing than Spending.  GAO-02-431R. Washington, D.C.: June 13, 2002.
Nursing Homes: More Can Be Done to Protect Residents from Abuse.  GAO-02-312. Washington, D.C.: March 1, 2002.
Nursing Homes: Federal Efforts to Monitor Resident Assessment Data  Should Complement State Activities. GAO-02-279. Washington, D.C.:  February 15, 2002.
Nursing Homes: Sustained Efforts Are Essential to Realize Potential of  the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.:   September 28, 2000.
Nursing Home Care: Enhanced HCFA Oversight of State Programs  Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.:  November 4, 1999.
Nursing Home Oversight: Industry Examples Do Not Demonstrate That  Regulatory Actions Were Unreasonable. GAO/HEHS-99-154R. Washington,  D.C.: August 13, 1999.
Nursing Homes: Proposal to Enhance Oversight of Poorly Performing  Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999.
Nursing Homes: Complaint Investigation Processes Often Inadequate to  Protect Residents. GAO/HEHS-99-80. Washington, D.C.: March 22, 1999.
Nursing Homes: Additional Steps Needed to Strengthen Enforcement of  Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.:   March 18, 1999.
California Nursing Homes: Care Problems Persist Despite Federal and  State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998.