Nursing Homes: CMS Needs Milestones and Timelines to Ensure Goals for the Five-Star Quality Rating System Are Met
Published: 2012-03-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2008, in an effort to provide helpful information to consumers and improve provider quality, the Centers for Medicare & Medicaid Services (CMS) developed and implemented the Five-Star Quality Rating System (Five-Star System). The Five-Star System assigns each nursing home an overall rating and three component ratingshealth inspections, staffing, and quality measuresbased on the extent to which the nursing home meets CMSs quality standards and other measures. The rating scale ranges from one to five stars, with more stars indicating higher quality.
The Patient Protection and Affordable Care Act directed GAO to review CMSs Five-Star System. This report examines (1) how CMS developed and implemented the Five-Star System and what key methodological decisions were made during development, (2) the circumstances under which CMS considers modifying the Five-Star System, and (3) the extent to which CMS has established plans to help ensure it achieves its goals for the Five-Star System. To conduct this work, GAO reviewed CMS documents, interviewed CMS officials and others, and assessed whether CMS uses certain strategic planning practices.
section_title: What GAO Found
paragraphs: CMS developed and implemented the Five-Star System largely during an 8-month period in 2008 with input from long-term care stakeholders, CMSs Five-Star System contractor, and members of a technical expert panela panel composed of nine individuals that CMS identified as experts in long-term care research. CMS made numerous methodological decisions during the development of the Five-Star System, including three key methodological decisions. GAO defines key methodological decisions as those that at least six technical expert panel membersof the nine that GAO contactedrecalled as eliciting the most intense review and discussion during the development of the Five-Star System. One key methodological decision was how to combine the component ratings to create an overall rating. The other two key methodological decisions pertained to how to create ratings that account for variation in the type of care provided across nursing homes.
CMS generally considers modifying the Five-Star System in response to (1) methodological issues raised by stakeholders, (2) its routine monitoring of the system, and (3) the availability of new data sources. CMS officials explained that when a methodological issue is raised by long-term care stakeholders, they review the Five-Star System to determine whether modifications should be made. Officials said that each issue raised does not always result in modifications to the Five-Star System, although some minor modifications have been made. CMS also considers making modifications to the Five-Star System based on its periodic analyses of trends of the system; however, to date, no modifications have been made based on these analyses. Lastly, CMS is currently determining how to modify the staffing and quality measure ratings of the Five-Star System based on newly available data.
CMS has several planned efforts intended to improve the Five-Star System, including evaluating the usability of the system, adding nursing home capability information, revising the staffing component, and developing additional quality measures. However, CMS lacks GAO-identified leading strategic planning practicesthe use of milestones and timelines to guide and gauge progress toward achieving desired results and the alignment of activities, resources, and goalsthat could help the agency to more efficiently and effectively accomplish its planned efforts intended to improve the Five-Star System. While CMS officials have given us broad estimates for when they anticipate some of these efforts to be implemented, CMS does not have milestones and timelines associated with implementing the efforts, which could help ensure that appropriate progress is made towards implementation. In addition, CMS has not established, through planning documents, how its planned efforts to improve the Five-Star System will help CMS achieve the goals of the systemto inform consumers and improve provider quality. As a result, CMS may not be identifying and prioritizing its intended improvements in a manner that best ensures that the goals are being achieved.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Administrator of CMS use strategic planning to establish how its planned efforts will help meet the goals of the Five-Star System, and develop milestones and timelines for each of its planned efforts. CMS agreed with these recommendations.

section_title: Letter
paragraphs: The nation’s almost 3.3 million nursing home residents are a vulnerable  population of elderly and disabled individuals who rely on nursing homes  to provide high-quality care. In 2009, these nursing home consumers and  their families had about 15,900 nursing homes participating in the  Medicare and Medicaid programs from which to choose for their care  needs. When deciding on a nursing home, consumers and their families  can make choices based on such factors as location, fees, specialties,  services and activities for residents, and what they know about the quality  of care provided in a facility. The Centers for Medicare & Medicaid  Services (CMS), an agency within the Department of Health and Human  Services (HHS), is responsible for establishing quality standards that  nursing homes must meet in the delivery of care to their residents and for  overseeing nursing homes’ compliance with those standards. In our prior  work, we have shown that the quality of care provided in nursing homes  can vary, often significantly, and have raised concerns about the quality  of care in some nursing homes. For example, our prior reports have found  that some nursing homes have been cited repeatedly for serious  deficiencies, such as residents having preventable pressure ulcers that  harmed them or put them at risk of death or serious injury.
Given the various factors nursing home consumers may consider when  choosing a facility, including quality of care, the variability in the quality of  care provided across nursing homes, and concerns over quality of care  problems in some nursing homes, CMS has taken steps to provide  assistance to individuals and their families in choosing a nursing home.  Specifically, in 1998 CMS began publicly reporting information related to  the quality of nursing homes on its Nursing Home Compare website. However, in 2007, some members of Congress raised concerns that this  information was not helpful to consumers because it was difficult to  understand.
To address these concerns, in 2008 CMS developed and implemented  the Nursing Home Five-Star Quality Rating System (Five-Star System),  which is posted on the Nursing Home Compare website. The primary goal  of the Five-Star System is to help consumers make informed decisions  about their care by providing understandable and useful information on  nursing home quality. The secondary goal of the Five-Star System is to  help improve nursing home quality by publicly reporting quality of care  information, as some research has suggested that publishing such  information can create an incentive for providers to improve their quality  of care. The Five-Star System assigns each nursing home an overall  rating and three component ratings based on the extent to which the  nursing home meets CMS’s quality standards and other measures. The  rating scale ranges from one to five stars, with more stars indicating  higher quality. www.Medicare.gov/NHCompare/home.asp.
Some long-term care stakeholders have voiced support for the Five-Star  System, stating that it helps consumers choose a nursing home for  themselves or a family member. However, other long-term care  stakeholders and some members of Congress have raised questions  about the Five-Star System, such as how the system was developed and  implemented and whether CMS is taking steps to make the system more  useful to consumers. For example, provider advocacy groups have raised  questions about the methodology used to develop the ratings and  consumer advocacy groups have made suggestions on the substance  and presentation of the Five-Star System in an effort to improve the  information provided to consumers.
The Patient Protection and Affordable Care Act directed us to review  CMS’s Five-Star System.and implemented the Five-Star System and what key methodological  decisions were made during development, (2) the circumstances under  which CMS considers modifying the Five-Star System, and (3) the extent  to which CMS has established plans to help ensure it achieves its goals  for the Five-Star System.
The CMS Five-Star System contractor is a consulting firm that employs health services  researchers and other support staff and conducts research in a range of fields, including  health care policy. to calculate the ratings, including the reasoning behind CMS’s final  methodological decisions. Finally, we identified the key methodological  decisions—which we define as the methodological decisions that at least  six of the members of the technical expert panel recalled as eliciting the  most intense discussion and review during development of the Five-Star  System—using a series of interviews and questionnaires. We interviewed  all nine technical expert panel members and received responses to our  questionnaire from seven of the nine members. For additional information  on our approach to identify the key methodological decisions, see  appendix I.
To describe the circumstances under which CMS considers modifying the  Five-Star System, we interviewed senior CMS officials about the factors  that prompt CMS to examine potential modifications to the system. We  also reviewed documents describing modifications made to the Five-Star  System since its implementation in December 2008 and the reasons  those modifications were made. In addition, we reviewed analyses of the  Five-Star System’s rating results that CMS’s Five-Star System contractor  has been conducting since the system’s implementation in December  2008.
To describe the extent to which CMS has established plans to help  ensure it achieves its goals for the Five-Star System, we interviewed  CMS officials, including senior officials responsible for the Five-Star  System, those involved in strategic planning for the agency, and those  involved in displaying health care information for consumers on CMS’s  websites. We identified and reviewed documents related to CMS’s plans  for improving the Five-Star System as well as documents that relate to  the system’s goals—informing consumers and improving provider quality.  We interviewed officials and reviewed documents from the HHS Agency  for Healthcare Research and Quality (AHRQ) regarding best practices  AHRQ has identified for effectively displaying health care quality  information to consumers on websites. Since the mid-1990s, we have  identified a variety of leading practices for effective strategic planning in  accordance with the Government Performance and Results Act  (GPRA).Star System and achieving its goals include two GAO-identified leading  practices for successful strategic planning and management: (1) using  intermediate goals and measures to show progress or contribution to  intended results, and (2) aligning activities and resources to support  agency goals.
We assessed whether CMS’s plans for improving the Five- We conducted this performance audit from June 2011 through March  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: CMS has publicly reported information on nursing home quality on its  Nursing Home Compare website since 1998 and has increased the  amount of information it reports on the website over time.2008, CMS announced its plans to make use of the information available  on the Nursing Home Compare website and begin assigning each nursing  home “star” ratings to help beneficiaries, their families, and caregivers  compare nursing homes more easily. Beginning in December 2008, CMS  made the Five-Star System publicly available on its Nursing Home  Compare website. The Five-Star System assigns star ratings for each  nursing home participating in the Medicare and/or Medicaid programs.
These star ratings include a separate rating for each of the three  components—health inspections, staffing, and quality measures—in  addition to an overall rating.
Health inspection rating. CMS contracts with state survey agencies  to conduct unannounced, on-site nursing home health inspections— known as surveys—to determine whether nursing homes meet federal  quality standards. Every nursing home receiving Medicare or  Medicaid payment must undergo a standard survey not less than  once every 15 months, and the statewide average interval for these  surveys must not exceed 12 months. State surveyors also conduct  complaint investigations in response to allegations of quality  problems. State surveyors may spend several days in the nursing  home to assess whether the nursing home is in compliance with  federal quality standards. If nursing homes are found to be out of  compliance with any requirements, state surveyors issue deficiency  citations that reflect the scope (number of residents affected) and  severity (level of harm to residents) of the deficiency. The health  inspection rating is a result of nursing home performance on surveys  and complaint investigations.
Specifically, this star rating is based on the scope and severity of  deficiencies from the last 3 years of routine surveys and complaint  investigations. To calculate this rating, the most recent survey findings  are weighted more heavily than those from the prior 2 years. A  nursing home’s health inspection rating is relative to other nursing  homes’ health inspection ratings in their state. This rating is updated  for each nursing home when new survey data become available for  that facility.
Staffing rating. Nursing homes self-report staffing hours for a 2-week  period at the time of the routine survey. CMS converts the reported  point-in-time staffing hours for nursing staff—registered nurses,  licensed practical nurses, and certified nursing assistants—into  measures that indicate the number of registered nurse and total  nursing hours per resident per day.
This star rating is based on the reported registered nurse and total  nursing staffing levels, adjusted for differences in the level of  complexity of nursing services required to care for residents across  nursing homes—referred to as resident acuity. The adjustment for  resident acuity is done using data from a resident assessment tool  called the Minimum Data Set (MDS), which nursing homes complete  and periodically report to CMS. MDS collects information on residents’  health, physical functioning, mental status, and general well-being. Each nursing home’s staffing rating is assigned based on how its total  nursing and registered nurse staffing levels compare to the  distribution of staffing levels for freestanding facilities in the nation  and staffing level thresholds identified by CMS. In addition, this  rating is updated when new staffing data are collected at the time of  the routine survey, generally every 12 months.
CMS measures resident acuity using the Resource Utilization Group (RUG)-III case mix  system that uses resident assessment data routinely collected in MDS. This system  classifies residents into 1 of 53 categories according to predicted resource needs,  particularly the expected amount of staff time required to care for residents. In October  2010, CMS implemented a new version of MDS—MDS 3.0—and this will change the data  used for the case mix system.
Quality measure rating. CMS uses data from MDS to calculate  various quality measures for each nursing home. These measures  include, for example, the prevalence of pressure sores and changes  in residents’ mobility.
This star rating is based on 10 different quality measures. This rating  is typically updated quarterly. Quality measure ratings are assigned to  generally achieve the following distribution: the top 10 percent of  nursing homes receive five stars, the bottom 20 percent receive one  star, and the middle 70 percent of nursing homes receive two, three,  or four stars, with equal proportions (23.33 percent) in each  category. ratings since January 2011 to allow CMS to collect resident  information from nursing homes using a new version of MDS and  refine and test quality measures using this revised assessment tool.
However, CMS has not updated the quality measure  The overall star rating is calculated using a process that combines the  star ratings from the health inspection, staffing, and quality measure  components. The overall rating is assigned based on the following steps:  1.  Start with the number of stars for the health inspection rating.  2.  Add one star if the staffing rating is four or five stars and also greater  than the health inspection rating. Subtract one star if the staffing rating  is one star. The rating cannot go above five stars or lower than one  star.
Performance on the two activities of daily living-related measures is weighted 1.6667  times as high as the other measures. In addition, thresholds for the two activities of daily  living quality measures are reset with each quarterly update of the quality measures data  based on the state-specific distribution of these measures. Thresholds for the other quality  measure ratings are fixed based on the national distribution of these measures on   January 5, 2009.
3.  Add one star if the quality measure rating is five stars. Subtract one  star if the quality measure rating is one star. The rating cannot go  above five stars or lower than one star.
See figure 1 for an example of how a nursing home’s overall rating is  calculated. This rating is updated when any of the three component  ratings change. For example, changes to the quality measure rating could  change a nursing home’s overall rating.
See appendix II for additional information on the methodology for  calculating nursing home ratings under the Five-Star System.
section_title: CMS Used Input from External Entities on the Development and Implementation of the Five-Star System, Which Included Three Key Methodological Decisions
paragraphs: CMS developed and implemented the Five-Star System largely during an  8-month period with input from long-term care stakeholders, its Five-Star  System contractor, and members of a technical expert panel. CMS made  numerous methodological decisions during the development of the Five- Star System, including three key methodological decisions that elicited  the most discussion during development.
subsections: section_title: CMS Solicited Input from Long-term Care Stakeholders, a Contractor, and a Panel of Experts
paragraphs: CMS largely developed and implemented the Five-Star System during an  8-month period from April to December 2008, soliciting input from long- term care stakeholders, the Five-Star System contractor, and members of  a technical expert panel. For example, after the Acting Administrator of  CMS directed CMS officials to develop a rating system for nursing homes  in late April 2008, CMS obtained comments about the planned rating  system from long-term care stakeholders through an Open Door Forum  in June 2008. CMS also reviewed and summarized comments it received  via an email account set up specifically for comments regarding the  planned rating system through late July. Concurrently, CMS developed  the methodology to calculate nursing home ratings in collaboration with  the Five-Star System contractor and members of a technical expert panel.  CMS’s contractor established this panel, composed of nine members that  the contractor and CMS identified as experts in long-term care and that  included researchers and an industry representative, to help guide the  development and implementation of the Five-Star System. CMS and its  contractor convened five meetings with the technical expert panel  between July and December 2008 to review and discuss analyses  conducted by the Five-Star System contractor regarding various options  for calculating the ratings. In November and December 2008, CMS  hosted meetings with reporters and other government entities, including  the HHS Administration on Aging, and hosted another Open Door Forum  to discuss the impending implementation of the Five-Star System with  stakeholders. In December 2008, CMS gave nursing home providers a  preview of their ratings and, on December 18, 2008, CMS made the Five- Star System publicly available on the Nursing Home Compare website.  (See fig. 2 for a timeline of CMS’s development and implementation of the  Five-Star System.)
CMS made numerous methodological decisions during the development  of the Five-Star System, including three key methodological decisions.  We define key methodological decisions as those that at least six  members of CMS’s technical expert panel recalled as eliciting the most  intense review and discussion during the development of the Five-Star  System. According to the panel experts, one key methodological  decision was how to combine the component ratings to create an overall  rating. The other two key methodological decisions pertain to how to  create ratings that account for variation in the type of care provided  across nursing homes. Specifically, the second key methodological  decision the experts recalled was whether to exclude hospital-based  nursing homes or set up separate ratings for hospital-based and  freestanding nursing homes.experts recalled was whether the staffing rating should be based on  nursing staffing levels that are adjusted to reflect resident acuity.
The third key methodological decision the
How to combine the component ratings to create an overall  rating. One key methodological decision that CMS made was  deciding how to best combine the health inspection, quality measure,  and staffing component ratings to create an overall rating. Technical  expert panel members told us that they discussed how much weight  should be assigned to each component when combining the  component ratings to calculate the overall rating. One technical expert  panel member told us that, in the end, the members generally agreed  to assign more weight to the health inspection and staffing  components and less weight to the quality measure component, which  reflects differences in the perceived validity and reliability of the data  used to calculate these components. Consistent with the technical  expert panel’s proposal, CMS decided to assign overall ratings to  each nursing home by starting with the health inspection rating, then  adding or subtracting stars if the nursing home’s staffing or quality  measure rating was particularly high or low, with a minimum and  maximum rating of one and five stars, respectively.
Whether to exclude hospital-based nursing homes or set up  separate ratings for hospital-based and freestanding nursing  homes. Another key methodological decision that CMS made was  whether to exclude nursing homes that are hospital-based from the  Five-Star System or to set up separate ratings for hospital-based and  freestanding nursing homes. CMS’s Five-Star System contractor  stated that hospital-based facilities may typically provide a different  type of care than freestanding facilities. That is, hospital-based  facilities may provide care to more acute residents who require more  extensive services than freestanding facilities. However, one senior  CMS official told us that some hospital-based and freestanding  facilities may provide care to similar types of residents. While CMS  considered creating separate ratings for hospital-based and  freestanding nursing homes, CMS ultimately decided not to exclude  hospital-based nursing homes or create a separate rating scale for  hospital-based and freestanding nursing homes.
Whether staffing ratings should be based on nursing staffing  levels that are adjusted to reflect resident acuity. A third key  methodological decision that CMS made was whether the staffing  rating that nursing homes receive should be based on reported  staffing levels that are adjusted to account for resident acuity. For  example, in a published article, two of the technical expert panel  members and others argued that the appropriate level of nurse  staffing may differ substantially in two nursing homes with identical  numbers of staff, because of the differences in the amount of time  needed to care for residents due to variation in resident acuity.  However, panel members told us that they debated whether and the  extent to which nursing staffing levels should be adjusted to reflect  resident acuity due to concerns about the validity of existing  adjustment approaches. Because CMS agreed that nursing home  staffing levels should reflect the care needs of the facility’s residents,  CMS decided to adjust staffing levels for differences in the expected  amount of time required to care for residents and worked with the  technical expert panel to select an adjustment method.
section_title: CMS Typically Considers Modifying the Five-Star System in Response to Stakeholders, after Routine Monitoring, and When New Data Sources Become Available
paragraphs: CMS generally considers modifying the Five-Star System in response to  (1) methodological issues raised by stakeholders, (2) its routine  monitoring of the system, and (3) the availability of new data sources.  CMS officials explained that when a methodological issue is raised by  long-term care stakeholders, they review the Five-Star System to  determine whether modifications should be made. Officials further said  that each issue raised does not always result in modifications to the Five- Star System, although some minor modifications have been made. For  example, in September 2009, in response to input it received from long- term care stakeholders, CMS updated the methodology for calculating a  nursing home’s health inspection rating. With the methodological change,  the health inspection rating for a nursing home is no longer updated  unless new survey data for that nursing home becomes available. Initially,  because each nursing home’s health inspection rating is relative to the  ratings of all nursing homes in the same state, the results of new surveys  and rating changes for some nursing homes could have affected the  ratings for other nursing homes in the state. Stakeholders had noted to  CMS that the rating of an individual nursing home therefore could be  changed even if that nursing home had not had a new survey. CMS made  the methodological change to address this concern, while noting that this  had happened to only a small percentage of nursing homes.
Some issues raised by stakeholders have not resulted in a modification of  the Five-Star System. For example, one stakeholder raised a concern  that surveys should have fixed thresholds instead of thresholds that are  relative to—or based on—other nursing homes’ survey scores in the  same state. CMS reviewed this concern with its Five-Star System  contractor and found that fixed thresholds would likely present  disadvantages to nursing homes because changes in survey policy could  lead to changes in survey scores and result in significant changes for  nursing homes’ health inspection ratings. CMS officials stated that using  a relative distribution allows nursing homes’ ratings to remain more stable  and, additionally, allows consumers to compare nursing homes within a  state.
CMS also considers making modifications to the Five-Star System based  on its periodic analysis of the system’s rating trends; however, to date, no  modifications have been made based on these analyses. CMS’s Five- Star System contractor conducts this analysis and provides monthly  reports to CMS as well as an annual report that describes changes over  time in nursing homes’ ratings and trend information on the individual  components of the system. intended to help CMS evaluate the Five-Star System and determine if  modifications are needed. For example, CMS examined whether it should  modify the health inspection component because some state survey  agencies have begun using the Quality Indicator Survey to collect survey  information. In 2010, as part of its analysis for one of its monthly reports,  CMS examined whether the health inspection rating of facilities that were  inspected using the Quality Indicator Survey differed from those that were  inspected with the traditional, paper-based survey. No significant  difference was found between the two approaches to conducting the  survey and, as a result, CMS did not modify the Five-Star System.
Through this analysis, CMS has found that nursing homes’ star ratings have improved in  the first 2 years since the Five-Star System has been implemented, but it is unclear if the  results are due to more accurate reporting of the data or an actual improvement in quality. ratings of the Five-Star System based on new data available from MDS  3.0, an updated version of the resident assessment tool. CMS is  examining how using data from MDS 3.0 to adjust staffing data to account  for resident acuity will affect the staffing ratings of nursing homes. In  addition, CMS is considering whether the availability of these new data  should result in a modification to the manner in which the thresholds for  the staffing component ratings are determined. CMS intends to complete  any modifications to the staffing component rating based on the new data  by April 2012. CMS is also currently refining and testing the nursing home  quality measures using MDS 3.0 data and is considering options for  modifying how the quality measure component rating for the Five-Star  System will be calculated using these new data. CMS anticipates that the  quality measure component of the Five-Star System will be modified at  the end of calendar year 2012. Because of these changes, CMS has not  updated the Five-Star System quality measure ratings for individual  nursing homes since January 2011—the last time period for which data  using the previous assessment tool were available.
Similarly, CMS is considering modifying the data it uses to calculate the  nursing home staffing rating based on another new data source— electronic payroll data. Although CMS currently calculates this rating  based on staffing data from a 2-week period of time, CMS plans to use  payroll records of nursing homes to collect staffing data from nursing  homes, once these data are available in the next 3 to 5 years. A CMS  official told us that the agency has determined what type of data it wants  to collect through nursing home payroll systems, such as data that will  allow the calculation of nursing staff turnover in a nursing home, and is  working with a private firm to develop the system to collect this  information. Electronic payroll data have several strengths, including that  they will allow CMS to collect data on several aspects of staffing that are  not currently available, including the percentage of staff that are full time  and the number of staff that provide direct patient care. In addition,  because payroll data originate from employees and are used to pay their  salaries, CMS officials stated that they have greater confidence in the  accuracy of the data. According to one senior official, CMS does not  expect to determine potential modifications to the nursing home staffing  rating of the Five-Star System until it has experience collecting payroll  data.
section_title: Although CMS Intends to Make Improvements, the Agency Has Not Ensured That Its Efforts Will Help Achieve the Goals of the Five-Star System
paragraphs: CMS has several planned efforts intended to improve the Five-Star  System. However, CMS has not taken steps to ensure that these efforts  will help CMS achieve its goals for the Five-Star System—to inform  consumers and improve provider quality. Specifically, CMS has not  established intermediate goals and measures—such as interim  milestones and related timelines—to guide the implementation of these  efforts. In addition, CMS has not established how any changes resulting  from these efforts, if implemented, will support the goals of the Five-Star  System.
subsections: section_title: CMS Has Identified Efforts Intended to Improve the Five-Star System, but Lacks Milestones and Timelines to Guide Implementation
paragraphs: CMS has awarded contracts for or begun discussions about several  planned efforts it has identified to improve the Five-Star System. These  plans include evaluating its usability, adding information on nursing home  capability, revising the staffing component, and developing additional  quality measures. Specifically, CMS’s planned efforts are:
To evaluate the usability of the Nursing Home Compare website,  which includes the Five-Star System. CMS plans to conduct a  multiphase evaluation of the usability of the Nursing Home Compare  website, which includes the Five-Star System, including testing the  website with consumers and surveying stakeholders. In an effort to  integrate the website with CMS’s other “compare” websites, such as  Hospital Compare and Home Health Compare, CMS is currently  redesigning the website. As part of this redesign, CMS is changing the  appearance of Nursing Home Compare to be similar to that of other  compare websites. To inform this redesign, in December 2011, CMS  tested proposed web page layouts for the revised website with a  group of nine participants. However, according to a senior CMS  official, this testing and evaluation have been limited in scope and  depth due to time constraintsgathering information on the usability of the Nursing Home Compare  website and Five-Star System. The official stated that CMS plans to  conduct a more in-depth evaluation of the Nursing Home Compare  website and the Five-Star System than any done to date—one that  includes more detailed consumer testing, a survey of long-term care  stakeholders, and a web-based pop-up survey. CMS’s Five-Star  System contractor will assist with the more in-depth evaluation of the  usability of the Nursing Home Compare website and the Five-Star  System in fiscal year 2012.   and are only the first step toward
To evaluate options to better distinguish among nursing homes’  various care capabilities. A senior CMS official stated that CMS is  evaluating options for refining the Nursing Home Compare website  and the Five-Star System to incorporate additional information about  particular nursing home capabilities, such as whether a nursing home  specializes in rehabilitative short-stay versus long-stay care. One  option is to post information on particular capabilities, such as whether  a nursing home has a rehabilitation or dementia support unit. Another  option under review is the creation of a rating system that assigns  ratings separately for nursing homes that primarily provide short-stay  care and for those that primarily provide long-stay care. In fiscal year  2012, CMS’s Five-Star System contractor will develop options for  additional information that might be collected regarding specific  capabilities of nursing homes that would be useful to CMS or Nursing  Home Compare and Five-Star System users.
To evaluate options to include other types of nursing home staff  in the staffing component rating. A senior CMS official stated that  CMS would like to include some non-nurse staff, such as therapy  staff, in the staffing component of the Five-Star System.CMS must first evaluate the feasibility and options for including  additional staff. Subject to this evaluation, CMS officials said they  would like to include some non-nursing staff in the staffing component  by January 2013. CMS already collects data from nursing homes on  some non-nursing staff levels, including therapy staff, but does not  currently use this information to calculate the staffing rating.
To develop more quality measures for the quality measure  component rating. CMS is in the initial stages of identifying  additional nursing home quality measures for use in the Five-Star  System. CMS’s Five-Star System contractor has been tasked with  identifying potential measures for use in the Five-Star System in fiscal  years 2011 and 2012. A senior CMS official stated that identifying  quality measures for the Five-Star System is an ongoing task under  the contract and includes working with experts in the field of nursing  home quality measure development, reviewing literature, and  analyzing the results of potential quality measure data, such as data  generated from MDS 3.0. In meetings with the technical expert panel,  CMS and its contractor have solicited feedback from panel members  on quality measures that would improve the Five-Star System.
Although CMS has several planned efforts intended to improve the Five- Star System, the agency has not established intermediate goals and  measures—such as interim milestones and related timelines—that could  be used to show progress or contribution towards implementing these  efforts. Specifically, while CMS officials have given us broad estimates for  when they anticipate some of these efforts to be undertaken, CMS does  not have planning documents or strategies that outline specific milestones  and timelines associated with implementing the agency’s planned efforts  to improve the Five-Star System. We have found, in our prior work, that  developing and using specific milestones and timelines to guide and  gauge progress toward achieving an agency’s desired results is a leading  practice for effective strategic planning and management. Particularly  because much of the implementation is to be conducted by the Five-Star  System contractor, the use of milestones and timelines to guide and  gauge implementation is especially useful for helping to ensure that CMS  has clear expectations for its contractor and a means for determining  whether appropriate progress has been made in implementing these  efforts.
section_title: CMS Has Not Established How Its Planned Efforts Will Help Achieve Goals
paragraphs: CMS has not established, through planning activities or resulting planning  documents, how its planned efforts to improve the Five-Star System will  help CMS achieve the goals of the system—to inform consumers and  improve provider quality. As a result, CMS may not be identifying and  prioritizing its efforts in a manner that best ensures that the goals are  being achieved. We have found, in our prior work, that aligning activities,  resources, and goals is a leading strategic planning practice that can help  agencies to more efficiently and effectively achieve their goals.
CMS officials stated that the agency has no planning activities or resulting  planning documents that link the Five-Star System’s goals with CMS’s  efforts to improve the system and that establish priorities among the  efforts, based on these goals. This lack of planning and priority setting  may explain why CMS has taken limited steps to determine whether or to  what extent the Five-Star System is achieving the primary goal of  providing consumers with understandable and useful information on  nursing home care. For example, consumer testing of the Five-Star  System could provide information on the extent to which this goal is being  achieved. According to AHRQ, consumer testing is a key practice for  ensuring that health care quality information is publicly reported in a  manner that is useful and understandable to consumers. However, CMS  has gathered very limited information from consumers regarding the  usefulness of the Five-Star Systemprioritized its efforts based on the goal of informing consumers.   and it appears that CMS has not  In addition, CMS has not set specific priorities among its planned efforts  based on how to best achieve the secondary goal of the Five-Star  System—to improve provider quality. For example, while public reporting  itself can be an incentive for providers to improve quality, there may be  other efforts that could help CMS to further accomplish this goal. Nursing  home quality measures serve a number of purposes, including providing  data to nursing homes to help with their own quality improvement efforts.  In addition, making quality measure information available to consumers  allows them to distinguish among nursing homes’ quality and provides an  incentive for providers to improve their quality. Therefore, adding more  individual quality measures to that rating component in the Five-Star  System could potentially create even greater incentives as well as provide  nursing homes with more data on how to improve the quality of their care.  However, CMS has not explicitly prioritized this planned effort or others  based on achievement of this secondary goal. This planning and  prioritization is especially important for quality measures because quality  measures can be costly and time intensive to develop, especially if new  data need to be collected in order to create the measures.
CMS officials said that the agency has no planning activities or resulting  planning documents related to the Five-Star System because there are  too many intervening circumstances that make planning difficult. These  include uncertainty about resources available for the Five-Star System  due to competing resource needs within the agency, mandatory activities  required by the Patient Protection and Affordable Care Act that take up  staff time and effort, and other agency initiatives. However, CMS officials  acknowledged that, as CMS’s budget is more constrained, the  development of planning documents that prioritize its efforts intended to  improve the Five-Star System will become increasingly important.
section_title: Conclusions
paragraphs: In an attempt to make information on nursing home quality easier for  consumers to understand and use, and to help improve provider quality,  CMS developed and implemented the Five-Star System in 8 months  using information that was readily available. This was a significant step  toward increasing the transparency of information important to  consumers, but for CMS to sustain the Five-Star System over time, the  agency will need to continue making a concerted effort. CMS has made  some efforts to update the Five-Star System as it reviews the system’s  underlying components to identify potential ways to improve the system  over time. However, there can be significant challenges to ensuring that  the Five-Star System remains useful and valid over time, especially when  the components of that system continue to evolve. While CMS has  identified efforts it intends to make to improve the Five-Star System, the  agency has not strategically planned how to carry out these efforts, such  as outlining the milestones and timelines that will help ensure that  progress is being made. In addition, CMS has not clearly identified how  each of its planned efforts will help achieve the goals of the Five-Star  System. As a result, CMS may not know how it will prioritize and best  leverage its available resources to implement these efforts and achieve  the goals of the Five-Star System. Additionally, during this period of fiscal  constraint, these strategic planning practices can help CMS to better  anticipate and make resource allocation decisions that minimize the effect  of funding constraints on accomplishing the goals of the Five-Star  System.
section_title: Recommendations for Executive Action
paragraphs: In order to strengthen CMS’s efforts to improve the Five-Star System, we  recommend that the Administrator of CMS use strategic planning  practices to: establish—through planning documents—how its planned efforts will  help CMS achieve the goals of the Five-Star System, and develop milestones and timelines for each of its planned efforts.
section_title: Agency Comments
paragraphs: We received written comments on a draft of this report from HHS on  behalf of CMS, which are reprinted in appendix III. CMS agreed with our  recommendations and submitted general comments on the draft.
Specifically, CMS agreed with our recommendation to establish—through  planning documents—how its planned efforts will help CMS achieve the  goals of the Five-Star System. CMS stated that it will work to develop a  strategic plan for the Five-Star System that will address the short- and  long-term goals for the system and the manner in which those goals will  be achieved. CMS said that it would include a mechanism for receiving  regular input from consumers and other stakeholders as part of a  strategic plan. CMS also agreed with our recommendation to develop  milestones and timelines for each of its planned efforts. CMS  acknowledged the importance of both a strategic plan and the use of  specific timelines and milestones for measuring progress toward meeting  the goals of the Five-Star System and budgeting for the resources  needed to meet those goals.
We are sending copies of this report to the Secretary of HHS and the  Administrator of CMS and other interested parties. In addition, the report  also will be available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-7114 or kohnl@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Methodology for Identifying Key Methodological Decisions Made during Five- Star Quality Rating System Development
paragraphs: To identify the key methodological decisions made during the  development of the Five-Star Quality Rating System (Five-Star System),  defined as those that caused the most intense discussion and review  according to at least six members of the Centers for Medicare & Medicaid  Services’ (CMS) technical expert panel, we solicited the views of panel  members through a series of interviews and questionnaires.
Specifically, we completed the following steps:  1.  We interviewed each member of the technical expert panel using a  structured interview set containing open-ended questions. During  these interviews, we asked each member to provide their views on the  top three methodological decisions that caused the most discussion  and review during the development of the Five-Star System and to  describe the differing views expressed by members of the panel on  the methodological approach that CMS was considering. 2.  We summarized the open-ended interview responses related to the  views on the methodological decisions that caused the most  discussion and review during the development of the Five-Star  System. 3.  We distributed a questionnaire to the members of the technical expert  panel that outlined the methodological decisions identified during our  interviews and asked each member to identify the six methodological  decisions they recalled as eliciting the most intense review and  discussion during the development of the Five-Star System. 4.  We analyzed the responses to our questionnaire to identify the key  methodological decisions—those that at least six members of CMS’s  technical expert panel recalled as eliciting the most intense review  and discussion during the development of the Five-Star System.  Seven of nine technical expert panel members responded to our  questionnaire. We made multiple unsuccessful attempts to obtain  completed questionnaires from the remaining two members over a   4 week period. (See table 1.)
section_title: Appendix II: Overview of CMS’s Five-Star Quality Rating System Methodology
paragraphs: In the Five-Star System, nursing homes are assigned ratings for three  components—health inspections, staffing, and quality measures—and an  overall rating. These ratings range from one star to five stars, with more  stars indicating higher quality.
subsections: section_title: Health Inspection Rating
paragraphs: Each nursing home is assigned a health inspection rating in comparison  to other nursing homes in its state using a point system. These points are  assigned based on the nursing home’s three most recent health  inspections—known as a survey—results, including survey revisits and  complaint surveys, over the past 3 years. Points are assigned based on  the number, scope, and severity of a nursing home’s health deficiencies  found during surveys, with deficiencies with greater scope and severity  equating to more points. Therefore, a lower survey point total results in a  better rating (see table 2). If multiple revisits are required to ensure that  major deficiencies are corrected, additional points are added to the health  inspection score (see table 3). Based on the totals received, the top   10 percent of nursing homes in a given state receive five stars, the  bottom 20 percent receive one star, and the middle 70 percent of nursing  homes receive two, three or four stars, with equal proportions   (23.33 percent) in each category.
Each nursing home’s staffing rating is calculated based on the facility’s  self-reported registered nurse (RN) and total nursing (sum of RN,  licensed practical nurse, and certified nursing assistant) staffing levels for  a 2-week period around the time a routine survey is conducted. Staffing  levels are converted to hours per resident day and are adjusted to reflect  varying levels of resident acuity. Each nursing home’s staffing rating is  assigned based on how its total nursing and RN staffing levels compare  to the distribution of staffing levels for freestanding facilities in the nation  and staffing level thresholds identified by CMS. (See table 4.)
Each nursing home’s quality measure rating is calculated based on the  nursing home’s performance over the three most recent quarters on   10 of 19 quality measures, including 7 long-stay and 3 short-stay  measures. Two of the long-stay measures capture aspects of activities of  daily living, which reflect nursing home residents’ ability to provide self- care. Performance on the two activities of daily living-related measures is  weighted 1.6667 times as high as the other measures. This, according to  CMS, reflects the greater importance of these measures to many nursing  home residents and ensures that the two activities of daily living  measures count for 40 percent of the overall weight of the long-stay  measures. For the individual quality measures used to calculate this  rating, nursing homes that have lower percentages are considered to  have higher quality of care and, thus, receive more points. For example,  for one of the quality measures used in the Five-Star System—the  percentage of patients who were physically restrained—nursing homes  with lower percentages of patients who were physically restrained are  considered to have higher quality of care. As a result, those nursing  homes receive more points towards their quality measure rating than  facilities in which a higher percentage of residents are physically  restrained. The points received for all quality measures are summed to  create a total score for each facility with a higher point total equating to a  better quality measure star rating. Quality measure ratings are then  assigned to generally achieve the following distribution: the top   10 percent of nursing homes receive five stars, the bottom 20 percent  receive one star, and the middle 70 percent of nursing homes receive  two, three, or four stars, with equal proportions (23.33 percent) in each  category.
section_title: Overall Rating
paragraphs: Each nursing home’s overall rating is based on its ratings for the three  components—health inspections, staffing, and quality measures. From  these three ratings, the overall rating is assigned based on the following  steps:  1.  Start with the number of stars for the health inspection rating.  2.  Add one star if the staffing rating is four or five stars and also greater  than the health inspection rating. Subtract one star if the staffing rating  is one star. The rating cannot go above five stars or lower than one  star. 3.  Add one star if the quality measure rating is five stars. Subtract one  star if the quality measure rating is one star. The rating cannot go  above five stars or lower than one star.
If the health inspection rating is 1 star, then the overall rating cannot be  upgraded by more than one star based on the staffing and quality  measure ratings. In addition, if a nursing home is a Special Focus  Facility that has not graduated, the maximum overall rating allowable is  3 stars.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Karen Doran, Assistant Director;  Danielle Bernstein; Deirdre Brown; Krister Friday; Giselle Hicks; Melanie  Krause; Lisa Motley; and Jessica Smith made key contributions to this  report.