Nursing Homes: Consumers Could Benefit from Improvements to the Nursing Home Compare Website and Five-Star Quality Rating System
Published: 2016-11-18T00:00:00

Fast Facts:
Millions of Americans are cared for in nursing homes each year, and that number is expected to increase as baby boomers age.
So how do you choose a nursing home?
Nursing Home Compare is an online tool that allows people to research and compare nursing homes using a rating system. We found that most people think the site is helpful, but it's not perfect. For example, it lacks some key information, such as consumer satisfaction scores.
The Centers for Medicare & Medicaid Services runs the site, and we recommended four ways CMS can improve it and make it a better tool for consumers, including adding consumer satisfaction scores.

Highlights:
section_title: Why GAO Did This Study
paragraphs: Approximately 15,600 nursing homes participating in the Medicare and Medicaid programs provide care to 1.4 million residents each year. To help consumers make informed choices about nursing homes, CMS developed the Nursing Home Compare website, and on the site made available the Five-Star System, which rates homes on quality components.
GAO was asked to assess the website and rating system as tools for consumers. GAO examined (1) the information CMS collects about the use of Nursing Home Compare, including its usefulness to consumers, and potential areas, if any, to improve the website, and (2) the extent to which the Five-Star System enables consumers to understand nursing home quality and make distinctions between homes. GAO reviewed CMS documents and interviewed CMS officials and national and a non-generalizable sample of state-level stakeholders from four states, selected on factors such as size. GAO also analyzed Five-Star System and consumer complaint data, and analyzed resident satisfaction data from two of the four selected states.
section_title: What GAO Found
paragraphs: GAO found that the Centers for Medicare & Medicaid Services (CMS) collects information on the use of the Nursing Home Compare website, which was developed with the goal of assisting consumers in finding and comparing nursing home quality information. CMS uses three standard mechanisms for collecting website information—website analytics, website user surveys, and website usability tests. These mechanisms have helped identify potential improvements to the website, such as adding information explaining how to use the website. However, GAO found that CMS does not have a systematic process for prioritizing and implementing these potential improvements. Rather, CMS officials described a fragmented approach to reviewing and implementing recommended website changes. Federal internal control standards require management to evaluate appropriate actions for improvement. Without having an established process to evaluate and prioritize implementation of improvements, CMS cannot ensure that it is fully meeting its goals for the website.
GAO also found that several factors inhibit the ability of CMS's Five-Star Quality Rating System (Five-Star System) to help consumers understand nursing home quality and choose between high- and low- performing homes, which is CMS's primary goal for the system. For example, the ratings were not designed to compare nursing homes nationally, limiting the ability of the rating system to help consumers who live near state borders or have multistate options. In addition, the Five-Star System does not include consumer satisfaction survey information, leaving consumers to make nursing home decisions without this important information. As a result, CMS cannot ensure that the Five-Star System fully meets its primary goal.
section_title: What GAO Recommends
paragraphs: GAO is making four recommendations, including, that CMS establish a process to evaluate and prioritize website improvements, add information to the Five-Star System that allows homes to be compared nationally, and evaluate the feasibility of adding consumer satisfaction data. HHS agreed with three of GAO's recommendations, but did not agree to add national comparison information. GAO maintains this is important information, as discussed in the report.

section_title: Letter
paragraphs: Nationwide, approximately 15,600 nursing homes participating in the  Medicare and Medicaid programs provide care to about 1.4 million  nursing home residents—a vulnerable population of elderly and disabled  individuals. As the estimated 76 million baby boomers born between  1946 and 1964 become elderly, the number of nursing home residents  may increase. When deciding on a nursing home, consumers—current  and potential nursing home residents and their family or friends—can  make choices based on such factors as location, fees, services and  activities for residents, and what they know about the quality of care  provided in a facility.
Given the various factors to consider when choosing a facility, the  Centers for Medicare & Medicaid Services (CMS)—an agency within the  Department of Health and Human Services (HHS) that is responsible for  establishing and overseeing the quality standards that nursing homes  must meet in order to participate in the Medicare and Medicaid  programs—has taken steps to help consumers. Specifically, in 1998 CMS  began publicly reporting information related to the quality of nursing  homes on its Nursing Home Compare website. CMS’s goals for the website include helping consumers compare nursing home quality and  assisting them in finding other information about nursing homes. In order  to further aid consumers using Nursing Home Compare, in 2008 CMS  implemented the Five-Star Quality Rating System (Five-Star System) on  the website. The Five-Star System assigns each nursing home an overall  rating and three component ratings based on the extent to which the  nursing home meets CMS’s quality standards and other measures.  According to CMS, the primary goal of the Five-Star System is to provide  consumers with an easy way to understand nursing home quality and  distinguish between high- and low- performing nursing homes.
In recent years, we and others have expressed concerns with Nursing  Home Compare and the Five-Star System, including concerns over  CMS’s efforts to ensure the usefulness of these tools for consumers. In  our March 2012 report we reviewed, among other things, how the Five- Star System was developed and how CMS ensures its goals are met, and  noted that while the Five-Star System was a significant step toward  increasing the transparency of information important to consumers, to  sustain the system over time the agency would need to make a concerted  effort. CMS agreed with our recommendation to establish—through  planning documents—how its planned efforts would help the agency  achieve the goals of the Five-Star System. Additionally, some nursing  home stakeholder groups have raised concerns about the usefulness of  Nursing Home Compare for consumers as well as the accuracy of its  Five-Star System.
Given the importance of providing useful information to consumers  choosing a nursing home, along with the concerns raised by nursing  home stakeholders and others, you asked us to examine the use of  Nursing Home Compare and the Five-Star System as tools for  consumers. In this report, we examine:  1.  the information CMS collects about the use of the Nursing Home  Compare website, including its usefulness to consumers, and  potential areas, if any, to improve the website, and  2.  to what extent the Five-Star System on Nursing Home Compare  enables consumers to understand nursing home quality and make  distinctions between high- and low- performing nursing homes.
To examine the information CMS collects about the use of Nursing Home  Compare, including its usefulness to consumers, and CMS’s efforts to  improve the website, we reviewed documents from CMS, such as reports  on Nursing Home Compare’s website statistics and assessments of its  usability. In addition, we interviewed CMS officials, including those  responsible for the development and oversight of Nursing Home  Compare. We also conducted interviews with 30 nursing home  stakeholders. These included advocacy groups representing consumers  or providers, an organization of government agencies, and nursing home  quality experts at the national level as well as the state level for four  selected states—California, Georgia, Kansas, and Rhode Island. We  selected these four states based on factors such as variation in  geographic region and size (number of nursing homes). We conducted  interviews with these nursing home stakeholders in order to identify the  strengths and weaknesses, as well as potential areas of improvement, of  Nursing Home Compare and the Five-Star System, among other things.  Using this information and CMS internal documentation, we identified  areas of improvement for Nursing Home Compare. We also examined the  information we collected against CMS’s goals for the website and relevant  standards for internal control in the federal government.
To examine the extent to which the Five-Star System enables consumers  to understand nursing home quality and distinguish between high- and  low- performing nursing homes, we analyzed CMS documents, including  its February 2015 Five-Star System Technical Users’ Guide, the relevant  guide at the time of our analysis. We also interviewed CMS officials,  including those knowledgeable about the methodology for determining  Five-Star System ratings, and national and state nursing home  stakeholders, as described above. Furthermore, we analyzed CMS’s  Five-Star System ratings data from the most recent full quarter of data  available at the time of our analysis, which ended December 2015. First,  for nursing homes with adjacent overall ratings (i.e., 4-star homes  compared to 5-star and 3-star homes) we examined the extent to which  these homes’ underlying component scores overlapped at the state level.  For example, we examined the extent to which the component scores for  nursing homes that received a 4-star overall rating were higher or lower  than the component scores for nursing homes that received a 5- or 3-star  overall rating. Second, we determined the average age of the data used  to calculate each nursing home’s overall rating and component ratings.  Third, we recalculated overall star ratings on a national basis using the  Five-Star System’s methodology to determine how nursing homes’ state- based ratings compare to their nationally based ratings. In addition, we  analyzed consumer complaint data collected and recorded in CMS’s  Automated Survey Processing Environment Complaints/Incidents  Tracking System from July 2015 through December 2015 to examine the  extent to which each nursing home’s overall rating correlated with the  number of consumer complaints registered against it. We examined the  last six months of 2015 to provide a fuller picture of each nursing home’s  routine complaint levels. We analyzed these data at the national level and  for all 50 states and Washington, D.C., as appropriate. Finally, we  obtained 2015 nursing home resident satisfaction survey data from two of  our four selected states and compared these data to each nursing home’s  overall Five-Star rating from that same year in order to determine the  extent to which the overall ratings for nursing homes in those two states  correlated with the level of consumer satisfaction. The findings from this  final analysis cannot be generalized to other states. For all data used in  these analyses, we interviewed knowledgeable officials and reviewed  related documentation, and based on these steps, determined that the  data were sufficiently reliable to explore the relationship between the  overall rating and the component ratings, describe the age of the data,  determine national rating distributions, and assess consumer satisfaction  information. We examined the information contained in the Five-Star  System against CMS’s goals for the Five-Star System and HHS’s Agency  for Healthcare Research and Quality’s National Quality Measures  Clearinghouse. For additional information about our methodology for  conducting these analyses, see appendix I.
We conducted this performance audit from September 2015 to November  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: CMS’s goals for Nursing Home Compare and its Five-Star System are  consistent with its strategy to improve the quality of health care by  providing transparent information about the quality of health care  services, including those delivered in nursing homes. According to the  strategy, to achieve better care, patients must be given access to  understandable information and decision support tools that help them  manage their health and navigate the health care delivery system.
subsections: section_title: Nursing Home Compare
paragraphs: Since 1998, CMS has publicly reported information on nursing home  quality on its Nursing Home Compare website and has increased the amount of information reported on the website over time. CMS initially  reported information only about nursing home characteristics and nursing  home health inspection results on Nursing Home Compare. Later, CMS  began reporting additional information on the website, such as the ratio of  nursing staff to residents, nursing homes’ performance on various quality  measures, and the number of complaints registered against nursing  homes. Additionally, CMS has updated the appearance and functionality  of the Nursing Home Compare website over time, with the most  significant change being the introduction of the Five-Star System in 2008.
section_title: Five-Star System
paragraphs: In December 2008, CMS made the Five-Star System available to the  public on its Nursing Home Compare website in order to help consumers  compare nursing homes more easily. The Five-Star System assigns  each nursing home participating in the Medicare or Medicaid programs an  overall “star” rating, ranging from one to five. Nursing homes with five  stars are considered to have much above average quality, while nursing  homes receiving one star are considered to have much below average  quality. Calculation of the overall star rating is based on separate ratings  that nursing homes receive for each of three components: health  inspections, staffing, and quality measures.
Health inspection rating. CMS contracts with state survey agencies  to conduct unannounced, on-site nursing home health inspections— known as standard surveys—to determine whether nursing homes  meet federal quality standards. Every nursing home receiving  Medicare or Medicaid payment must undergo a standard survey not  less than once every 15 months, and the statewide average interval  for these surveys must not exceed 12 months. State surveyors also  conduct complaint investigations in response to allegations of quality  problems. If nursing homes are found to be out of compliance with  any requirements, state surveyors issue deficiency citations that  reflect the scope (number of residents affected) and severity (level of  harm to residents) of the deficiency. Surveyors conduct revisits to the  nursing home to ensure that the deficiencies identified have been  corrected. A nursing home’s health inspection rating is relative to  other nursing homes’ health inspection ratings in their state. As such,  health inspection ratings are assigned to generally achieve the  following distribution within each state: the top 10 percent of nursing  homes receive five stars, the bottom 20 percent receive one star, and  the middle 70 percent of nursing homes receive two, three, or four  stars.
Staffing rating. Nursing homes self-report staffing hours worked for a  2-week period at the time of the standard survey. CMS converts the  reported point-in-time staffing hours for nursing staff—registered  nurses, licensed practical nurses, and certified nursing assistants— into measures that indicate the number of registered nurse and total  nursing hours per resident per day. CMS adjusts the staffing levels  for differences in the level of complexity of nursing services required  to care for residents across nursing homes—referred to as resident  acuity. Each nursing home’s staffing rating is assigned based on how  its total nursing and registered nurse staffing levels compare to the  distribution of staffing levels for freestanding homes in the nation and  staffing level thresholds identified by CMS.
Quality measure rating. Nursing homes regularly collect assessment  information on all their residents, including information on the  residents’ health, physical functioning, mental status, and general  well-being. Nursing homes self-report this information to CMS. CMS  uses some of the assessment information to measure the quality of  certain aspects of nursing home care, such as the prevalence of  pressure sores and changes in residents’ mobility. At the time of our  analysis, CMS calculated this rating for each nursing home based on  11 of the 18 quality measures posted on Nursing Home Compare.  Information on the remaining 7 quality measures is posted on the  website but not used in the calculation of the rating. A nursing  home’s quality measure rating is assigned based on national  thresholds established by CMS.
The overall star rating is calculated using a process that combines the  star ratings from the health inspection, staffing, and quality measure  components—with the greatest weight given to the health inspection  rating. The overall rating is assigned based on the following steps:  1.  Start with the number of stars for the health inspection rating.  2.  Add one star if the staffing rating is four or five stars and also greater  than the health inspection rating. Subtract one star if the staffing rating  is one star. The overall rating cannot go above five stars or below one  star.  3.  Add one star if the quality measure rating is five stars. Subtract one  star if the quality measure rating is one star. The rating cannot go  above five stars or below one star.
See figure 1 for an example of how a nursing home’s overall rating is  calculated. CMS updates the ratings on a monthly basis; however, a  particular home’s overall rating will only change if it had new data that  affected any one of the component ratings. For example, when a home  has a health inspection survey, either a standard survey or a complaint  investigation, the deficiency data from the survey will become a part of  the calculation for the health inspection rating and the overall rating will  also be adjusted, if necessary.
section_title: CMS Collects Information on the Use of the Nursing Home Compare Website, but Lacks a Systematic Process for Prioritizing and Implementing Improvements
subsections: section_title: CMS Collects Information on Website Use
paragraphs: We found that CMS utilizes three standard mechanisms for collecting  information on the use of the Nursing Home Compare website: website  analytics, website user surveys, and website usability tests.
Website analytics. CMS utilizes website analytics to gauge the  performance of Nursing Home Compare and improve the visibility of  the website in search engine listings. Through this mechanism, CMS  is able to track data such as how many users, sessions, bounce rates,  and page views Nursing Home Compare has per year. For example,  these data show that from 2013 to 2015, Nursing Home Compare  averaged 1.5 million sessions per year and 914,000 users per year.  The website analytics also track the average session duration and the  average number of pages that are viewed per session. For this same  time period, the average session duration was 5.8 minutes and the  average number of pages that were viewed per session was 4.8.
Website user surveys. CMS utilizes website user surveys to collect  information about Nursing Home Compare users, how they use the  site, and their opinions about the site. According to CMS officials,  these surveys, which CMS began using in 2013, randomly pop up in  web browsers to 50 percent of the website’s visitors. The surveys ask  the user to identify themselves (for example, a caregiver or a  researcher), the primary purpose for visiting the site, and the user’s  experience in using the site. These surveys provide the only way CMS  determines the type of users who come to the website, according to  CMS officials. In October 2015, website survey data showed that 59  percent of users of Nursing Home Compare identified themselves as  consumers, and the majority of users report coming to the site to  research or select nursing homes for themselves or a family  member.
Website usability tests. CMS utilizes usability tests—in the form of  one-on-one sessions with nine consumer participants—to assess how  well consumers navigate the website. CMS has conducted four  usability tests from 2011 through 2015. The tests focus on the  navigability of the website; however, they also include a few  background questions about consumer use. For example, the usability  tests ask if participants were previously aware of Nursing Home  Compare, what factors consumers find most important in searching  for a nursing home, and what expectations consumers have for a  nursing home comparison website such as this.
In addition to these three standard mechanisms, CMS officials also told  us that they gain insight into the use of Nursing Home Compare by  holding ad-hoc meetings with a variety of stakeholders that are familiar  with Nursing Home Compare to discuss the website. CMS held three  stakeholder meetings from 2010 through 2015. According to CMS  documents, stakeholders have included groups that represent  consumers, such as ombudsmen, consumer advocate groups, provider  advocate groups, and others that are involved in nursing home services.  Information exchanged during stakeholder meetings includes CMS  presentations on pending changes, such as the development of new  quality measures, and stakeholder feedback.
The mechanisms that CMS utilizes to collect information about the use of  Nursing Home Compare provide the agency with valuable information.  However, these mechanisms do not provide CMS with information on the  usefulness of the website to a broader range of consumers. Specifically,  the usability tests are not designed to assess the website’s usefulness to  consumers, and the website analytics and user surveys only provide  information about consumers who access the website. Therefore, the  mechanisms do not provide CMS with information on nursing home  consumers who have not used the website because they are unaware of  it, or choose not to use it, as well as the reasons why. In stakeholder  interviews we conducted, some nursing home stakeholders noted that  many consumers do not know about the website and that consumers  collect information from other sources. Obtaining information from  consumers who do not access Nursing Home Compare would likely  require the dedication of resources to, for instance, consumer-oriented  focus groups or broader surveys.
section_title: While Improvements Have Been Identified for the Nursing Home Compare Website, CMS Does Not Have a Systematic Process for Implementing Them
paragraphs: We identified five key areas of improvement CMS could make to Nursing  Home Compare in order to make it more helpful for consumers.  Specifically, we reviewed over 300 individual improvements identified in  CMS documents—in part resulting from the mechanisms described  above—and in interviews with national and state stakeholders; for  example, one internal CMS analysis included over 40 individual  recommendations for improvement. Through our analysis, we found the  key areas of improvement are: 1) explanation of how to use the website,  2) additional information about the nursing home, 3) community and  consumer outreach, 4) clarity of the website, and 5) navigability of the  website.
Table 1 below provides more information about these key areas of  improvements. For example, the first improvement addresses the fact that  the Nursing Home Compare website does not currently have an  explanation of how to use the website prominently displayed on its home  page; there is not an introduction to the website, or an obvious  explanation of how it should be used. According to many stakeholders,  Nursing Home Compare is a valuable tool for consumers but a few  specified that additional explanatory information is needed; without such  information, the usefulness of the website may be limited.
Although CMS has identified the need for improvements to its Nursing  Home Compare website, the agency does not have a systematic process  that prioritizes recommended website changes and sets a timeline for  implementation. In response to a recommendation in our 2012 report, in  August 2013, CMS developed a strategic plan for evaluating the usability  of Nursing Home Compare. The plan described tasks, including an  expert review of the website, an analysis of competitor websites, and  usability testing, some of which resulted in the formal mechanisms that  CMS now has in place to collect information on the use of Nursing Home  Compare, as previously described. However, CMS does not have a  documented and systematic approach describing how to prioritize  recommended changes to the website and assessing the potential  improvements. Instead, officials described a fragmented approach to  reviewing and implementing recommended website changes that may  include verbal discussions of various factors, such as which changes  would provide the broadest impact. CMS officials stated that their current  approach to handling website changes had been working well, but since  the website has become more complex in recent years, they  acknowledged the need for a more formalized approach in addressing  identified website changes.
CMS has stated the goal for its Nursing Home Compare website as  assisting consumers in finding and comparing information about nursing  home quality. In addition, under federal internal control standards,  management should address identified program deficiencies on a timely  basis and evaluate appropriate actions for improvement. However, in the  absence of an established process to evaluate and prioritize  implementation of improvements, CMS cannot ensure that it is fully  meeting its goal for the website.
section_title: Several Factors Inhibit the Ability of the Five-Star System to Help Consumers Understand Nursing Home Quality and Choose Between Homes
paragraphs: Our analysis for the Five-Star System’s ratings data found that its overall  rating provided consumers with distinctions between the highest and  lowest performing nursing homes for health inspections in most states.  Specifically, we found that, in 37 out of 50 states, homes that received an  overall rating of 5-stars consistently had higher health inspection scores— the component measure that most significantly contributes to the overall  rating—than homes that received an overall rating of 1-star. This means  that in the 37 states, consumers can safely assume that, in the case of  health inspections, the performance of any nursing home in their state  with an overall high 5-star rating is better than the performance of any  home with an overall low 1-star rating. Some stakeholders we spoke with  agreed that distinctions between nursing homes are clearest at the  extremes. For example, one stakeholder noted that the Five-Star System  is best at helping consumers identify the poorest performing homes to  avoid. Stakeholders also noted the value of having a national resource  that uses standardized and objective nursing home quality information.
However, we also identified four factors that may inhibit the ability of  consumers to use the Five-Star System ratings as an easy way to  understand nursing home quality and identify high- and low- performing  homes, CMS’s stated goal for the Five-Star System.  1.  Interpreting overall ratings. As previously described, the Five-Star  System’s overall rating is calculated using a process that combines  three component ratings. However, the formula for combining the  components is not intuitive, which can make interpreting overall  ratings difficult for consumers by both complicating the comparison of  overall ratings and masking the importance of the component ratings.  Specifically, the comparison of overall ratings can be complicated  because a consumer cannot assume that the performance on a  particular component of the higher-rated home is better than that of  the lower-rated home. In our review, we generally did not find  distinctions in the scores for homes in the same state with adjacent  overall ratings—e.g., 2- and 3-star homes—or for homes with middle  overall ratings—2-, 3-, and 4-star homes. For example, in one state,  28 percent of homes with a 3-star overall rating had a better health  inspection score than the average health inspection score for homes  with an overall 4-star rating. Furthermore, the way CMS calculates  the overall rating can mask for consumers issues that may be present  in the component ratings. A consumer comparing nursing homes will  see each home’s overall rating and component ratings, but they may  not understand the impact each component score has on the overall  rating. This could lead a consumer to rely more on the overall rating  when their individual needs may require attention to one specific  component more than the others. For example, two nursing homes  that both have a 4-star overall rating could have opposite quality  measure component ratings—one with a low 1-star quality measure  rating and the other with a high 5-star quality measure rating. (See fig.  2).
Many stakeholders stated that it is difficult to distinguish between  nursing homes with adjacent or middle ratings. In addition, some  stakeholders expressed concern about the overall rating, with one  explaining that often consumers make decisions based on the overall  rating without understanding it or looking at the underlying  components. According to CMS officials, the overall rating provides a  summary of complex information to guide consumers—not an explicit  report card—in as simple a way as possible. They also added that by  providing individual component ratings, consumers have the ability to  dig deeper into the source of the overall rating.  2.  Timeliness of ratings data. Each of the three rating components— which influence the overall star rating—use a unique source of data  that are collected from nursing homes at different time intervals.  Specifically, the number of stars assigned to a nursing home is a  point-in-time picture of performance based on a prior snapshot of the  home’s performance and may not reflect a nursing home’s current  status. (See table 2). Some stakeholders we spoke with expressed  concern that a consumer may make a decision about a nursing home  based on data that does not reflect current conditions in the home.  According to CMS officials, a delay is always present due to  administrative processes such as validating data prior to being  posted. For example, the health inspection component data may be  delayed due to additional information from the outcomes of revisits to  the nursing home to check that deficiencies have been corrected.  CMS officials and stakeholders said the Five-Star System should not  be the only source of information a consumer uses; they both  encourage consumers to explore additional information including  visiting the home.  3.  Comparing nursing homes across states. The overall rating and  health inspection rating do not allow consumers to compare the  quality of homes across states, limiting the ability of the rating system  to help consumers who live near state borders or have multistate  options where they could place their family members. Because ratings  are relative to other nursing homes within a state, homes that receive  the highest and lowest ratings in their state may not be the highest or  lowest performing homes in another state or nationally. So, a  consumer cannot assume that a 5-star nursing home in one state  would be rated as a 5-star home in any other state. (See fig. 3).  Furthermore, we found that when we recalculated the star ratings  using a national distribution rather than a state distribution, homes’  ratings often changed, sometimes dramatically. For example, about  23 percent of nursing homes with a 1-star overall rating in December  2015 had improved ratings when compared nationally and about 30  percent of homes with a 5-star overall rating had decreased ratings  when compared nationally. When looking at individual states, we  found that the nursing homes in some states would fare better or  worse under a national rating. Specifically, the percentage of homes  receiving an overall 1-star rating doubled in 4 states and the  percentage of homes receiving an overall 5-star rating doubled in 9  states. See appendix III for additional information about the results of  our analysis.
According to CMS Five-Star System documentation, the rating system  is not designed to compare nursing homes nationally. Instead, ratings  are only comparable for homes in the same state. CMS made the  decision to base the health inspection component on the relative  performance of homes within the same state primarily due to variation  across the states in the execution of the standard surveys. Because  the health inspection component most significantly contributes to the  overall rating, this means that the overall rating also cannot be  compared nationally. However, the addition of national ratings would  be helpful for consumers and we have previously made  recommendations to CMS that would help decrease survey variation  across states. CMS has taken action on many of these  recommendations.  4.  Lack of consumer satisfaction information in ratings. Because the  Five-Star System does not include consumer satisfaction  information—a key quality performance measure—the rating system  is missing important information that could help consumers distinguish  between high- and low- performing nursing homes. We believe  consumer satisfaction surveys could be a more direct measure of  nursing home satisfaction than other available measures. For  example, our analysis of consumer satisfaction data shows that  nursing homes with higher overall star ratings did not necessarily  have higher resident satisfaction scores or fewer complaints. (See fig.  4). Specifically, our analysis found that the Five-Star System overall  ratings for each nursing home in two states that conduct resident  satisfaction surveys were only slightly correlated with the percentage  of residents that would recommend the home to other consumers—an  indicator of consumer satisfaction included on the state surveys.  Similarly, when analyzing complaint data for all states—a proxy for  consumer satisfaction—we also found only a slight correlation  between the total number of consumer complaints registered against  a home in each state and the home’s overall Five-Star System  rating. Many stakeholders told us that they would like to see resident  satisfaction included in the Five-Star System. For example, one state  stakeholder group explained that they think it is important for a  consumer making a nursing home decision to understand how the  administration resolves an issue with a resident when one arises. That  type of information is not currently captured in the Five-Star System,  but could be captured through a resident satisfaction survey, which  could strengthen the ratings.
According to CMS officials, they recognize that consumer satisfaction  is important information, but collecting the data in a consistent,  objective way for all of the nursing homes in the country is a  challenge. They acknowledged that some states have been able to  overcome these implementation challenges and administer statewide  nursing home consumer satisfaction surveys. Until consumer  satisfaction information is included in the rating system, consumers  will continue to make nursing home decisions without the benefit of  this key performance measure and may not be choosing the home  that would best meet their needs. While we recognize that gathering  this information is challenging, CMS has done so in its hospital rating  system. Specifically, CMS developed a hospital consumer satisfaction  survey with assistance from HHS’s Agency for Healthcare Research  and Quality—an agency that, among other things, focuses on quality  measurement and includes consumer satisfaction as one of its  National Quality Measures Clearinghouse’s clinical quality measures.
In addition to the items discussed above, presentation of the Five-Star  System does not prominently display key explanatory information that  could help consumers better understand how to use the ratings.  Specifically, we found that CMS does not prominently provide  descriptions of how to understand the ratings and what consumers should  consider when using the ratings or information on how the overall rating is  calculated. In addition, CMS clearly discloses the date of the data used to  assign stars for the health inspection component, but not for the staffing  or quality measure components, and does not prominently state the  previously discussed limitation that homes can only be compared within a  state. For example, in order to find descriptions of how the overall rating  is calculated, consumers must follow links that take them off of their  nursing home search and results webpages and, as noted previously, an  average webpage visit is less than 6 minutes. Many stakeholders we  spoke with explained that consumers often have very little time to make a  nursing home decision, and a few noted that it is also a stressful process,  therefore making prominent and readily available information crucial. In  addition, many stakeholders expressed concerns that consumers may not  understand the ratings and how they are calculated. Further, many  stakeholders expressed concern about the timeliness of the data, with  some noting that consumers were generally unaware of the timing of the  data. CMS officials described the tension between keeping the Five-Star  System as simple as possible for consumers so that they can quickly  understand the ratings and also providing enough information on how and  when the ratings are calculated. Collectively, the four factors that hinder  consumers’ ability to use the Five-Star System ratings, along with the lack  of explanatory information provided by CMS, may limit the Five-Star  System’s ability to meet CMS’s goal of providing consumers with an easy  way to understand nursing home quality and make distinctions between  high- and low- performing homes.
section_title: Conclusions
paragraphs: Nursing Home Compare and the Five-Star System seek to help  consumers choose among nursing homes. Nursing home selection can  be a stressful and time-sensitive process, so these are important tools  that CMS makes available to the public. However, our review found  opportunities for improvement in both the website and the ratings. CMS  has given much attention to the website since its inception almost 20  years ago. For example, the agency has put in place mechanisms for  reviewing the website’s use and has identified numerous improvements  that could be made. However, without a systematic process for reviewing  options and determining priorities for improvement—currently absent from  their efforts—CMS is unable to ensure that the website is meeting its  intended goal.
A key component of the website is the Five-Star System, containing  important quality information on every nursing home so consumers can  differentiate between them and choose those that can best meet their  needs. Because the Five-Star System contains multiple types of  information, compiled from different sources, and has complexities  inherent in ratings systems, it can be challenging for consumers to fully  understand how to take advantage of the varied information it contains.  Additional capability and information not currently included in the rating  system could also benefit consumers trying to differentiate between high-  and low- performing nursing homes—such as the ability to compare  homes nationally and the addition of consumer satisfaction survey  information. In addition, prominently displaying explanatory information on  how to use the ratings, which does not require users to navigate off the  nursing home search and results pages, could help address challenges  consumers face when trying to understand the ratings. Absent such  enhancements, CMS cannot ensure that the Five-Star System is fully  meeting its stated goal of helping consumers easily understand nursing  home quality and distinguish between high- and low- performing homes.
section_title: Recommendations
paragraphs: To strengthen CMS’s efforts to improve the usefulness of the Nursing  Home Compare website for consumers, we recommend that the  Administrator of CMS establish a systematic process for reviewing  potential website improvements that includes and describes steps on how  CMS will prioritize the implementation of potential website improvements.
To help improve the Five-Star System’s ability to enable consumers to  understand nursing home quality and make distinctions between high-  and low- performing homes, we recommend that the Administrator of  CMS take the following three actions: add information to the Five-Star System that allows consumers to  compare nursing homes nationally; evaluate the feasibility of adding consumer satisfaction information to  the Five-Star System; and develop and test with consumers introductory explanatory information  on the Five-Star System to be prominently displayed on the home  page. Such information should explain, for example, how the overall  rating is calculated, the importance of the component ratings, where  to find information on the timeliness of the data, and whether the  ratings can be used to compare nursing homes nationally.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS for its review and comment.  HHS provided written comments, which are reproduced in appendix V. In  its comments, HHS described the history of the Nursing Home Compare  website and the Five-Star System, improvements the agency has made  to both, and concurred with three of our four recommendations. In  particular, HHS concurred with our recommendations to establish a  process for reviewing potential website improvements that describes how  it will prioritize their implementation, evaluate the feasibility of adding  consumer satisfaction information to the Five-Star System, and develop  and test explanatory information on the Five-Star System to be displayed  on the home page. HHS did not concur with our recommendation to add  information to the Five-Star System that would allow consumers to  compare nursing homes nationally. HHS indicated that because of state  variation in the execution of standard surveys, it is difficult to compare  homes nationally on the health inspection component. They also noted  that the Five-Star System is just one of many factors consumers should  use when selecting a nursing home. As we describe in this report, efforts  have been and should continue to be made to reduce state variation in  standard surveys. For example, CMS regional offices are tracking state  differences in deficiency citations. We maintain that the ability for  consumers to compare nursing homes nationally is critical to making  nursing home decisions, especially for those consumers who live near  state borders or have multistate options, and that our recommendation  remains valid.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution of it until 30 days from its  date. At that time, we will send copies to the Secretary of HHS. In  addition, the report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or clowersa@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on   the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VI.
section_title: Appendix I: Additional Detail on GAO’s Data Analyses
paragraphs: This appendix describes additional details of the data analyses we  conducted to examine the Five-Star Quality Rating System (Five-Star  System). For this examination, we analyzed data from three sources. We  analyzed Five-Star System data from the Centers for Medicare &  Medicaid Services (CMS). These data provide detailed rating information  on over 15,000 nursing homes included in the Five-Star System from the  most recent full quarter available at the time of our analysis, which ended  December 2015. Additionally, we analyzed CMS consumer complaint  data for a six-month period ending in December 2015. We collected and  analyzed these data for all 50 states and Washington, D.C. Furthermore,  we collected and analyzed 2015 nursing home resident satisfaction  survey data from two of our four selected states that collect survey data.
We conducted the following analyses:  1.  To determine the extent to which the Five-Star System provides  consumers with information distinguishing between high- and low-  performing nursing homes, we analyzed December 2015 data from  CMS’s publicly available Five-Star Scores and Ratings data.  Specifically, for each state, we determined the range of scores that  underlie each component star rating: for health inspection we used  the weighted all cycles score, for staffing we looked at the total  adjusted staffing score, and for quality measures we calculated the  total quality measure score—for each nursing home’s overall rating.  We then determined whether, within each state and for each  component rating, the scores of the worst performing 5-star nursing  homes overlapped with the scores of the best performing 1-star  nursing homes. We also conducted this analysis for each  combination of the star ratings.  2.  To determine the timeliness (or age) of the data used for each  component Five-Star ratings for consumers viewing the ratings in  December 2015, we analyzed data from CMS’s Five-Star Scores and  Ratings data for that month downloaded from CMS’s website.  Specifically, we calculated the average age of the data for each  component rating at that point in time. For the health inspection  component, we analyzed the standard survey date, but did not  analyze the complaint investigation date because a meaningful  average age cannot be calculated.  3.  To determine the extent to which nursing homes’ ratings changed  when compared nationally rather than compared only within each  state, we analyzed December 2015 data from CMS’s Five-Star Score  and Ratings data downloaded from CMS’s website. Specifically, we  recalculated each nursing home’s health inspection and quality  measure scores that are normally assigned based on state  distributions so that they were based on a national distribution (new  distribution allotments were based on CMS’s state distribution  guidelines). We then recalculated each home’s overall rating using  our new health inspection component rating, our new quality measure  rating, and CMS’s staffing component rating. In addition, we  analyzed the change in overall nursing home ratings when applying  the methodology nationally.  4.  To determine the relationship between nursing home satisfaction data  and CMS’s Five-Star ratings, we did the following:  a.  We used complaints registered against nursing homes by  residents, families, ombudsmen, or others as a proxy  measurement of satisfaction. Specifically, we analyzed complaint  data recorded in CMS’s Automated Survey Processing  Environment Complaints/Incidents Tracking System from July 1  through December 31, 2015. We examined the last six months of  2015 to provide a fuller picture of each nursing home’s routine  complaint levels. For each state and nationally, we determined the  correlation between each nursing home’s total number of  registered consumer complaints and its overall Five-Star rating.  b.  We used the results of 2015 nursing home resident satisfaction  surveys from two of our selected states that collect such  information. Specifically, we focused on the responses to whether  the resident would recommend that nursing home to others as a  measure of satisfaction. In one state, this measure was the actual  percentage of residents that recommended the home and in the  other state it was the ranking of the home based on residents’  responses. For both states, we determined the correlation  between each nursing home’s resident response on the state  survey and its overall Five-Star rating. The findings from this  analysis cannot be generalized to other states.
For all data used in these analyses, we interviewed knowledgeable  officials and reviewed related documentation and based on these steps  determined that the data were sufficiently reliable to explore the  relationship between the overall rating and the component ratings,  determine national rating distributions, assess consumer satisfaction  information, and describe the age of the data.
section_title: Appendix II: Nursing Home Stakeholder Views on Consumer Use of Nursing Home Compare
paragraphs: We held interviews with 30 nursing home stakeholders—eight national  stakeholders and 22 state stakeholders from four states (Rhode Island,  Georgia, Kansas, and California) we selected based on factors such as  variation in geographic region and size (number of nursing homes).  These stakeholders represent a range of provider groups, consumer  groups, government agencies, and technical experts. We selected  organizations in each state and nationally that are relevant to nursing  home consumers and providers. Technical experts were identified by their  prominence in the nursing home quality research field. In addition, some  stakeholders we interviewed identified other groups that would be  appropriate to interview. Our interviews included a set of questions  regarding consumer use of Nursing Home Compare. Responses to these  questions cannot be generalized beyond the stakeholders we  interviewed.
We found that stakeholders generally could not quantify the number of  consumers who use Nursing Home Compare, but most speculated that  consumers use the site “a little” to “somewhat” and a few stakeholders  said that consumers use the website “a lot”. Some stakeholders thought  the number of people using Nursing Home Compare was growing, and  one stakeholder thought this was because people are generally trying to  make more educated decisions about nursing home care. Another  stakeholder thought this increase could also be a result of people using  the Internet to look things up more frequently, nursing homes included.
Some stakeholders noted that use of Nursing Home Compare probably  differs depending on whether the patient is searching for care in an urban  or a rural setting. Specifically, they stated that they think Nursing Home  Compare is used more frequently in urban areas, where more nursing  home options are available, compared to rural areas where there may be  only one home in a town. See Table 3 below for summary of stakeholder  responses.
Some stakeholders stated that they believe the extent to which  consumers use Nursing Home Compare may depend on the amount of  time that the consumer has to research and pick a nursing home. For  example, according to a few stakeholders, if someone’s family member is  getting discharged from the hospital and needs to be placed in a nursing  home immediately, the consumer is less likely to use Nursing Home  Compare. In contrast, one stakeholder noted that if consumers are  planning for the future and researching nursing homes before care is  needed, then they are more likely to use Nursing Home Compare.
We found that stakeholders were split in their responses about when  consumers would typically use Nursing Home Compare – whether as an  initial step in beginning their nursing home search or as a way to confirm  recommendations obtained from others. Most stakeholders stated that the  consumer using the website is most likely a family member—usually an  adult child or grandchild—and rarely the individual in need of a nursing  home placement.
When asked how valuable the information provided on Nursing Home  Compare is to consumers who are researching and choosing a nursing  home, most stakeholders stated that it was “somewhat valuable”, and  some said that they thought it was “very valuable”. One stakeholder said  that the information was “of little value”. See Table 4 below for a summary  of stakeholder responses.
Some stakeholders stated that the information on Nursing Home  Compare is a good place to start and may help consumers narrow down  their search, but ultimately it is not likely to be the only source of  information. Many stakeholders agreed that in addition to conducting  online research, consumers should also always try to visit nursing homes  in person before a making a decision. A few stakeholders stated that  observing a nursing home and its current residents firsthand on any given  day provides the most valuable information when making a decision. In  addition, stakeholders noted that consumers also obtain information  about nursing homes through other sources—primarily through word of  mouth from friends, family, and neighbors, and from information provided  by primary care physicians, hospital discharge planners, and local  ombudsmen. Many stakeholders noted that in most cases though, the  location of a nursing home is often the main determinant of where a  family member is placed.
Stakeholders mentioned that consumers also use third party, private  websites, and in some states, such as California and Kansas, consumers  may rely on websites with state-specific nursing home information. Some  stakeholders thought that consumers used these other sources of  information more often than Nursing Home Compare. A couple of  stakeholders thought consumers preferred these third-party, private  websites because some of them provide a more personalized experience  and offer the opportunity to speak with someone on the phone, where  CMS’s Nursing Home Compare does not provide either of those options.  A couple other stakeholders thought that consumers may be more likely  to use and trust Nursing Home Compare compared to these other private  websites simply because it is a government website. A couple of  stakeholders expressed concern about consumers using third-party,  private websites because nursing homes may pay to be included, and so  the website may not provide objective information on nursing home  options for consumers. Additionally, these private websites often appear  on search engine results before Nursing Home Compare, so consumers  may use them before seeing and using CMS’s Nursing Home Compare  website.
Finally, stakeholders provided mixed responses regarding whether they  suggested consumers use Nursing Home Compare when helping them  search for nursing homes. For example, several stakeholders told us that  they routinely referred consumers to the website, while another said that  she would only direct consumers to Nursing Home Compare if they were  not familiar with the area, or if did not have any time to spend on the  nursing home search process. The stakeholder would ultimately  recommend the consumer coming in and talking to her, and then would  not use Nursing Home Compare at all.
section_title: Appendix III: Potential Changes to Ratings If National Distribution Used Rather than Existing State Distribution, December 2015
section_title: Appendix IV: Correlation between Consumer Complaints, July through December 2015, and Overall Ratings, December 2015
section_title: Appendix V: Comments from the Department of Health and Human Services
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, Linda Kohn, Director, Karin  Wallestad, Assistant Director, Kathryn Richter, Analyst-In-Charge, Amy  Andresen, Julianne Flowers, Shannon Smith, and Brienne Tierney made  key contributions to this report. Also contributing were Jacques Arsenault,  Wesley Dunn, Krister Friday, Rich Lipinski, Dae Park, Vikki Porter, and  Steven Putansu.